In general the most active locations on the file system will be:
- During nightly data transfer (usually 00:04 through ~01:00 or later): The Console Data Repository directory (where the UDR data is written on the Gateway Server). This directory is a property of the Manager run. By default it is /usr/adm/best1_workspace/repository.
- During nightly data processing (usually 00:15 through ~04:00 or later): The /[Manager Output Directory] and UDR Console Data Repository. The /[Manager Output Directory] is a property of the Manager run (OUTPUT_DIRECTORY). It's the unique output directory per Manager run. This is where the files like *ProcessDay are written.
- Throughout the day: The $BEST1_HOME/local/manager/status, $BEST1_HOME/local/manager/log directories
Note that the TSCO Agent/BPA agent will also be running on the console and it will be accessing logs in the following directory:
$BEST1_HOME/bgs/log
Here is an overview of the nightly TSCO Gateway Server BPA console data processing process activity.
== Every 1 minute ==
Every minute throughout the day the $BEST1_HOME/bgs/bin/pcron command is executed by cron. It determines which TCCO Gateway Server/BPA commands need to be executed that minute and executes them as necessary.
Output files:
- Log file: $BEST1_HOME/bgs/log/pcron/[hostname]-[user].log
Input/Output Files:
- $BEST1_HOME/bgs/pcron/repository/[hostname]-[user].pcron
- $BEST1_HOME/bgs/pcron/repository/[hostname]-[user].counter
== 23:30 or 23:45 (vcmds:COLLECT_LEAD_TIME minutes before vcmds:DAILY_BEGIN) ==
The /[Manager Output Directory]/[begin]-[end].Manager script is executed by pcron. It calls the $BEST1_HOME/bgs/scripts/best1manager script which in turn calls the $BEST1_HOME/bgs/bin/best1manager.awk script.
WARNING: If the manager script file is deleted then this entire process will fail. This can happen if the customer decides to delete old files manually on the Gateway
Server system.
The TSCO Application Server Gateway Manager default COLLECT_LEAD_TIME value is 30 minutes (so Manager scripts are scheduled to run at 23:30 by default). In earlier releases of the product (where runs were started via the TSCO Gateway Server X-Windows console) the default COLLECT_LEAD_TIME was 15 minutes so runs were scheduled at 23:45 to issue data collection requests starting at 00:00.
When using AutoNodeDiscovery the first step of the Manager script execution uses the $BEST1_HOME/bgs/bin/AutoNodeDiscovery command to refresh the domain (*.dmn) file (which defines which computers will be managed by this run) based upon the master computer list source file defined for the run.
Input files:
- The run's *.vcmds file
- The run's *.dmn file
Output files:
- Log file: /[Manager Output Directory]/best1manager.log
- Created: /[Manager Output Directory]/[date]-[date].ProcessDay
- Created: /[Manager Output Directory]/[date]-[date].Collect
- Created: /[Manager Output Directory]/[date]-[date].XferData
Executed:
- $BEST1_HOME/bgs/bin/udrCollectMgr process which sends collection request to all of the remote agents (throttled based upon the $BEST1_HOME/local/setup/collectManager.cfg COLREQ_CONCURRENT_RUN and COLREQ_CONCURRENT_GLOBAL parameters -- which, by default, allow 10 concurrent start collection request threads at a time).
The udrCollectMgr process would then continue to run for the rest of the day (during the whole collect and transfer period of the run).
udrCollectMgr files:
- Log file: $BEST1_HOME/local/manager/log/[hostname]-[RunName]-MM-DD-YYYY-############.log
- $BEST1_HOME/local/manager/run/[RunName]-MM-DD-YYYY-############.mrc
- $BEST1_HOME/local/manager/run/[hostname]-MRQ
- Semaphore: $BEST1_HOME/local/manager/run/colreq_concurrent_global.key_##
== Every 15 minutes throughout the day ==
Every 15 minutes the udrCollectMgr will query the status of each remote agents (throttled based upon the $BEST1_HOME/local/setup/collectManager.cfg COLREQ_CONCURRENT_RUN and COLREQ_CONCURRENT_GLOBAL parameters -- which, by default, allow 10 concurrent start collection request threads at a time) and update the UCM Status Reports (if vcmds:COLLECT_STATUS_REPORTING set to YES).
Output files:
- $BEST1_HOME/local/manager/status/ UCMStat-[RunName]-MM-DD-YYYY-############.xml
- $BEST1_HOME/local/manager/status/ UCMStat-MM-DD-YYYY.xml
== 23:58 (and every 15 minutes afterwards at 13, 28, 43, and 58 minutes past the hour for the next 24 hours) ==
The /[Manager Output Directory]/[date]-[date].Collect script is executed.
The [date]-[date].Collect script executes a udrCollectMgr process for the run. The udrCollectMgr process checks the $BEST1_HOME/local/manager/run/[hostname]-MRQ and if there is already a udrCollectMgr process running for the run it exits. If there isn't one currently running the newly executed udrCollectMgr process takes ownership of the run.
The following steps happen 24 hours after the start of the collection for the day (these are the TRANSFER and PROCESSING stages of the run):
== [Next day] 00:04 (collectManager.cfg:COLMGR_TRANSFER_DELAY minutes after vcmds:DAILY_END) ==
The running udrCollectMgr process transitions from COLLECT mode to TRANSFER mode and begins initiating data transfer request to the remote agents (throttled based upon the $BEST1_HOME/local/setup/collectManager.cfg TRANSFER_CONCURRENT_RUN and TRANSFER_CONCURRENT_GLOBAL parameters -- which, by default, allow 10 concurrent transfer request threads at a time).
== [Next day] 00:09 (vcmds:DELAY_RUN_BY minutes after vcmds:DAILY_END) ==
The /[Manager Output Directory]/[date]-[date].XferData script is executed by pcron. The [date]-[date].XferData script executes a udrCollectMgr process for the run. The udrCollectMgr process checks the $BEST1_HOME/local/manager/run/[hostname]-MRQ and checks the status of the run. If the run status indicates that all of the computers in the run have successfully transferred or the run is beyond the vcmds:TRANSFER_DURATION time then the [date]-[date].XferData script executes the [date]-[date].ProcessDay script to begin processing. If not, the [date]-[date].XferData script goes into a 15 minute wait cycle waiting for one of those conditions to become true.
Output files:
- Log file: /[Manager Output Directory]/[date]-[date].manager.log
== [Next day] After the transfer period (So usually between 00:15 and 01:15 or later) ==
The /[Manager Output Directory]/[date]-[date].ProcessDay is executed by the [date]-[date].XferData script. This is the script that manages data processing in the environment.
Output files:
- Log file: /[Manager Output Directory]/[date]-[date].ProcessDay.out
- Many other files in the /[Manager Output Directory]
The [date]-[date].ProcessDay script manages data processing. That means that for each vcmds:VISUAL_INTERVAL of the day it is going to do the following:
(1) Execute a $BEST1_HOME/bgs/bin/bgsanalyze process that will read the UDR data for the computers defined in the domain and create a model (*.md) file and Visualizer (*.vis) file that contains the processing results
(2) [If vcmds:RUN_PREDICT set to YES]: Execute a $BEST1_HOME/bgs/bin/bgsbest1 (Predict) process that will read the model (*.md) file and generate a Predict Visualizer (*.vis) file.
Then once all of the intervals have been processed it will:
(3) Run the $BEST1_HOME/bgs/bin/combine.pl script that will combine the individual interval Visualizer files into a single combined Visualizer file (with a name like MDDh[domain].vis).
(4) [If vcmds:POPULATE set to YES]: Run the $BEST1_HOME/bgs/scripts/mpopulate.sh which will populate the Visualizer file into the defined Oracle database
(5) [If vcmds:BEGIN_CLEANUP_OUTPUT set to YES]: Clean up the temporary files created during the Manager execution (based upon file extensions selected in the vcmds:FILETYPES list)
(6) [If vcmds:BEGIN_DATA_COMPRESSION set to YES]: Find and compress the UDR data older than vcmds:BEGIN_DATA_COMPRESSION -> DAYSOLD
(7) [If vcmds: BEGIN_DATA_DELETION set to YES]: Find and delete UDR data older than vcmds: BEGIN_DATA_DELETION -> DAYSOLD
New in version 10.7.00 amd later: This option isn't enabled by default but there is a new UDR DATA RETENTION available in the collectManager.cfg file that can be used to define a maximum UDR retention across all console repositories. This can be used instead of per Manager run UDR data retention properties or in addition to it. This setting specifies the number of days to keep UDR data for all the repositories in the console.
# This value should be considerable greater then the delete and compression intervals specified in the manager scripts.
# The primary purpose is to delete nodes that are no longer collected and will never be deleted by any manager runs
# (Default : 0 (Not Deleted))
UDR_GLOBAL_DATA_RETENTION = 30
(8) Run $BEST1_HOME/bgs/bin/managerExceptions.pl at the end of the run to generate the Manager Exception files in the $BEST1_HOME/local/manager/status/MM-DD-YYYY directory.
Related Products:
- TrueSight Capacity Optimization
- BMC Performance Assurance
Legacy ID:KA424847