Start-up of ARServer failing or taking too long |
This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers. Legacy ID:KA342293 Start-up of ARServer failing or taking too long
Since ARS 7.0 , arserver has the ability to log the timing sequence of the various routines required to complete initialization. This document details how to enable ARS Start-Up Logging and some practical examples on how to interpret and leverage this information.
To Enable Start-up Logging: Since it may be helpful to have arserver(d) log various milestones very early in the start-up sequence, this feature requires a parameter to be passed into the actual arserver(d) invocation command line. Add the "-t" parameter before any other parameter.
UNIX example: - You need to alter armonitor.conf. This is usually located in /etc/arsystem/<arservername>. - Be sure save a copy of the original before making any modifications.
In the following output, note "-t" parameter just after "arserverd, and just before "-s", with spaces in between:
# diff armonitor.conf.0604 armonitor.conf 4c4 < /db/AR/701itsm/bin/arserverd -s bobw_701itsm -i /db/AR/701itsm -l /etc/arsystem/bobw_701itsm --- > /db/AR/701itsm/bin/arserverd -t -s bobw_701itsm -i /db/AR/701itsm -l /etc/arsystem/bobw_701itsm
Windows example: - You need to alter armonitor.cfg. This file is usually located in your "<AR_INSTALL>\<SVR_NAME>\conf" Where: AR_INSTALL = where you installed arsystem/arserver.exe SVR_NAME = Your arsystem server name
In the following output, note "-t" parameter just after "arserver.exe, and just before "-i", with spaces in between:
>diff armonitor.cfg.ORIG armonitor.cfg < "...\arserver.exe" -i "C:\Program Files\AR System\..." -l "C:\Program Files\Common Files\AR System\Licenses\..." -m --- > "...\arserver.exe" -t -i "C:\Program Files\AR System\..." -l "C:\Program Files\Common Files\AR System\Licenses\..." -m Name and location of output log file. The log file will have a name like - "arstartup_####.log" Where: #### = Process ID (pid) of the arserver(d) process.
The location of the file will be the working directory of the arserver(d) process at start-up time. On Unix, this is usually the same directory where the arsystem and arserverd files reside. On Windows, it usually the working directory of the Service Manager, which is usually C:\Windows\System32
Examining and leveraging AR Start-Up logging output.
For example, the most common slowdown in the start-up procedure is the loading of the Schema Field Display Properties table. Here's some analysis from an actual customer system.
First line: Tue May 29 2007 19:24:49.8622 <Startup> <TID: 0000000001> Set up thread control block with key = 0
Last line: Tue May 29 2007 19:45:00.0899 <Startup> <TID: 0000000001> svc_run forever
20 minutes, 10 seconds to start-up -- YIKES!!!
Tue May 29 2007 19:25:16.5056 <Startup> <TID: 0000001029> Begin LoadDisplayInfoList
Tue May 29 2007 19:41:45.6155 <Startup> <TID: 0000001029> End LoadDisplayInfoList - Rows = 652893
*16 mins, 29 seconds -- whoa
Tue May 29 2007 19:42:23.0702 <Startup> <TID: 0000001029> Begin LoadFieldPermissions ... Tue May 29 2007 19:44:56.5742 <Startup> <TID: 0000001029> End LoadFieldPermissions - Rows = 348848 *** 2 mins, 33 secs.
Totals: Time to LoadDisplayInfoList 16min, 29 secs Time to LoadFieldPermissions 2 min, 33 secs (989 + 153 ) / 1210 secs = 94% of startup time spent in these loads
*note: The performance of these SELECT statements rely heavily on the configured database's ability to retrieve and transmit these rows to the arserver(d) process. Experience in BMC has shown this mostly to be an issue with the fetching and transferring of large amounts of data. In most cases, arserver(d) is just waiting for the SELECT statement to complete.
2. Another, more urgent, usage of AR Start-up logging would be in the event the start-up sequence failed for some reason. Although the logging doesn't really show error conditions, you could compare start-up sequence log outputs of failed start-ups with start-up logging taking before any start-up problem may have occurred. The point in the log output that no longer appears will be the point that requires investigation.
Notes: 1. These log files have no maximum size limit, and never get automatically cleaned up. You will have to remove these files manually or have it done in a script. Related Products:
|