How to prevent failures when upgrading AtriumCore from 7.5 to 8.1 and 9.1.

Knowledge Article

Article Number

000338377

Old Article Number

000097520

Article Type

Solutions to a Product Problem

Title

How to prevent failures when upgrading AtriumCore from 7.5 to 8.1 and 9.1.

Summary

Upgrading AtriumCore from 7.5 to 8.1 and 9.1 - What you should know to prevent upgrade failures

Product

BMC Atrium Core - CMDB

Component

BMC Atrium Core

Applies to

BMC Atrium Core - CMDB 7604, 8.1, 9.0 through 9.1.04

Problem

Upgrading from versions prior to the 7.6.04 release can be tricky, but fairly simple to acheive if you know the steps covered below.

There are various KA's on this topic already. This KA rolls up all the known causes into one article.

Starting point : BMC AtriumCore 7.5 CMDB, AIS, Product Catalog and plugins.

Finish: BMC AtriumCore 9.1.04

NOTE: This KA does not take AtriumCore WSC into account because we've changed the Tibco WSC to Axis 2 with Java UDDI (JUDDI). This change alone virtually removed all install issues of the Atrium Core Web Services.

Cause

Upgrade can fail when the application isn't in good condition before upgrading it.

Solution

This knowledge article may contain information that does not apply to version 21.05 or later which runs in a container environment. Please refer to Article Number 000385088 for more information about troubleshooting BMC products in containers.

Legacy ID:KA406262

The following areas are known to cause issues at some time during the installer:

1. Prechecker Utility of the CMDB Installer
2. Disable Escalations
3. Best Practice Conversion Utility - BPCU
4. Custom Workflow and Indexes for cmdb fields
5. Attachment Size Limit
6. Warning suppression
7. SIM Extensions out of synch with ARS Metadata
8. 32 bit CMDB vs. 64 bit ARS Server
9. Keeping Windows Explorer opened in the AtriumCore folder during the upgrade
10. Expected component services not running/started during the upgrade
11. Invalid "Midtier" password that causes "AtriumCore console flyout error" when accessing it.
12. ZDT (Zero Down Time) Roll back (9104 exclusive)
13. Atrium Web Services fails because the Tomcat could not be started during upgrade.

1. Prechecker Utility of the CMDB Installer

Part of the installation includes a CMDB Pre-Checker that validates field ids and names. This prechecker should be only executed if the version of CMDB is already 7604 and not executed if the version is prior to 7603 SP1, however it is part of the installer and therefore it will run anyway and add ERRORs to the install log. This is OK and can be ignored.

Here is some background on this. If you run the prechecker before 7603 SP1 then the prechecker will have some failure exit codes that may also show up in the arerror.log. These errors are there because there are Field IDs for Company that get checked in the following forms:

PCT:VersionCompanyAssocStatusFlags
PCT:PatchCompanyAssocStatusFlags

The installer needs to change these Field IDs from value "420000165" to this value "1000000001". Since this is part of the installer and the prechecker runs before that can happen and hence these errors are ignored.

The log will show an exit code of 1025 after running this command:

LOG EVENT {Description=[AR ChangeID command],Detail=[C:UsersE020156AppDataLocalTempUtilitiesrikarchgid.exe -x ARSALIAS -u Action Request Installer Account -p ****** -t 0 -c 10003 -N 7200 -L 14400 -X 21600 -F C:UsersCURENTUSERAppDataLocalTempUtilitiespcchangeFiledIDs.txt

Since there isn't a way to easily remove the Pre-Checker from the available AtriumCore Installer program then all you need to do is to understand that this is a false positive result.

2. Disable Escalations

Upgrade takes a while to complete and some of the changes during the install impact CMDB data that trigger AR Escalations or qualify escalations to run.
This is can halt a stage from completing and cause a failure of the install. Usually this type of failure is not obvious and causes the installer to fail at different stages.
Please always remember to Disable AR Escalations before the upgrade or install.

3. BPCU (Best Practice Conversion Utility)

BMC has produced a customization conversion utility that converts custom workflow into Overlays. For customers who have Asset Management and other applications where the CMDB is providing data structure service and have customized workflow this utility is best used AFTER the AtriumCore Upgrade is completed. Althoug the BPCU utility does not overlay any CMDB schemas and workflow even if customizations are found, however running this utility sets the expectation that even if the CMDB was overlaid than this would be corrected by the BPCU. This is not true.

Please see https://selfservice.bmc.com/casemgmt/sc_KnowledgeArticle?sfdcid=000096586 for reference.

Bottom line for this point is - run the BPCU after the upgrade of CMDB is completed, and successful and was not impacted by overlay issues. We've done these upgrades now with 100% success rate and are confident to support this practice.

4. Custom Workflow for cmdb fields

This item is related to the previous one, but more specific to data that is being inserted into BMC.CORE CDM forms.

The installer inserts various template data into the BMC.CORE forms and this step can be blocked if additional data restrictions or attribute requirements are enforced by custom workflow. These fields may include MarkAsDeleted, Company, Region, Site and so on. If you have customizations in place that require a value in a field that is not already required out of the box then you should disable the workflow for those fields before you start the installer.

The install log would have captured such failure with exit code of 1025 and log entry would look like this:

C:UsersCurrentUserAppDataLocalTempUtilitiesrikrik.exe" loadapp -x ARSALIAS -t 0 -u "Action Request Installer Account" -p <not_displayed> -l "D:Program FilesBMC SoftwareAtriumCore" -n CMDB-RIK_Install -f "D:Program FilesBMC SoftwareAtriumCorecmdbenworkflowupgrade764patch000wf-RIK-CMDB.xml" -L -C

Where the name of the log that actually captured the real failure codes is "CMDB-RIK_Install.log"

This log would then show an error like this

INFO - Error importing record 1: ERROR (806201): ; Please supply a Category, Type and Item for this Configuration Item. Those fields require and entry to create or modify a CI.
DATA "" "" "" "" "" "" "" "BMC_BUSINESSSERVICE" "0;" "" "" "" "" 1282162748 "BMC.ASSET" "" "" "" "" 0 "" "STANDARD" "BMC_GLOBAL_DEFAULT_SRVC" "" "" "Demo" "" "" "" "" "" "" "" "" "" "" 1285101154 "BMC_GLOBAL_DEFAULT_SRVC" "" 30 "" "" "" "" "" 0 "BMC_GLOBAL_DEFAULT_SRVC" "" "" "" "000000000000011|000000000000383" "" 10 10 "Default Service" "" "" "" "" "" "" "" "" "" "" 0 "1284595003Demo" "Demo" 0 "" "0" "" "" "" "" "" 0
INFO - Import Completed in 8.237 seconds. 0 records were imported to BMC.CORE:BMC_BusinessService; 1 Records were not. From File D:Program FilesBMC SoftwareAtriumCorecmdbenworkflowupgrade764patch000.USM_BusinessService_DefaultData.arx

[ERROR] LoadComponent- Data Import failed with code 1025 for file D:Program FilesBMC SoftwareAtriumCorecmdbenworkflowupgrade764patch000.USM_BusinessService_DefaultData.arx

And there would additional records like this that fail. Exit code of 1025 can almost always be associated with a data collision of some type. Either the record could not be imported, or it could not be deleted (code 1024) for what ever reason.

Please disable workflow that would prevent the installer from managing records in the CMDB. No customer records are deleted during the upgrade, only CMDB template or default loads like the BusinessService data will be impacted.

5. Attach Size Limit

We've found that the ARDBC setting of "Db-Max-Attach-Size: 0" has impact on attachments during upgrades. If you have this setting in the ar.cfg (ar.conf if Unix/Linux) then change the value to 200000000 (~200 MB) or just remove it from the config file altogether.

ARS also has a maximum attachment size limit that can conflict with the Data Visualization Components that need to be loaded into the Data Visualization forms during the install. The size of these attachments varies from few kilobytes to several megabytes.
Rule of thumb is to remove the attachment size restriction for the installer to complete successfully. This means that you should set the size limit to 0 (unlimited).

Set Attachment Size Maximum to 0 for the ARDBC (ARS Data Base Configuration values a.k.a. the ARDBC metadata).

You can also run a test before the upgrade by attaching a sample file that's no more than 100 MB to see if the attachment will work. This setting is in the AR Server Configuration Panel. You can leave it at 0 if it already is set that way.

The following settings in the ar.conf file will imdepe imports of files larger than the set limit:

RPC-Max-Record-Size: 2000000

DB-Max-Attach-Size: 2000000

AR-Max-Attach-Size: 2000000

2000000 is 2 MBs, so that is too small. If you run into attachment size issues like this then you'll need make sure that after the change these size limits are removed.

Remove these while the install activity is taking place or increase it to allow for attachments beyond 100 MB (100000000).

6. Warning Supression (ARS)

There are several Warnings interpreted by the installer as failures. These warning can be ingored by supressing them during the time of the installation.

You can add the following the Warning supressions:

See : https://selfservice.bmc.com/casemgmt/sc_KnowledgeArticle?sfdcid=000039883

Suppress-Warnings: 9936

Additional warnings maybe there already, so just add 9936 to the list for the upgrade to complete successfully. This should no longer be an issue in 8.1 or even 8.0 upgrades, however you'd be better off to see that done to prevent this from being a false positive failure of the install.

Our expectation and coding of the Installer should not have this issue surface at all. However we'd like to mention it here for those customers that are using this KA for upgrading to 7.6.04 SP4 of AtriumCore where this issue has manifested itself.

Please see https://selfservice.bmc.com/casemgmt/sc_KnowledgeArticle?sfdcid=000074621 for additional information.

7. SIM Extensions being out of synch with ARS Metadata

If you installed SIM extensions and the installation did not go succesfully then your CMDB metadata could be out of synch.

Fortunatelly we have a new utility that you can get from the Utilities folder. The cdmchecker. It will check the metadata of CMDB and compare it wiht ARS when you run this utility with the -a option.

Syntax:

cdmchecker -a -u <ARS USER> -p <ARPASSWORD> -s <ARHOST_ALIAS> -t "<ARPORT>" -o "missing_fields.log" -i "BMC_MAINFRAME*OB000F1F66BB3CX44aQQCfftAAMgAA|BMC_PRINTER*OB000F1F66BB3CX44aQQCfftAAMgAA|BMC_FEDERATEDDATAINTERFACE*OB005056BE6784SK1zSAZnC9gwRocG"

The -i option gives a list of known attributes to be out of synch and hence they need to be included in the command arguments exactly as you see it above.

The "missing_fields.log" will then show any discrepancies between CDM and ARS that will need to be resolved before you start the installer.

This utility is available in 8.1 release of the CMDB and can be found in the ATRIUMCORE_HOMEcmdbserverbin folder and also in the %TEMP%Utilitiescmdb folder during installation.

A copy of this binary that can be run on 64 bit Windows has been put here:

ftp://ftp.bmc.com/pub/BSM/AtriumCore/8.1/CMDChecker81.zip

It can be ran from Windows even if the AR Server is on UNIX. However if you need to run it from non-Windows systems then start the installer and let it create the /tmp/Utilities directory. The command will be found in the cmdb folder of that utility. You can then run this command and fix any issues before you continue with the installer.

Using this utility before running the installer is recommended. You may run into a defect Defect SW00465091: Upgrade to 8.1.x fails on Dimensions_pool if the CDM was extended with new class as child of ComputerSystem where install fails while running cdmchecker:

com.bmc.smbu.install.common.rule.engine.CommandExecutionException: com.bmc.smbu.install.common.rule.engine.CommandExecutionException: java.lang.Exception: CDM Checker found missing fields on underlying AR forms. Please contact BMC Support with the file : "<C:?Program Files?BMC Software?AtriumCore?Logs? missing_fields.log>" CDM checker Return code =132 Output message=No response

missing_fields.log shows:

[ERROR] Error during ARGetMultipleFields in FindMissingFields Status: Field does not exist on current form. Details: BMC.CORE:BMC_ComputerSystem : <301185300>

[ERROR] In class BMC.CORE:BMC_ComputerSystem, for attribute Dimensions_pool with attribute id OB000F1F66BB3CX44aQQCfftAAMgAA, could not find corresponding AR field with ID=301185300 on form BMC.CORE:BMC_ComputerSystem.

1. Class: BMC.CORE:BMC_ComputerSystem Attribute: Dimensions_pool

2. Class: PNC.CORE:Custom_Desktop Attribute: Dimensions_pool

However even If you run the command by itself, it will not ignore it if you add it to the ignore list. As shown:

cdmchecker -a -u Demo -p Demo -s MYARSHOSTNAME -t "0" -o "missing_fields2.log" -i "BMC_MAINFRAME*OB000F1F66BB3CX44aQQCfftAAMgAA|BMC_PRINTER*OB000F1F66BB3CX44aQQCfftAAMgAA|BMC_FEDERATEDDATAINTERFACE*OB005056BE6784SK1zSAZnC9gwRocG|BMC_COMPUTERYSTEM*OB000F1F66BB3CX44aQQCfftAAMgAA"

Deleting the record "Dimensions_pool" from the AttributeDefinitions form before proceeding with the Install will work and pass the cdmchecker.

8. 32 bit AtriumCore CMDB vs. 64 bit ARS Server

If you've decided to upgrade to ARS Server 7.6.04 and later and do so on a 64 bit Operating System such as Windows or Linux, then you will need to upgrade to 7604 AtriumCore that is also 64 bit. These modes can not be mixed. The ARS Server will need to remain 32 bit if you want to stay with 32 bit CMDB (i.e. CMDB 7.5 SP5). Ignoring this compatibility requirement will result in Win32 error in the arerror.log while loading cmdbsvr75.dll.

For example you'd see an error in arerror.log immediately after starting the ARS Server:

Thu Feb 27 23:25:02 2014 : Action Request System(R) Server x64 Version 7.6.04 SP5 201308012025
(c) Copyright 1991-2011 BMC Software, Inc.
Thu Feb 27 23:25:02 2014 ProcessMain : Error encountered while loading the shared library (ARERR 9750)
Thu Feb 27 23:25:02 2014 cmdbsvr75.dll: %1 is not a valid Win32 application.

Please make sure you maintain the same bit width when upgrading. This is also why the AtriumCore installer first makes changes to configuration files and shuts down the server to replace the binaries before starting it back up.

9. Keeping Windows Explorer opened in the AtriumCore folder during the upgrade

You should not have any command line interface (CLI) windows/sessions opened or browse any directories within the AtriumCore directories during the upgrade.If you do so during the ugprade you'll see install failures with the following errors: