This can happen if performance data cannot be fetched from the vCenter or if the performance extraction query times out. Simply connecting to the vCenter URL isn't a sufficient test. Instead, use the vSphere client to validate that charts at the right resolutions can be generated for the problem systems.
In an healty vCenter environment clusters have metrics, which are summarized starting from the hosts and machines below. If this doesn't happen it means that the summarization jobs into the vCenter database are failing or are not scheduled, or the VMware vCenter database needs to be rebuilt by VMware Technical Support, since there are tables which have "degraded performance" (scattered lines, non updated stats...) after long periods of activity.
The vCenter Service Extractor extracts the real-time performance metrics for VMware Hosts and Guests and hosts and extracts the summarized metrics for Clusters and Pools.
Following the steps in VMWARE ETL missing data statistics metrics, one should be able to:
- connect to the vcenter with the VI client
- try to chart data at live resolution for warned systems
- see if you can spot data holes, or require much time to load the charts
Here is the VMware KB for Troubleshooting gaps in performance data or missing performance data in vCenter Server: VMware KB link . Please provide this link to your Vcenter admin, following these steps he will be able to fix the issue quickly
Debugging vCenter data collection problems using the vSphere PowerCLI
Downloading, installing, and configuring the vSphere PowerCLI
The attached vSpherePowerShellScripts.zip file contains two scripts that can be used to check vCenter functionality and VMware API response times outside of CO which is useful to identify vCenter performance issues.
The vSphere PowerCLI must be run on a Microsoft Windows machine (so it cannot be implemented directly on the CO Application Server or ETL Engine and to be installed requires that you have full Administrator rights to the machine. Additionally the machine where the vSphere PowerCLI is being run must have access to vCenter on port 443 (HTTPS). One can test if the vCenter is reachable by attempting to connect to https://yourvcenterip/ via a web browser. A successful connection will generally return a VMware web page and a failure will generally return although a 404: Not Found error is also acceptable (that would mean that you can connect to the port but the user administration pages aren't enabled on that vCenter).
vSphere PowerCLI notes:
- The vSphere PowerCLI requires that the Windows Powershell be installed (it is typically available by default on Windows 2008 and later)
- The vSphere PowerCLI requires administrator access to initially configure (although then anyone can use it) and it requires a Powershell security check to be disabled
- The vSphere PowerCLI can be downloaded from VMware's web site (and requires a VMware [free] account be created to download it)
(1) If the vSphere PowerCLI is not already installed, download it from VMware. For instrucitons on the vSphere PowerCLI setup & settings refer to http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-and-vsphere-articles/general/getting-started-with-vmware-powercli.html).
Download link: https://my.vmware.com/web/vmware/details?productId=285&downloadGroup=VSP510-PCLI-510
NOTE: A VMware account is required to download the software but one can be created on the VMware site for free.
2a. Install the PowerShell CLI (Create a desktop shortcut).
2b. Using an account with Administrator privleges on the machine, enable the execution of scripts by running the following command at the PowerCLI prompt (Run the 'powershell' command which will launch the Windows PowerShell. The command prompt of the PowerShell beings with 'PS':
Set-ExecutionPolicy RemoteSigned
3. Once the VMware PowerCLI is installed it should be possible to run the VMware Powershell (not regular Windows Powershell) from the Start menu (All Programs -> VMware -> VMware vSphere PowerCLI -> VMware vSphere PowerCLI)
Running the vSphere PowerCLI debugging scripts
--------------------------------
If your errors are like:
---------------------------------
BCO_ETL_FAIL_306: Unable to query performance data for compute resource XXXXXXXX
execute the following steps:
1. Edit the runCollection_clusterpool.ps1 file from the vSpherePowerShellScripts.zip and specify the same credentials (user and password) being used by the vCenter ETL in CO.. I set the cluster to try to collect from to XXXXXXXX as in the logs.
So, in the script update the following:
connect-viserver -server bcm-vcenter -User Administrator -Password SecretPwd
$Cluster="BE_CCL_TDA_MA"
Fields to update:
- Change 'bcm-vcenter' to your vCenter hostname
- Change 'Administrator' to the vCenter user account that you have specified for the VMware ETL in CO
- Change 'SecretPwd' to the password for that vCenter user account
- Change 'BE_CCL_TDA_MA' to the Cluster that is generating collection errors in the logs
2. start the PowerCLI and run the runCollection_clusterpool.ps1 file you just modified (it will create an output folder containing collected performance data, and logging response times/timeouts)
--------------------------------
If your errors are like:
---------------------------------
BCO_ETL_FAIL_306: Fail to get performance data for host and child vm's
BCO_ETL_FAIL_306: Fail to query performance data for host and vm
execute the following steps:
1. Edit the runCollection_hostvm.ps1 file from the vSpherePowerShellScripts.zip, and specify the same credentials (user and password) being used by the vCenter ETL in CO. Also, specify the VMware hostname to attempt to collect data for based upon the host that is generating errors in the vCenter ETL logs.
So, in this script update the following:
connect-viserver -server bcm-vcenter -User Administrator -Password SecretPwd
$HostName="vhost1.bmc.com"
Fields to update:
- Change 'bcm-vcenter' to your vCenter hostname
- Change 'Administrator' to the vCenter user account that you have specified for the VMware ETL in CO
- Change 'SecretPwd' to the password for that vCenter user account
- Change 'vhost1.bmc.com' to the VMware host that is generating collection errors in the logs
2. Start the PowerCLI and run the runCollection_hostvm.ps1 file you just modified (it will create an output folder containing collected performance data, and logging response times/timeouts)
3.Regardless of the test script you used the generated output files will include Vmware APi response times, whenever the response time goes over 5 mins, the ETL would have triggered the error/warnings above. Provide the generated output files to BMC support for validation/checking of the output.
