Knowledge Article

Article Number

000224393

Old Article Number

000112536

Article Type

Solutions to a Product Problem

Title

TrueSight Capacity Optimization (TSCO) : vCenter History Extractor always ending in FAILED status with error referencing "This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'"

Summary

Error collecting data for clusters and resource pools or data non collected for these types of entities

Product

TrueSight Capacity Optimization

Component

Capacity Optimization 11.5.01

Applies to

TrueSight Capacity Optimization (all supported versions )
Capacity Optimization (all supported versions )

Problem

The following errors are being reported in the vCenter History Extractor ETL log and the vCenter Historic Recovery ETL is always ending in FAILED status:

- VMware version 6:

FAILED Blocking exception raised by vCenter for entity:domain-c7. Exception: com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'. Contact your system administrator. Please see the server log to find more detail regarding exact cause of the failure.
     at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)

- VMware version 5.5U3:

exception raised by vCenter for entity:domain-c320. Exception: com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: A specified parameter was not correct.FAILED Blocking 
querySpec.size Please see the server log to find more detail regarding exact cause of the failure

- VMware vCenter Server 5.5.0 build-4180647 : 5.0 - api 2.5:

FAILED BCO_ETL_ERR011: Detected an abnormal ETL termination. Reason: com.sun.xml.internal.ws.encoding.soap.DeserializationException: [failed to localize] Failed to deserialize the response.(javax.xml.bind.UnmarshalException
 - with linked exception:
[javax.xml.stream.XMLStreamException: ParseError at [row,col]:[7,11111]
Message: Connection reset])..
.. om.sun.xml.internal.ws.encoding.soap.DeserializationException: [failed to localize] Failed to deserialize the response.(javax.xml.bind.UnmarshalException
 - with linked exception:

- Another symptom one may notice is that a small number of clusters and resource pools that are collected by the vCenter Extractor Service aren't being extracted by the the VMware History Extractor ETL in TrueSight Capacity Optimization (TSCO).

Cause

In vCenter version 5.5 build 3,6.x & 7 vSphere has a new behavior for some APIs that are subject to a limit the number of metrics extracted for each API call.

Solution

- Summary

If you are running the VMWare History Extractor for historical recovery purposes:

Can continue to use it, taking into account that the end status will always be "failed"
Can safely ignore the failure of the etl, which is due and limited to the above cluster and resource pool metrics extraction only
All the other metrics will be extracted by the ETL anyway, so the recovery of all the other metrics from Clusters, Resource Pools, VMs, Virtual Hosts, and Datastores will take place according to the vCenter data retention settings

- Introduction

In vCenter version 5.5 build 3 and 6.x & 7, vSphere has a new behavior for some APIs that are subject to a limit the number of metrics extracted for each API call.
See Support for vSphere 5.5 build and vSphere 6 in CO integration for more details

- Problem details

vCenter itself has issues when collecting the following metrics:

For clusters
- Memory Utilization
- Memory Overhead
For resource pools
- Memory Overhead

These issues impact the collection of all the metrics performed by the ETL (for clusters and resource pools)

- Effect on TSCO

Effect on data collection

No data collected for some clusters and resource pools. All the metrics are not collected, not only the ones in the list

Error in the log

These are some example of errors in the log

- VMware version 6:

FAILED Blocking exception raised by vCenter for entity:domain-c7. Exception: com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'. Contact your system administrator. Please see the server log to find more detail regarding exact cause of the failure.
     at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)

- VMware version 5.5 U3:

FAILED Blocking exception raised by vCenter for entity:domain-c320. Exception: com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: A specified parameter was not correct. querySpec.size Please see the server log to find more detail regarding exact cause of the failure


- VMware vCenter Server 5.5.0 build-4180647 : 5.0 - api 2.5:

FFAILED BCO_ETL_ERR011: Detected an abnormal ETL termination. Reason: com.sun.xml.internal.ws.encoding.soap.DeserializationException: [failed to localize] Failed to deserialize the response.(javax.xml.bind.UnmarshalException
 - with linked exception:
[javax.xml.stream.XMLStreamException: ParseError at [row,col]:[7,11111]
Message: Connection reset])..
.. om.sun.xml.internal.ws.encoding.soap.DeserializationException: [failed to localize] Failed to deserialize the response.(javax.xml.bind.UnmarshalException
 - with linked exception:


- TSCO 10.0 and 9.5 (only a warning is reported in the log and only if the ETL is set at log level 5)

AxisFault

faultCode: ServerFaultCode
faultSubcode:
faultString: A specified parameter was not correct.
querySpec.size
faultActor:
faultNode:
faultDetail:
{urn:vim25}InvalidArgumentFault:<invalidProperty>querySpec.size</invalidProperty>

- Affected TSCO versions

In 9.5 and 10.0, this affects both vCenter Service Extractor and History Extractor ETL
In 10.3 and higher, this affects only History Extractor ETL (Service Extractor is going after RAW APIs, not affected)

- How to confirm on the problem on vCenter side

(A) Look at vCenter log

In the vpxd.log (log of the vCenter) there is this error at same timestamp of ETL error

2016-04-11T14:42:49.255+01:00 [11848 info 'Default' opID=6a2972ec] [VpxLRO] -- ERROR task-internal-11786123 -- -- vim.PerformanceManager.queryStats: vmodl.fault.InvalidArgument:
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    dynamicType = <unset>,
-->    faultCause = (vmodl.MethodFault) null,
-->    invalidProperty = "querySpec.size",
-->    msg = "",
--> }
--> Args:
-->

(B) Check metrics with vSphere client

You have to check these metrics in one of the cluster that is not reporting data into TSCO. Use the same resolution configured into the ETL (e.g. past week if the ETL is configured to import data at 30 minutes of resolution). What is expected if that you will not get any charts

User-added image

(C) Use PowerCLI

Following article BCO_ETL_FAIL_306: Fail to get performance data error/warnings from VMware Service collector ETL, you can run the script attached here (use the script of this article, not on the one of the link)

If the test via the PowerCLI script generates the same error reported by the ETL, it will confirm the problem exists on the the vCenter side.

- Solution

You can use one of these solution:

For daily data collection, if you are on TSCO version 10.3 or newer and are using History Extractor for daily collection, switch to the vCenter Service Extractor
For historic data recovery, ignore the 'vpxd.stats.maxQueryMetrics' message and the ending FAILED ETL status
Ask to VMware administrator to follow this article https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2107096

This behavior is being tracked by Improvement DRCOZ-28000: 'VMware history Extractor ETL always ends in failed status' and is being considered by Product Management for a future release of TrueSIght Capacity Optimization.

Attachment(s):

runCollection_cluster_queryLimits.ps1

Manage Support IDs

BMC Contact Options