How do I troubleshoot a BMC Helix Control-M Agent connection when a Proxy server is in the middle? |
Starting with BMC Helix Control-M Agent version 9.0.21.080, 'agproxy.json' can be updated with a current proxy configuration as done on install, by running 'ag_saas_set_config.sh proxy'. Note: Please whitelist below URLs before proceeding. *.controlm.com *.amazonaws.com Troubleshooting Proxy connection Internet connection If proxy is not configured properly, you probably won't be able to send any https request (same as if network is not opened). This can be checked with any curl/wget as to google.com, which should probably block and return after some time with error, as in below examples. curl https://google.com curl: (7) Failed to connect to ... Network is unreachable curl --trace - https://google.com == Info: About to connect() to google.com port 443 (#0) == Info: Trying 142.250.217.78... == Info: Connection timed out == Info: Trying 2607:f8b0:400a:801::200e... == Info: Failed to connect to 2607:f8b0:400a:801::200e: Network is unreachable == Info: Failed connect to google.com:443; Network is unreachable == Info: Closing connection 0 curl: (7) Failed to connect to 2607:f8b0:400a:801::200e: Network is unreachable wget https://google.com Connecting to google.com (google.com)|142.251.33.110|:443... failed: Connection timed out. ... failed: Network is unreachable. OK response when proxy is configured properly should be as: curl --trace - https://google.com == Info: About to connect() to proxy bmc-devops-proxy-server-r53.ci.ctmsaas.com port 3128 (#0) == Info: Trying 10.20.5.193... == Info: Connected to bmc-devops-proxy-server-r53.ci.ctmsaas.com (10.20.5.193) port 3128 (#0) == Info: Establish HTTP proxy tunnel to google.com:443 => Send header, 112 bytes (0x70) ... == Info: Proxy replied OK to CONNECT request … Check using curl/wget that the network is opened, if fail to connect and proxy is used on the machine verify proxy is configured properly for the agent user. Note that the customer should be responsible for proper proxy configuration on the machine, yet need to be aware of the outcome if not done properly. In Windows, if curl/wget is not available, check access to URL's using the Internet browser or Firefox, you’ll probably get after some time message as: “The connection has timed out”. Check also curl/wget with https://static-endpoints.prod.controlm.com, which is the first https used on SaaS agent install, if fails install will not succeed. Other https url endpoints used by the agent can be found (after install) in ctm/data/SAASCONF.dat (as register: SAAS_API_GTW_ENDPOIN, iot:SAAS_IOT_GET_TEMP_CRED_URL). curl/wget for them will probably fail due to missing credentials (as in below) which is accepted, yet will not block and return with timeout due to not opened network or not properly configured proxy. {"message":"Missing Authentication Token"} curl: (58) NSS: client certificate not found (nickname not specified) <Error><Code>AccessDenied</Code><Message>Access Denied</Message>…</Error> Note: we also seen with specific customer using Cisco Umbrella as a proxy, below 403 Forbidden error due to not opened network as requested for amazonaws.com: <head><title>403 Forbidden</title></head><body><center><h1>403 Forbidden</h1></center><hr><center>Umbrella Cloud Security Gateway</center></body></html> Installation failure Note: If proxy in use also requires some customer's self-signed certificate, the install will probably fail with PKIX error as: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" curl/wget could work OK if the customer’s self-signed certificate is updated to the machine certificates used by curl, yet https will fail on agent install or agent run since java ca-certificates are used. More details can be found in KA 000411157 If installation fails, it is important to send us the ‘ctm_ag_saas_cmd.log’, as detailed in below install error message: Linux: “More details in ‘ctm_ag_saas_cmd.log‘ under Control-M/Agent proclog directory or after rollback in $HOME/BMCINSTALL/log” Windows: “More details in ‘ctm_ag_saas_cmd.log‘ under Control-M/Agent proclog directory or after rollback in %temp%” If installation failed, the Agent’s proxy settings file - ctm/data/agproxy.json is not created, Yet you can still check which proxy settings were used by install in the SaaS install log - proclog/ctm_ag_saas_cmd.log. If proxy settings were detect expect log lines as: 0622 14:08:45.551 INFO main c.b.c.a.t.p.ProxySettingsHandler - *Set java proxy-settings with: http.proxyHost=your-proxy-server.com , http.proxyPort=3128 0622 14:08:45.551 INFO main c.b.c.a.t.p.ProxySettingsHandler - *Set java proxy-settings with: https.proxyHost=your-proxy-server.com , https.proxyPort=3128 If proxy settings were not detect expect log lines as: 0622 11:10:02.435 INFO main c.b.c.a.t.p.ProxySettingsHandler - getProxySettingsAndSetJavaProp: No proxy-setting found so none defined for java! Other agent logs SaaS agent install log - proclog/ctm_ag_saas_cmd.log SaaS agent main log - proclog/ctmagj_*.log - errors details can be found here if install done ok yet agent connectivity fails (ag_ping / ag_diag_comm) SaaS agent log with configured proxy details - proclog/agjstd_*.log |