Typically when implement a vCenter Extractor Service data becomes visible in the workspace domain within 1 to 2 hours of the service startup, running a vCenter Extractor Service and it has been running for several hours (over 3) and the workspace domain still hasn't been updated. How can one identify whether there is a problem with the ETL, Datahub's Data API, or something else? |
(1) Check the vCenter Extractor Service service##.log files in the $CPITBASE/scheduler/log directory on the ETL Engine Server. Look to see if there is a successful LOAD event being reported. For example, here are messages related to a successful vCenter Extractor Service LOAD event: 2016-03-22 16:25:09,550 INFO [service-64-1]- Processing table SYSDAT 2016-03-22 16:25:09,550 INFO [service-64-1]- [ServiceDataSaver] Creating LOADER of type com.neptuny.cpit.etl.loader.CSVL - loading dataset: SYSDAT 2016-03-22 16:25:09,550 INFO [service-64-1]- [ServiceDataSaver] ETL Engine: LOAD step number 1 2016-03-22 16:25:09,550 INFO [service-64-1]- Loader: Dataset size is 101118 2016-03-22 16:25:09,550 INFO [service-64-1]- Loader: default rows limit = 1000 2016-03-22 16:25:09,904 INFO [service-64-1]- Loader: CSV rows limit has been reached 2016-03-22 16:25:09,904 INFO [service-64-1]- Data loaded to file /opt/bmc/BCO/etl/output/SYSDATQ3MG00764 2016-03-22 16:25:09,904 OUTPUT [service-64-1]- [CSVL] /opt/bmc/BCO/etl/output/SYSDATQ3MG00764 2016-03-22 16:25:09,905 INFO [service-64-1]- [ServiceDataSaver] Creating LOADER of type com.neptuny.cpit.etl.loader.SeriesMessageL - loading dataset: SYSDAT 2016-03-22 16:25:09,905 INFO [service-64-1]- [ServiceDataSaver] ETL Engine: LOAD step number 2 2016-03-22 16:25:09,905 INFO [service-64-1]- Splitting output xml file every 10000 lines 2016-03-22 16:25:09,905 INFO [service-64-1]- Configured metric profile id: 0 with level: STANDARD 2016-03-22 16:25:09,905 INFO [service-64-1]- Enabling lookup filtering on entities: 0;25;31;38;43;44;45;46;47;115;12;16;17 2016-03-22 16:25:09,906 INFO [service-64-1]- [Series Message Loader] Processing dataset SYSDAT. Dataset size is 101118 2016-03-22 16:25:21,874 INFO [service-64-1]- [Series Message Loader] Discarded 54174 rows. 2016-03-22 16:25:21,874 INFO [service-64-1]- [Series Message Loader] Loaded 5 messages. 2016-03-22 16:25:21,874 OUTPUT [service-64-1]- [Series Message Loader] /opt/bmc/BCO/etl/output/SYSDATQ3MG00764_10_1.xml;/opt/bmc/BCO/etl/output/SYSDATQ3MG00764_10_2.xml;/opt/bmc/BCO/etl/output/SYSDATQ3MG00764_10_3.xml;/opt/bmc/BCO/etl/output/SYSDATQ3MG00764_10_4.xml;/opt/bmc/BCO/etl/output/SYSDATQ3MG00764_10_5.xml 2016-03-22 16:25:22,406 INFO [service-64-1]- [ServiceDataSaver] Load executed correctly 2016-03-22 16:25:23,295 INFO [service-64-1]- ... end saver [64] 2016-03-22 16:30:00,001 INFO [service-64-2]- Starting service [64] - execTimeDelay: 0ms (2) Next on the Datahub Application Server side, check the see that the Data Service is processing data) by reviewing the $CPITBASE/datahub/log/data.log file. It isn't possible to directly map the ETL LOAD XML file name to the input file read by the Data Service (since the file that gets loaded is *.gz file) but any activity within the Data Service logs would be an indication that the service is running properly. One thing that can be validated is that data from this ETL is being processed by the Data Service. First, check the 'Source ID' of the ETL (under Administration -> ETL & System Tasks -> ETL Tasks -> Select the ETL -> Point at the picture of the gear at the top level. There will be a hoverover that says ETL: taskid: 64 ; srcid: ##.) - In the following sample output we can see that data for ETL Source ID 10 is being imported by the Data Service: 2016-03-22 06:28:19,025 INFO - [Lookup cache] - Loading lookup cache for srcid: 10 - shared catalog: 2 - entFilter: 0;25;31;38;43;44;45;46;47;115;12;16;17 2016-03-22 06:28:22,839 INFO - [Lookup cache] - Lookup loaded correctly for srcid: 10 Shared:2 - cache objects [sys]: 67 [wkld]:27 - Tue Mar 22 06:28:22 EDT 2016 2016-03-22 06:28:22,840 REPORT - Inserting new SYS entity [SRCID 10] [HOSTNAME#HOSTNMAE.com##NAME#hostname##PARENT_CLUSTERNAME#CLustername##PARENT_HOSTNAME#hostname.domain.com##PARENT_VCNAME#Wvcname##PARENT_VCUUID#1234-4567-3456-B123-ABCGD##UUID#1234-5467-1b3e-c0f0-abcdef##VMW_VMREF#vm-12345##_COMPATIBILITY_#422edf09-7177-2b3e-c0f0-7cd408865698] [hostname] [gm:vmw] [LKTYPE:1] 2016-03-22 06:28:23,462 INFO - Updating type for entity SYS:21640 to [gm:vmw] - was [null] 2016-03-22 06:28:23,934 REPORT - Inserting new SYS entity [SRCID 10] [HOSTNAME#hostname.domain.comt##NAME#hostname##PARENT_CLUSTERNAME#cluster##PARENT_HOSTNAME#parenthostname.domain.com##PARENT_VCNAME#vchostname##PARENT_VCUUID#2BD3CA02-A836-43E5-B553-DEADBEEF##UUID#564d972a-7591-4221-b5d3-deadbeef##VMW_VMREF#vm-12345##_COMPATIBILITY_#564d972a-7591-4221-b5d3-deadbeef] [hostname] [gm:vmw] [LKTYPE:1] - Note that there will be a delay when importing vCenter data from a Remote ETL Engine into CO because the data import becomes a 3 step process: (1) The vCenter Extractor Service needs to execute the LOAD phase which will load the data to the Datahub's Data API (2) The Datahub's Data API needs to take the input file from the LOAD and insert them into the Data Staging Area within the database (3) The Datahub needs to read the data from the Staging Area, process it, and import it into the Data tables (3) One possibility is that the Hierarchy Rule associated with the ETL hasn't run yet. That would explain the Entities being imported into CO but not showing up in the expected location within the hierarchy. One option would be to wait until tomorrow morning see if the entities are still in the Unassigned domain. If they are we can find the Hierarchy Rule under Administration -> Data Warehouse -> Hierarchy Rules and then check the hierarchy rule log to see what messages have been generated in relation to that hierarchy rule. - One can check the ETL logs to see if it looks like the Object Relationship data is being imported: service64.log.2016-03-22-12:2016-03-22 12:25:46,292 INFO [service-64-2]- Processing table OBJREL service64.log.2016-03-22-12:2016-03-22 12:25:46,292 INFO [service-64-2]- [ServiceDataSaver] Creating LOADER of type com.neptuny.cpit.etl.loader.CSVL - loading dataset: OBJREL service64.log.2016-03-22-12:2016-03-22 12:25:47,411 INFO [service-64-2]- Data loaded to file /opt/bmc/BCO/etl/output/OBJRELQ3MC15964 service64.log.2016-03-22-12:2016-03-22 12:25:47,411 OUTPUT [service-64-2]- [CSVL] /opt/bmc/BCO/etl/output/OBJRELQ3MC15964 service64.log.2016-03-22-12:2016-03-22 12:25:47,411 INFO [service-64-2]- [ServiceDataSaver] Creating LOADER of type com.neptuny.cpit.etl.loader.SeriesMessageL - loading dataset: OBJREL service64.log.2016-03-22-12:2016-03-22 12:25:47,413 INFO [service-64-2]- [Series Message Loader] Processing dataset OBJREL. Dataset size is 6716The delay might be that the ETL can't run the Hierarchy Manager following the data import because at that point the data might not be there yet (since it could be queued at the Data API level). That would mean that for a Remote ETL there would be a delay until the nightly execution of the Hierarchy Manager before the initial hierarchy was created. |