Sample 'top' output showing the ISN process:
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16260368 total, 7514920 free, 3650064 used, 5095384 buff/cache
KiB Swap: 8388604 total, 8388604 free, 0 used. 11698492 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5111 root 20 0 25.556g 913172 18612 S 0.0 5.6 23:52.00 java
Be sure that the MaxHeap is set appropriately. For example, in a Large configuration, the value should be 4096m.
Ensure that the ISN doesn't exceed the RAM on the OS. For example, if there are 3 ISNs on the box, each with a MaxHeap of 4096m and there is only 8Gb of RAM on the box, this can lead to a problem.
If the Virtual Memory goes beyond 15Gb for one or more of the ISN processes (after checking the MaxHeap/RAM value as expressed above), then do the following:
By default, jmap will not be installed on the system, as confirmed by 'which jmap'. If you don't have it, install it. For example:
yum install java-1.8.0-openjdk-devel-220.127.116.11-0.b17.el6_7.x86_64
Now you should have it in /usr/bin/jmap
Generate a heap dump:
Get the PID of one of your ISNs.
(PID 5111 is used in the example below; use yours instead)
Create an hprof from the PID:
jmap -dump:format=b,file=/data1/ISN1.hprof 5111
Dumping heap to /data1/ISN1.hprof ...
Heap dump file created
ls -l /data1
-rw------- 1 root root 127640497 Dec 13 09:55 ISN1.hprof
You can write it to the directory of your choosing. I've placed mine in /data1 in the example above.
Compress the file & send it to BMC for analysis.
Also send the following from the ISN: