TrueSight.log shows following error in logs. When the Process memory usage was checked for the CSR process, it is normal, the OS also has free memory available. This is not because of heap space.
 
[2018-04-18T01:10:31,704][INFO ][o.e.m.j.JvmGcMonitorService] [Niv1l2z] [gc][53398] overhead, spent [305ms] collecting in the last [1s]
[2018-04-18T03:30:07,484][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[Niv1l2z][refresh][T#2]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
                at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
                at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
                at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_131]
                at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_131]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_131]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
 
Problem solution
This happens because of constraint on processes. Following setting helps to increase the limit.
 

1. Get the CSR and index server process ID

tssh  server status

1. ps –p processID –lfT |wc –l

The command above should show the threads usage. For CSR it is around 2000 and for indexserver it is around 200.

1. ulimit –a

Look for max user processes (Default 2048)
Make it unlimited or higher value of 4096.
ulimit –u unlimited

1. Check the init files if this value is being changed.
2. Reboot the server.
3. Test ulimit –a