My BMC Atrium Orchestrator (BAO) / TrueSight Orchestration (TSO) environment crashes with the error message "java.lang.OutOfMemoryError: unable to create new native thread" in the logs. What causes this error, and how can I prevent it?
This error has two primary causes:
1. There is no room in peer memory to accommodate new threads.
2. The number of threads exceeds the operating system limit.
TSO/BAO writes this error when all available thread space has been claimed, such that new threads fail during their creation process. You can resolve this issue by following the below steps in order:
Approach 1:
Restart your peer. This will clear the current threads and create a new space, though you may only resolve the issue temporarily.
Modify TSO/BAO to change the amount of available thread space. You must add the setting "-Xss128k" to the peers Java settings; the exact steps vary based on your operating system:
Unix:
Edit the file [TSO_HOME]/bin/server.sh and add the setting to the variable JAVA_OPTS Example: JAVA_OPTS="-Dorg.apache.tomcat.util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE=true -server -Xms1024m -Xmx2048m -Xss128k -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath='$CATALINA_HOME/logs' -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -Dorg.apache.activemq.SERIALIZABLE_PACKAGES=com.realops,java.net,java.util,java.lang,org.jdom,com.bmc,javax.security,org.springframework,sun.util.calendar -Datsso.share.cookies=false -Dao.home='$AO_HOME'"
Restart your TSO peer.
Windows:
Run the below command, substituting your TSO installation directory and the name of your BMC service:
7.8.00 and 7.8.01: "$BAO_HOME\tomcat\bin\tomcat7w.exe //ES/BAO-CDP",
Click on the "Java" and add the -Xss setting to the "Java Options" text box.
Example:
Restart your TSO peer.
Note:If you do not know your TSO service name, open the Services pane by pressing Ctrl+R and running "services.msc". Right-click the TSO peer service and select "Properties". The Properties panel shows the service name.
Approach 2: Follow if the issue is not resolved after implementing approach 1
Install any patches available for your version of BAO/TSO. You may be seeing a defect that causes a memory leak, locked thread, or another issue, and the most recent patches contain the latest updates for peer memory management. You can find more information in the release notes for each version and patch.
Approach 3: Follow if issue is not resolved even after implementing approach 1 and 2 (Unix Only)
Contact your Unix administrator about increasing the server maximums for user processes and open file handles. You can check the current limits with the command “ulimit-a”. High-load environments, or environments that run other programs, may need higher limits than medium-load or dedicated systems.
Please see video resolving the problem on Unix & Windows