BMC Atrium Orchestrator / TrueSight Orchestration (TSO) does not appear to be stable. Grid Manager navigation does not function correctly, and peers fail to connect to each other. The most common error is in the grid.logs is this:

01 Jan 2019 01:01:01,001 WARN HANode : Failed to obtain high availability node's lock within the configured timeout of 20000 ms while processing a heartbeat message from another high availability node. Cancelling this activity.
01 Jan 2019 01:01:01,001 WARN HANode : Lock owner's stack trace:
  sun.misc.Unsafe.park(Native Method)
  java.util.concurrent.locks.LockSupport.park(Unknown Source)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
  java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
  java.util.concurrent.FutureTask.get(Unknown Source)
  com.realops.foundation.gridframework.HAHub.onBecomingSlave(HAHub.java:6742)
  com.realops.foundation.gridframework.HAHub.stateChanging(HAHub.java:6159)
  com.realops.foundation.gridframework.HANode.setState(HANode.java:627)
  com.realops.foundation.gridframework.HANode.goToSlaveState(HANode.java:1004)
  com.realops.foundation.gridframework.HANode.access$700(HANode.java:34)
  com.realops.foundation.gridframework.HANode$3.call(HANode.java:319)
  com.realops.foundation.gridframework.HANode$3.call(HANode.java:301)
  com.realops.foundation.gridframework.HANode.doWithLock(HANode.java:1313)
  com.realops.foundation.gridframework.HANode.heartbeat(HANode.java:301)
  com.realops.foundation.gridframework.HAHub.heartbeat(HAHub.java:99)
  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  java.lang.reflect.Method.invoke(Unknown Source)
  com.realops.foundation.gridframework.TopologyParticipant.invoke(TopologyParticipant.java:2169)
  com.realops.foundation.gridframework.TopologyParticipant.processInboundMessage(TopologyParticipant.java:1592)
  com.realops.foundation.gridframework.TopologyParticipant$MyReceiver.processMessage(TopologyParticipant.java:409)
  com.realops.foundation.gridframework.StatsRecorderDecorator$2.processMessage(StatsRecorderDecorator.java:97)
  com.realops.foundation.gridframework.DefaultMessageDispatcher$1DispatcherTask.run(DefaultMessageDispatcher.java:124)
  java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  java.lang.Thread.run(Unknown Source)

If the peers cannot communicate, they cannot change the state of adapters because a peer cannot enable an adapter without first notifying all other peers. A peer may also try to reach other peers to determine the current job list. If it fails, the peer assumes the other peer is offline and elects itself as the peer in charge of managing jobs, which means that the peers are no longer coordinating job management. 

If lowering the grid workload or increasing network speed is not possible, you can use the tuning-config.xml file to make the TrueSight Orchestration grid handle peer-to-peer messages in a more robust manner. This can mitigate problems with network connectivity or throughput. For more information, please see the BMC Knowledge Base or contact BMC TrueSight Orchestration Customer Support.