TrueSight Automation Console (TSAC) fails due to low diskspace, where '/var' is getting 100% full and application is not accessible. |
Docker uses "/var/lib/docker" as the default image installation directory. Inside the directory, docker keeps the json.log file for every container and this log file can sometimes grow huge in size taking the whole "/var" file system. Docker internally uses the json.log files for backward compatibility etc. but for most use cases, there is no need to keep these perpetually increasing log files. There are ways to change this default location to another filesystem but the perpetual growth of the json.log file will cause issues there too. Please follow the below steps to truncate the large JSON log file and enable log rotation and keep the log file size within limit on all the TSAC hosts: 1. Stop TSAC Application and Database by running the stackmanager command: .../bmc/stackmanager stop --deployment application .../bmc/stackmanager stop --deployment database 2. Stop the docker service on the TSAC application and database server (if the DB is containerized) using the below command: systemctl stop docker 3. Check the current file sizes of the json.log files using the following command: du -sch `find /var/lib/docker/containers -name *-json.log*` | grep G
Example:
[root@Example-TSAC-Server ~]# du -sch `find /var/lib/docker/containers -name *-json.log*` | grep G
23G /var/lib/docker/containers/95afa7f5b0f7c5cb9273d2dd4c59880961c4f40fa461843178da7c9d59f330e4/95afa7f5b0fc4-json.log
1.6G /var/lib/docker/containers/a5984a45d29face6de9c5fb281dd07a28406d1b0730bf613b42a38f2ddeaf0cd/a5984a45d29fa-json.log
28G total
4. Then truncate the large json.log file to free up space using the below command. truncate -s 0 "json.log"
Example:
[root@Example-TSAC-Server ~]# cd /var/lib/docker/containers/95afa7f5b0f7c5cb9273d2dd4c59880961c4f40fa461843178da7c9d59f330e4/
[root@Example-TSAC-Server 95afa7f5b0f7c5cb9273d2dd4c59880961c4f40fa461843178da7c9d59f330e4]# truncate -s 0 95afa7f5b0fc4-json.log
Note: Please make sure to truncate only the JSON log file. If not sure then open a support case. 5. Enable log rotation and keep these file sizes within the limit on all the TSAC hosts. Create/Update the /etc/docker/daemon.json file as per the steps mentioned in the documentation - Configure log file settings 6. Start the docker service and check the service status, it should not show any errors. systemctl start docker systemctl status docker 7. Start the database and application by running the stackmanager command: .../bmc/stackmanager start --deployment application .../bmc/stackmanager start --deployment database 8. Validate the TSAC application. Feel free to reach out to BMC Support if you have any more questions or concerns. |