Discovery: The appliance has shut down and cannot be restarted because a disk is full INCLUDES VIDEO

Manage Support IDs

Use checkboxes to 'Remove support IDs/Users' or 'Allow to create Cases'
Support IDs for BMC Customer Support Site Guest User
Profiles Which Include    Back to All Support IDs
 

Knowledge Article

Discovery: The appliance has shut down and cannot be restarted because a disk is full INCLUDES VIDEO
The appliance has shut down. It cannot be restarted. The disk/partition/storage is full/saturated or almost full/saturated.
BMC Discovery
BMC Discovery
all versions
Symptoms :
Discovery Appliance is shut down because of insufficient disk space,
Disk Space is full on one or more cluster members.
The GUI is not accessible.
Disk usage warning on /dev/sdxx on the BMC Discovery appliance,
Appliance Specification Baseline - MAJOR: This appliance has insufficient resources,
DataStore Size Warning is seen in discovery GUI , Discovery Datastore is showing 100 % in CLI result for "df -h"



The spanish version of this article is below:
BMC Discovery: Imposible iniciar los servicios debido a falta de espacio en el disco duro. (The appliance has shut down because available disk space is low)

By the time the permanent solution is implemented, here are 2 temporary workarounds:

1- Add more disk space and configure discovery to use it. For more details, see the article below:

Discovery: How to modify the disk configuration without using the GUI (tw_disk_utils)? 

2- Configure Discovery to consume less disk (without fixing any issues if any). For example: stop scans or compact.

Permanent solutions requires an investigation to find the root cause and fix it. The article below shows the relations between the different event/issues that can lead to a disk full:

Discovery: Impact diagram showing the causes of disk full and performance issues

The potential root causes are represented by the black boxes.

The first step is to determine which partition is full. This can be done by running "df -h" from the command line.

Most of the time, the saturated partitions are /usr/tideway or the datastore partition. The output of 'df -h' may not show any partition at 100% capacity, but whichever one reports the highest percentage usage is likely the one that is having the problem.

Potential root causes of saturation of the datastore partition:
- The storage is undersized. In the case of a standalone appliance, it is not compliant with the documented requirements

- The disk that contains the datastore is a virtual and it was extended but the partition were not. For more details, see the article below:

- Discovery accumulated an unreasonable amount of nodes to delete. See the article below:

- Discovery accumulated an unreasonable amount of fragmentation. Extract: " fragmented, meaning that the data within them is structured inefficiently". Most of the time, this is because the datastore is not compacted frequently enough
 

- Multi generational datastore is enabled and the 3 settings below are not suitable:
  1. the new generation frequency
  2. the compact frequency
  3. the size of the db partition
They lead Discovery to consume too much disk space. The solution is to either add storage and/or compact more frequently or and/or create new generations less frequently.

- Defect DRUD1-46885. See this article

Potential root causes of saturation of the transaction log partition (if moved to a new disk):

- The "Time before history entries are purged" was decreased. See the article below
 

- In theory, a performance issue could slow down the datastore, lead to an accumulation of datastore transactions pending in the datastore log partition and then a saturation of this partition. This cause-effect link has never been confirmed so far, this is a suspected root cause.

Potential root causes of a /usr/tideway saturation:

- If the datastore (or datastore transaction logs) is stored in /usr, see the section "Potential root causes of saturation of the datastore partition" above.

- Core dumps in /usr/tideway/cores. They can be deleted after the investigation.

- The logs are unreasonably big. Disable debug" logs if needed. Most of the time, this is not required but logs can be moved to a new disk

- Large reasoning persist (*.pq) files 

- On the consolidation appliance, the /usr partition is saturated by files in /usr/tideway/var/persist/consolidation

- A local backup takes too much space in /usr/tideway/var/localdisk/backup.

- There is too much recorded data in

/usr/tideway/var/pool
and/or
/usr/tideway/var/record.

The contents of these folders can be deleted, but not the folders themselves. Make sure that record mode is not enabled (see "Recording Mode" in this documentation page)
 

Potential root causes of a /var/log saturation:

- A customization can generate an unreasonable amount of lines in /var/log/messages.  For example: a customization in /etc/rsyslog.conf can redirect Discovery logs into /var/log/messages.

You can identify the root cause with commands like:
df -h 
du -h /usr/tideway | grep [0-9]G

If a Support case will be opened, please always send the results of "df -h".

Please also see the following video "How to resolve disk space problems in BMC Discovery".
BMC Discovery
BMC Discovery
all versions
BMC Discovery
BMC Discovery
all versions
Attachment(s):