Troubleshooting VCSA Partition getting full

This article is provided to assist in troubleshooting if any of the partition is full on the vCenter Server Appliance is getting full.

The below is the list of partitions in the vCenter Server Appliance.

/storage/log Full

You can find the big logs and truncate them.

root@vcsa1 [ /storage/log ]# du -sh *

Step 1:

Validate if any log bundles are present in /storage/log. When you run, vc-support to generate the log bundle, the logs are stored in /storage/log.

root@vcsa1 [ /storage/log ]# ls
lost+found remote vc-vcsa1.technorians.local-2018-09-30–05.21-15777.tgz vmware

Remove the old log bundle.
root@vcsa1 [ /storage/log ]# rm *.tgz

Step 2:

localhost_access_log, vmware-identity-sts and vmware-identity-sts-perf takes larger space. Its safe to delete them.
This is caused to the fact that the logs in /storage/log/vmware/sso are not compressed.
The issue is fixed in 6.0 U3.
Ref. KB: https://kb.vmware.com/s/article/2143565

Also you ll see multiple files like localhost_access_log..2018-05-04.txt. They can also be removed.

root@vcsa1 [ /storage/log/vmware/sso ]# rm *.txt

Step 3:

The other logs which takes a larger space is WebClient logs and Cloudvm
Note: Do not delete any active logs. It would lead to service failure as the service might not have permission to create the logs.

To delete the contents of the logs, please use the following command.

root@vcsa1 [ /storage/log/vmware/cloudvm ]# > cloudvm-ram-size.log
root@vcsa1 [ /storage/log/vmware/vsphere-client/logs ]# > vsphere_client_virgo.log

Step 4:

You will also see java_pidxxxx.hprof files which would be occuping space. Its safe to delete those files.
The issue is fixed in 6.0 U1d.
Ref. KB: https://kb.vmware.com/s/article/2151394

/dev/sda3 Full

You will see that the audit log will grow till to fill up the entire system partition.
This is caused to the fact that the root password is expired.
Ref. KB: https://kb.vmware.com/s/article/2149278

root@vcsa1 [ /var/log/audit ]# > audit.log

Sometimes the dnsmasq log file will grow.
This issue occurs because dnsmasq.log is being directed to /var/log/ instead of /var/log/vmware/
The issue is fixed in 6.0 U3g.
Ref. KB: https://kb.vmware.com/s/article/52258

root@vcsa1 [ /storage/log/vmware ]# > dnsmasq.log

Most of the times, the space will not be re-claimed in this partition. It needs a reboot for the space reclamation.

/storage/core Full

When there are too many core dump getting created for the vpxd process within a short time, this partition will get full.
Its safe to delete all the files of this folder unless you dont want the dump to analyze the crash. Please do not delete any folder.

root@vcsa1 [ ~ ]# cd /storage/core
root@vcsa1 [ ~ ]# rm *.*

/storage/seat Full

Stats, Events, Alarms and Tasks (SEAT) directory for VMware Postgres. It gets full when there are too many events within the event rotation period.
When this partition reaches 95%, the vpxd service auto shuts down.

Connect to the vCenter Database.
https://kb.vmware.com/s/article/2147285

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

To display the top 20 biggest tables, run the below command.

SELECT nspname || ‘.’ || relname AS “relation”, pg_size_pretty(pg_total_relation_size(C.oid)) AS “total_size” FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN (‘pg_catalog’, ‘information_schema’) AND C.relkind <> ‘i’ AND nspname !~ ‘^pg_toast’ ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 20;

relation | total_size
——————————+————
vc.vpx_task | 1688 MB
vc.vpx_event_arg_21 | 552 MB
vc.vpx_event_arg_29 | 544 MB
vc.vpx_event_29 | 528 MB
vc.vpx_event_arg_22 | 528 MB
vc.vpx_event_22 | 520 MB
vc.vpx_event_30 | 504 MB
vc.vpx_event_arg_30 | 504 MB
vc.vpx_event_21 | 504 MB
vc.vpx_event_28 | 496 MB

Usually the VPX_EVENT_ARG table would be bigger.
Truncate all the Event tables to clear the old events.
https://kb.vmware.com/s/article/1025914

truncate table VPX_EVENT_ARG_1;
truncate table VPX_EVENT_ARG_2;
truncate table VPX_EVENT_ARG_3;

It is also safe to truncate the VPX_TASK, VPX_ENTITY_LAST_EVENT, VPX_EVENT.

/storage/imagebuilder Full

This will get full when the customer has many images profiles for the autodeploy services.
Never delete any file in this location. Deleting a file leaves a stale entry on the VCDB.
Request the customer to delete the unused image profiles.

Notes

If any of the partition is full, sometimes you will not be even authenticate via SSH. However, VAMI login works.
In that case, login to the GRUB mode and delete the log files.

vCenter 6.0: https://kb.vmware.com/s/article/2069041 –> Follow upto Step 10
vCenter 6.5: https://www.altaro.com/vmware/reset-root-password-vcsa-6-x/ –> Follow upto Step 4

If nothing helps, then proceed with Increasing the partition size. https://kb.vmware.com/s/article/2126276

Sometime it happens, after we deleted the old files, some services do not start.

– inventory service: https://kb.vmware.com/s/article/2147280
– eam service : https://kb.vmware.com/s/article/2148582
– vdcs (content-library) service : https://kb.vmware.com/s/article/2147891