This article is provided to assist in troubleshooting if any of the partition is full on the vCenter Server Appliance is getting full.
The below is the list of partitions in the vCenter Server Appliance.
You can find the big logs and truncate them.
root@vcsa1 [ /storage/log ]# du -sh *
Validate if any log bundles are present in /storage/log. When you run, vc-support to generate the log bundle, the logs are stored in /storage/log.
root@vcsa1 [ /storage/log ]# ls
lost+found remote vc-vcsa1.technorians.local-2018-09-30–05.21-15777.tgz vmware
Remove the old log bundle.
root@vcsa1 [ /storage/log ]# rm *.tgz
localhost_access_log, vmware-identity-sts and vmware-identity-sts-perf takes larger space. Its safe to delete them.
This is caused to the fact that the logs in /storage/log/vmware/sso are not compressed.
The issue is fixed in 6.0 U3.
Ref. KB: https://kb.vmware.com/s/article/2143565
Also you ll see multiple files like localhost_access_log..2018-05-04.txt. They can also be removed.
root@vcsa1 [ /storage/log/vmware/sso ]# rm *.txt
The other logs which takes a larger space is WebClient logs and Cloudvm
Note: Do not delete any active logs. It would lead to service failure as the service might not have permission to create the logs.
To delete the contents of the logs, please use the following command.
root@vcsa1 [ /storage/log/vmware/cloudvm ]# > cloudvm-ram-size.log
root@vcsa1 [ /storage/log/vmware/vsphere-client/logs ]# > vsphere_client_virgo.log
You will also see java_pidxxxx.hprof files which would be occuping space. Its safe to delete those files.
The issue is fixed in 6.0 U1d.
Ref. KB: https://kb.vmware.com/s/article/2151394
You will see that the audit log will grow till to fill up the entire system partition.
This is caused to the fact that the root password is expired.
Ref. KB: https://kb.vmware.com/s/article/2149278
root@vcsa1 [ /var/log/audit ]# > audit.log
Sometimes the dnsmasq log file will grow.
This issue occurs because dnsmasq.log is being directed to /var/log/ instead of /var/log/vmware/
The issue is fixed in 6.0 U3g.
Ref. KB: https://kb.vmware.com/s/article/52258
root@vcsa1 [ /storage/log/vmware ]# > dnsmasq.log
Most of the times, the space will not be re-claimed in this partition. It needs a reboot for the space reclamation.
When there are too many core dump getting created for the vpxd process within a short time, this partition will get full.
Its safe to delete all the files of this folder unless you dont want the dump to analyze the crash. Please do not delete any folder.
root@vcsa1 [ ~ ]# cd /storage/core
root@vcsa1 [ ~ ]# rm *.*
Stats, Events, Alarms and Tasks (SEAT) directory for VMware Postgres. It gets full when there are too many events within the event rotation period.
When this partition reaches 95%, the vpxd service auto shuts down.
Connect to the vCenter Database.
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
To display the top 20 biggest tables, run the below command.
SELECT nspname || ‘.’ || relname AS “relation”, pg_size_pretty(pg_total_relation_size(C.oid)) AS “total_size” FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN (‘pg_catalog’, ‘information_schema’) AND C.relkind <> ‘i’ AND nspname !~ ‘^pg_toast’ ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 20;
relation | total_size
vc.vpx_task | 1688 MB
vc.vpx_event_arg_21 | 552 MB
vc.vpx_event_arg_29 | 544 MB
vc.vpx_event_29 | 528 MB
vc.vpx_event_arg_22 | 528 MB
vc.vpx_event_22 | 520 MB
vc.vpx_event_30 | 504 MB
vc.vpx_event_arg_30 | 504 MB
vc.vpx_event_21 | 504 MB
vc.vpx_event_28 | 496 MB
Usually the VPX_EVENT_ARG table would be bigger.
Truncate all the Event tables to clear the old events.
truncate table VPX_EVENT_ARG_1;
truncate table VPX_EVENT_ARG_2;
truncate table VPX_EVENT_ARG_3;
It is also safe to truncate the VPX_TASK, VPX_ENTITY_LAST_EVENT, VPX_EVENT.
This will get full when the customer has many images profiles for the autodeploy services.
Never delete any file in this location. Deleting a file leaves a stale entry on the VCDB.
Request the customer to delete the unused image profiles.
If any of the partition is full, sometimes you will not be even authenticate via SSH. However, VAMI login works.
In that case, login to the GRUB mode and delete the log files.
vCenter 6.0: https://kb.vmware.com/s/article/2069041 –> Follow upto Step 10
vCenter 6.5: https://www.altaro.com/vmware/reset-root-password-vcsa-6-x/ –> Follow upto Step 4
If nothing helps, then proceed with Increasing the partition size. https://kb.vmware.com/s/article/2126276
Sometime it happens, after we deleted the old files, some services do not start.
– inventory service: https://kb.vmware.com/s/article/2147280
– eam service : https://kb.vmware.com/s/article/2148582
– vdcs (content-library) service : https://kb.vmware.com/s/article/2147891