Search the VMware Knowledge Base (KB)
View by Article ID

vCenter Server Appliance 6.0 services fail due to 100% usage of /storage/seat disk (2119809)

  • 26 Ratings
Language Editions

Symptoms

  • The vCenter Server Appliance services fail to start.
  • Running df -h on the vCenter Server Appliance shows /storage/seat as full.

    Filesystem Size Used Avail Use% Mounted on
    /dev/sda3 11G 3.9G 6.4G 38% /
    udev 4.0G 164K 4.0G 1% /dev
    tmpfs 4.0G 40K 4.0G 1% /dev/shm
    /dev/sda1 128M 38M 84M 31% /boot
    /dev/mapper/core_vg-core 25G 2.7G 21G 12% /storage/core
    /dev/mapper/log_vg-log 9.9G 2.5G 7.0G 26% /storage/log
    /dev/mapper/db_vg-db 9.9G 214M 9.2G 3% /storage/db
    /dev/mapper/dblog_vg-dblog 5.0G 379M 4.3G 8% /storage/dblog
    /dev/mapper/seat_vg-seat 9.9G 9.4G 0 100% /storage/seat
    /dev/mapper/netdump_vg-netdump 1001M 18M 932M 2% /storage/netdump
    /dev/mapper/autodeploy_vg-autodeploy 9.9G 151M 9.2G 2% /storage/autodeploy
    /dev/mapper/invsvc_vg-invsvc 5.0G 191M 4.5G 4% /storage/invsvc


  • In the /var/log/vmware/vpxd/vpxd.log file, see entries similar to:

    <YYYY-MM-DD>T<TIME> warning vpxd[7F447FF7E700] [Originator@6876 sub=Default opID=StatsTruncateExpiredPartitions-7460ee5a] [VdbStatement] Connection diagnostic data from driver is HY000:0:110:
    <YYYY-MM-DD>T<TIME> error vpxd[7F44865F3700] [Originator@6876 sub=Default] [VdbStatement] Execute result code: -1
    <YYYY-MM-DD>T<TIME> warning vpxd[7F44865F3700] [Originator@6876 sub=Default] [VdbStatement] SQL execution failed: INSERT INTO VPX_EVENT (EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, STORAGEPOD_ID, STORAGEPOD_NAME, CHANGE_TAG_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    <YYYY-MM-DD>T<TIME> warning vpxd[7F44865F3700] [Originator@6876 sub=Default] [VdbStatement] Execution elapsed time: 3 ms
    <YYYY-MM-DD>T<TIME> warning vpxd[7F44865F3700] [Originator@6876 sub=Default] [VdbStatement] Statement diagnostic data from driver is 53100:0:7:ERROR: could not extend file "pg_tblspc/16396/PG_9.3_201306121/16384/16641": No space left on device;
    --> Error while executing the query

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs due to a large amount of events collected on vCenter Server filling the database and fail vCenter Server.

Resolution

To resolve this issue, find and resolve the issue causing the events to be created.  After this is done, the events tables must be truncated.

To find the ESX host that is generating the events:

  1. Connect to vCenter Server Appliance through the console or using an SSH session and root credentials.
  2. Enable the shell by running this command:

    shell.set --enabled true

  3. To enter the shell, type shell and press Enter.
  4. Navigate to /opt/vmware/vpostgres/current/bin by running this command:

    cd /opt/vmware/vpostgres/current/bin

  5. Run this command to log in to the vCenter Server Appliance database.

    ./psql -d VCDB -U postgres

  6. Run this query to determine the source of the events:

    select event_type, host_name, count(*) from vpx_event group by host_name, event_type having count(*) > 10 order by count(*) desc; 

    You see output similar to:

    vmfs.heartbeat.timedout |192.168.50.71  | 12191576
    vim.event.UserLogoutSessionEvent |192.168.50.71  | 12191215
    vim.event.VmAcquiredTicketEvent |192.168.50.71 | 15568


  7. In the above example we see 12191576 events generated from the ESXi host with the IP of 192.168.50.71 related to vmfs.heartbeat.timedout.

  8. This host must be further investigate to resolve the issue that is being reported.

To clean up the events in the vCenter Server Appliance database:

  1. From the console or SSH session connected to vCenter Server Appliance, run these commands to truncate the event table data:

    truncate table vpx_event cascade;

    and:

    truncate table vpx_event_arg cascade;

  2. Exit the vCenter Server Appliance database by running this command:

    \q

  3. Verify the space is reclaimed by running the df -h command.

    You see output similar to:

    Filesystem Size Used Avail Use% Mounted on
    /dev/sda3 11G 3.7G 6.6G 36% /
    udev 4.0G 164K 4.0G 1% /dev
    tmpfs 4.0G 32K 4.0G 1% /dev/shm
    /dev/sda1 128M 38M 84M 31% /boot
    /dev/mapper/core_vg-core 25G 173M 24G 1% /storage/core
    /dev/mapper/log_vg-log 9.9G 514M 8.9G 6% /storage/log
    /dev/mapper/db_vg-db 9.9G 210M 9.2G 3% /storage/db
    /dev/mapper/dblog_vg-dblog 5.0G 491M 4.2G 11% /storage/dblog
    /dev/mapper/seat_vg-seat 9.9G 194M 9.2G 3% /storage/seat
    /dev/mapper/netdump_vg-netdump 1001M 18M 932M 2% /storage/netdump
    /dev/mapper/autodeploy_vg-autodeploy 9.9G 151M 9.2G 2% /storage/autodeploy
    /dev/mapper/invsvc_vg-invsvc 5.0G 165M 4.6G 4% /storage/invsvc

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 26 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 26 Ratings
Actions
KB: