Search the VMware Knowledge Base (KB)
View by Article ID

After enabling Virtual Flash, VMware vCenter Server 5.5 consumes most of the CPU, memory, or disk I/O (2072392)

  • 36 Ratings

Symptoms

  • Virtual Flash (vFlash) is enabled in your vSphere Server environment.
  • Most of the CPU and memory is consumed by:

    • vpxd.exe
    • java.exe

  • Disk space is consumed rapidly because the VMware vCenter Inventory Service data folder increase in size (located at C:\Program Files\VMware\Infrastructure\Inventory Service).
  • Log into the vSphere Web Client fails with the below error:

    Client is not authenticated to VMware Inventory Service - https://<VC>:10433

    Note: Even if the login is successful, the whole inventory is not available in the vSphere Web Client.

  • Searching objects in vCenter Server via vSphere Client fails with this error:

    Unable to connect to web services to execute query.Verify that the 'VMware VirtualCenter Management Webservices' service is running on https://VC:10433

  • The ds.log file, located at C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs, contains entries similar to:
ERROR com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl] Store eror processing atom feed:
com.vmware.vim.query.server.store.exception.StoreException: com.vmware.vim.query.server.store.exception.StoreException: org.w3c.dom.ls.LSException: XML docu
ment structures must start and end within the same entity.
at com.vmware.vim.query.server.store.impl.StoreImpl.updateVmomiPullAtomFeed(StoreImpl.java:2168)
at com.vmware.vim.query.server.store.impl.StoreImpl.update(StoreImpl.java:2008)
at com.vmware.vim.query.server.provider.AbstractAtomPullProviderBase.checkFeed(AbstractAtomPullProviderBase.java:144)
at com.vmware.vim.query.server.provider.impl.ProviderManagerServiceImpl$FeedPump.run(ProviderManagerServiceImpl.java:763)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.vmware.vim.query.server.store.exception.StoreException: org.w3c.dom.ls.LSException: XML document structures must start and end within the sam
e entity.
at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:129)
at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:46)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more
Caused by: org.w3c.dom.ls.LSException: XML document structures must start and end within the same entity.
at xDB_9_0_11.qP.a(xdb:340)
at xDB_9_0_11.qP.a(xdb:354)
at xDB_9_0_11.qP.a(xdb:138)
at xDB_9_0_11.qP.c(xdb:133)
at xDB_9_0_11.qP.parse(xdb:46)
at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryUtil.create(AtomFeedProcessor.java:549)
at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$PullEntryProcessor.processEntry(AtomFeedProcessor.java:308)
at com.vmware.vim.query.server.store.impl.AtomFeedProcessor$EntryProcessor.call(AtomFeedProcessor.java:101)
... 9 more
Caused by: xDB_9_0_11.Ex: XML document structures must start and end within the same entity.
::::1:900395:900394:XML document structures must start and end within the same entity.
at xDB_9_0_11.jD.a(xdb:333)
at xDB_9_0_11.jD.a(xdb:106)
at xDB_9_0_11.qP.a(xdb:352)
... 15 more
Caused by: ::::1:900395:900394:XML document structures must start and end within the same entity.
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)
at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.scanLiteral(Unknown Source)
at org.apache.xerces.impl.XMLScanner.scanAttributeValue(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanAttribute(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at xDB_9_0_11.jD.a(xdb:306)
... 17 more
  • The vpxd.log file, located at C:\ProgramData\VMware\VMware VirtualCenter\Logs, contains entries similar to:

[06236 error 'commonvpxVmomi' opID=3C35744C-001F520A-a8] [VpxVmomi] Exception while sending result: Not initialized: vim.host.FileSystemVolume volume
YYYY-02-04T07:54:16.157+01:00 [06236 verbose 'commonvpxVmomi' opID=3C35744C-001F520A-a8] backtrace[00] rip 000000018018cd7a
backtrace[01] rip 0000000180106c48
backtrace[02] rip 000000018010803e
backtrace[03] rip 00000001800907f8
backtrace[04] rip 0000000000524103
backtrace[05] rip 000000000059c307
backtrace[06] rip 00000000005a572c
backtrace[07] rip 00000000005a5be5
backtrace[08] rip 00000000005a654d
backtrace[09] rip 00000000005a68e7
backtrace[10] rip 00000000005a6a04
backtrace[11] rip 0000000000582407
backtrace[12] rip 00000000005833d6
backtrace[13] rip 000000013fd446d9
backtrace[14] rip 000000000055a527
backtrace[15] rip 000000000055cf80
backtrace[16] rip 00000000005663c9
backtrace[17] rip 0000000180154112
backtrace[18] rip 0000000180159031
backtrace[19] rip 000000018015a8b7
backtrace[20] rip 000000018015add8
backtrace[21] rip 000000018015398a
backtrace[22] rip 0000000180156269
backtrace[23] rip 0000000180155853
backtrace[24] rip 0000000180156793
backtrace[25] rip 0000000180156ad3
backtrace[26] rip 000000018019ef4a
backtrace[27] rip 0000000074052fdf
backtrace[28] rip 0000000074053080
backtrace[29] rip 0000000076d0652d
backtrace[30] rip 0000000076e3c521

Resolution

This is a known issue affecting VMware vCenter Server 5.5.
 
This issue is resolved in vCenter Server 5.5 Update 2, available at VMware Downloads. For more information, see the Resolved Issues section in the vCenter Server 5.5 Update 2 Release Notes.

Note: You may need to apply this workaround steps if there are affected hosts prior to applying vCenter Server 5.5 Update 2.

To verify your environment has affected hosts, query the database: 
  1. Connect the SQL Server where the vCenter Server's database resides using SQL Management Studio.
  2. Query the vCenter Server database, with the command:
  • In vCenter Server:
select distinct fm.ID, h.DNS_NAME
from vpx_file_system_mount_info fm,
     VPX_HOST h
where not exists
 (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
  where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID
  • In the VMware vCenter Server appliance with embedded database:
  1. Log in to the VCSA as root.
  2. Run this command:

    cd /opt/vmware/vpostgres/9.0/bin

  3. Run this command to connect to the VCDB database:

    ./psql -d VCDB -U vc

  4. Run this command to query the vCenter Server database:

    select distinct fm.ID, h.DNS_NAME from vpx_file_system_mount_info fm, VPX_HOST h where not exists (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID) and fm.ID = h.ID;

  5. To quit, type:

    \q

To work around this issue, execute the provided SQL commands on the vCenter Server database and modify the vmConfigInfo.xmlfile. Then restart vCenter Server.
 
To update the vCenter Server database configuration and modify vmConfigInfo.xml:
  1. Use the query above to list the affected ESXi hosts from vCenter Server.
  2. Stop the VirtualCenter Server service. For more information, see Stopping, starting, or restarting vCenter services (1003895).
  3. Go to:

    • In vCenter Server: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\

    • VMware vCenter Server appliance:  /usr/lib/vmware-vpx/

  4. Take a backup of  vmConfigInfo.xml.

    Warning: Do not skip this step.

  5. Open  vmConfigInfo.xml in a text editor.
  6. Insert the following information after the section that begins with <Table name="VPX_VMFS_VOLUME"... after the </Table> and before the  <Table name="VPX_SCSI_DISK_PARTITION" section.

    <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VffsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
            <Column name="VPX_FILE_SYSTEM_MOUNT_INFO_ID" isKey="1" type="int" option="NOT NULL" />
            <Column name="UUID" path="fileSystemVolume.mountInfo.volume.uuid" type="string" />
            <Column name="MAJOR_VERSION" path="fileSystemVolume.mountInfo.volume.majorVersion" type="int" />
            <Column name="VERSION" path="fileSystemVolume.mountInfo.volume.version" type="string" />
            <PrimaryKey name="PK_VPX_VMFS_VOLUME" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
            <ForeignKey name="FK_V_VM_VOL_REF_V_FI_SYS_MO_IN" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" ref="VPX_FILE_SYSTEM_MOUNT_INFO" refVal="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
    </Table>

    In this example the new data is added in red:

    <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VmfsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
    ...
    </Table>
    <Table name="VPX_VMFS_VOLUME" vmodl_name="vim.host.VffsVolume" parentClass="vim.host.FileSystemVolume" parentAttr="fileSystemVolume.mountInfo.volume" >
    <Column name="VPX_FILE_SYSTEM_MOUNT_INFO_ID" isKey="1" type="int" option="NOT NULL" />
    <Column name="UUID" path="fileSystemVolume.mountInfo.volume.uuid" type="string" />
    <Column name="MAJOR_VERSION" path="fileSystemVolume.mountInfo.volume.majorVersion" type="int" />
    <Column name="VERSION" path="fileSystemVolume.mountInfo.volume.version" type="string" />
    <PrimaryKey name="PK_VPX_VMFS_VOLUME" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
    <ForeignKey name="FK_V_VM_VOL_REF_V_FI_SYS_MO_IN" key="VPX_FILE_SYSTEM_MOUNT_INFO_ID" ref="VPX_FILE_SYSTEM_MOUNT_INFO" refVal="VPX_FILE_SYSTEM_MOUNT_INFO_ID" />
    </Table>
    <Table name="VPX_SCSI_DISK_PARTITION" vmodl_name="vim.host.ScsiDisk.Partition" parentClass="vim.host.VmfsVolume" parentAttr="fileSystemVolume.mountInfo.volume.extent" isArray="1" isUnkeyed="1" >
    ...
    </Table>

  7. Save changes in vmConfigInfo.xml and close the file.
  8. Take a backup of the vCenter Server SQL database.
  9. Execute this query to update the vCenter Server database configuration:
  • In vCenter Server:

    update VPX_HOST set MASTER_GEN=0, MASTER_SPEC_GEN=0
    where ID in
    (
    select distinct fm.ID
    from vpx_file_system_mount_info fm
    where not exists
    (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi
    where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID)
    )

  • In the VMware vCenter Server appliance with embedded database:
  1. Login to the VCSA as root.
  2. Run this command: cd /opt/vmware/vpostgres/9.0/bin
  3. Run this command to connect to the VCDB database:

    ./psql -d VCDB -U vc

  4. Run this command to update the vCenter Server database configuration:

    update VPX_HOST set MASTER_GEN=0, MASTER_SPEC_GEN=0 where ID in (select distinct fm.ID from vpx_file_system_mount_info fm where not exists (select 1 from VPX_FS_MOUNT_VOLUME_INFO vi where fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID));
  1. Execute this query to update the vCenter Server database configuration:
  • In vCenter Server (running on Windows), connect to the SQL Server (running the vCenter Server database) and run this command:

    delete from vpx_file_system_mount_info
    where VPX_FILE_SYSTEM_MOUNT_INFO_ID in
    (
    select fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID
    from vpx_file_system_mount_info fm left outer join VPX_FS_MOUNT_VOLUME_INFO vi
    on fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID
    where vi.FS_ID is NULL
    )


  • In the VMware vCenter Server appliance with embedded database:
    1. Log in to the VCSA as root.
    2. Run this command: cd /opt/vmware/vpostgres/9.0/bin
    3. Run this command to connect to the VCDB database:

      ./psql -d VCDB -U vc

    4. Run this command to update the vCenter Server Appliance database configuration:

      delete from vpx_file_system_mount_info where VPX_FILE_SYSTEM_MOUNT_INFO_ID in (select fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID from vpx_file_system_mount_info fm left outer join VPX_FS_MOUNT_VOLUME_INFO vi on fm.VPX_FILE_SYSTEM_MOUNT_INFO_ID = vi.FS_ID where vi.FS_ID is NULL);
  1. Restart vCenter Server to reflect changes.
  2. After the vCenter Server is running, vCenter Server forces a fill sync with all affected hosts, updating the correct value.

    Notes:
  • If the issue still persists, the query returns the ESXi host's Managed Object ID (MOID) as well as the DNS name. To resolve the issue, remove and re-add that host to the vCenter Server.

    Caution: Do not remove the hosts from inventory if the hosts are used for vCloud Director or VMware Horizon View, removing the hosts causes virtual machines to become invalid. VMware recommends you to contact support for further troubleshooting.

  • If this does not resolve your issue, open a VMware Support Request. For more information, see Filing a Support Request in My VMware (2006985).

Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 36 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 36 Ratings
Actions
KB: