vCenter Fails to Start with the Error: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
Article ID: 326594
Updated On:
Products
VMware vSAN
Issue/Introduction
Symptoms:
- vCenter Server service VPXD crashes or fails to start. In the log /var/log/vmware/vpxd/vpxd.log you see errors similar to:
error vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb] [VdbStatement] SQLError was thrown: "ODBC error: (23503) - ERROR: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_SN_CONFIG_INFO (ID, CHANGE_VERSION, ARRAY_ID, CHANGE_TRACKING_ENABLED, CPU_HOT_ADD_ENABLED, CPU_HOT_REMOVE_ENABLED, MEM_HOT_ADD_ENABLED, VASSERTS_ENABLED, HARDWARE_NUM, HARDWARE_MEMORY, TOOLS_BEFORE_GUEST_STANDBY_FLG, TOOLS_BEFORE_GUESTSHUTDOWN_FLG, TOOLS_TOOLS_UPGRADE_POLICY, TOOLS_AFTER_RESUME_FLG, TOOLS_AFTER_POWER_ON_FLG, TOOLS_SYNC_TIME_WITH_HOST_FLG, TOOLS_TOOLS_VERSION, GUEST_FULL_NAME, INSTANCE_UUID, UUID, ANNOTATION, VERSION, TEMPLATE_FLG, MEMORY_ALLOCATION_RESERVATION, MEMORY_ALLOC"
error vpxd[7F512A1C3700] [Originator@6876 sub=DbSave opID=HB-host-20@604097-691c56bb] [VpxdDbSave::ProcessInsertsInOrder] Failed to insert 2 records into the DB
error vpxd[7F512A1C3700] [Originator@6876 sub=vpxCommon opID=HB-host-20@604097-691c56bb] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
error vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23503) - ERROR: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_SN_CONFIG_INFO (ID, CHANGE_VERSION, ARRAY_ID, CHANGE_TRACKING_ENABLED, CPU_HOT_ADD_ENABLED, CPU_HOT_REMOVE_ENABLED, MEM_HOT_ADD_ENABLED, VASSERTS_ENABLED, HARDWARE_NUM, HARDWARE_MEMORY, TOOLS_BEFORE_GUEST_STANDBY_FLG, TOOLS_BEFORE_GUESTSHUTDOWN_FLG, TOOLS_TOOLS_UPGRADE_POLICY, TOOLS_AFTER_RESUME_FLG, TOOLS_AFTER_POWER_ON_FLG, TOOLS_SYNC_TIME_WITH_HOST_FLG, TOOLS_TOOLS_VERSION, GUEST_FULL_NAME, INSTANCE_UUID, UUID, ANNOTATION, VERSION, TEMPLATE_FLG, MEMORY_ALLOCATION_RESERVATION, MEMORY_ALLOC"
panic vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-7515524, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002B3E43]
--> backtrace[03] libvmacore.so[0x002B3F1E]: Vmacore::PanicExit(char const*)
--> backtrace[04] vpxd[0x0072584A]
--> backtrace[05] vpxd[0x0073B18B]
--> backtrace[06] vpxd[0x0073D47B]
--> backtrace[07] vpxd[0x007427B2]
--> backtrace[08] libvmacore.so[0x0023AC47]
--> backtrace[09] libvmacore.so[0x0023AF2E]
--> backtrace[10] libvmacore.so[0x002BB737]
--> backtrace[11] libpthread.so.0[0x00007454]
--> backtrace[12] libc.so.6[0x000E8C1D]
--> [backtrace end]
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_SN_CONFIG_INFO (ID, CHANGE_VERSION, ARRAY_ID, CHANGE_TRACKING_ENABLED, CPU_HOT_ADD_ENABLED, CPU_HOT_REMOVE_ENABLED, MEM_HOT_ADD_ENABLED, VASSERTS_ENABLED, HARDWARE_NUM, HARDWARE_MEMORY, TOOLS_BEFORE_GUEST_STANDBY_FLG, TOOLS_BEFORE_GUESTSHUTDOWN_FLG, TOOLS_TOOLS_UPGRADE_POLICY, TOOLS_AFTER_RESUME_FLG, TOOLS_AFTER_POWER_ON_FLG, TOOLS_SYNC_TIME_WITH_HOST_FLG, TOOLS_TOOLS_VERSION, GUEST_FULL_NAME, INSTANCE_UUID, UUID, ANNOTATION, VERSION, TEMPLATE_FLG, MEMORY_ALLOCATION_RESERVATION, MEMORY_ALLOC"
error vpxd[7F512A1C3700] [Originator@6876 sub=DbSave opID=HB-host-20@604097-691c56bb] [VpxdDbSave::ProcessInsertsInOrder] Failed to insert 2 records into the DB
error vpxd[7F512A1C3700] [Originator@6876 sub=vpxCommon opID=HB-host-20@604097-691c56bb] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
error vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23503) - ERROR: insert or update on table "vpx_sn_config_info" violates foreign key constraint "fk_v_sn_config_info_ref_vpx_sn";
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_SN_CONFIG_INFO (ID, CHANGE_VERSION, ARRAY_ID, CHANGE_TRACKING_ENABLED, CPU_HOT_ADD_ENABLED, CPU_HOT_REMOVE_ENABLED, MEM_HOT_ADD_ENABLED, VASSERTS_ENABLED, HARDWARE_NUM, HARDWARE_MEMORY, TOOLS_BEFORE_GUEST_STANDBY_FLG, TOOLS_BEFORE_GUESTSHUTDOWN_FLG, TOOLS_TOOLS_UPGRADE_POLICY, TOOLS_AFTER_RESUME_FLG, TOOLS_AFTER_POWER_ON_FLG, TOOLS_SYNC_TIME_WITH_HOST_FLG, TOOLS_TOOLS_VERSION, GUEST_FULL_NAME, INSTANCE_UUID, UUID, ANNOTATION, VERSION, TEMPLATE_FLG, MEMORY_ALLOCATION_RESERVATION, MEMORY_ALLOC"
panic vpxd[7F512A1C3700] [Originator@6876 sub=Default opID=HB-host-20@604097-691c56bb]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-7515524, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002B3E43]
--> backtrace[03] libvmacore.so[0x002B3F1E]: Vmacore::PanicExit(char const*)
--> backtrace[04] vpxd[0x0072584A]
--> backtrace[05] vpxd[0x0073B18B]
--> backtrace[06] vpxd[0x0073D47B]
--> backtrace[07] vpxd[0x007427B2]
--> backtrace[08] libvmacore.so[0x0023AC47]
--> backtrace[09] libvmacore.so[0x0023AF2E]
--> backtrace[10] libvmacore.so[0x002BB737]
--> backtrace[11] libpthread.so.0[0x00007454]
--> backtrace[12] libc.so.6[0x000E8C1D]
--> [backtrace end]
Environment
VMware vSAN 6.x
Cause
This issue is caused by a 256 character limit on the column DEVICE_INFO_SUMMARY in the tables VPX_VM_VIRTUAL_DEVICE and VPX_VM_SN_VIRTUAL_DEVICE.
This limit is hit when inserting information about a snapshot for a VM that resides on VSAN storage.
This limit is hit when inserting information about a snapshot for a VM that resides on VSAN storage.
Resolution
This issue is resolved by increasing the length of the DEVICE_INFO_SUMMARY column in tables VPX_VM_VIRTUAL_DEVICE and VPX_VM_SN_VIRTUAL_DEVICE on the vCenter database.
- Log into the vCenter Service Appliance as root via SSH or Console.
- Shutdown Specific vCenter Server Services:
service-control --stop vmware-vpxd
service-control --stop vmware-content-library
service-control --stop vmware-content-library
- Connect to the vCenter Server PostgreSQL Database:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
- Run the following DDL commands on the vCenter Server PostgreSQL Database:
alter table VPX_VM_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY type varchar (512);
alter table VPX_VM_SN_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY type varchar (512);
For MSSQL Database:
alter table VPX_VM_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY nvarchar (512);
alter table VPX_VM_SN_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY nvarchar (512);
To exit the SQL DB run \q to quit
For MSSQL Database:
alter table VPX_VM_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY nvarchar (512);
alter table VPX_VM_SN_VIRTUAL_DEVICE alter column DEVICE_INFO_SUMMARY nvarchar (512);
To exit the SQL DB run \q to quit
- Add the following to /etc/vmware-vpx/vpxd.cfg in the <config> section:
<stressOption>
<vdb>
<schemaLoader>
<forceODBC>1</forceODBC>
</schemaLoader>
</vdb>
</stressOption>
<vdb>
<schemaLoader>
<forceODBC>1</forceODBC>
</schemaLoader>
</vdb>
</stressOption>
- Start Specific vCenter Server Services
service-control --start vmware-vpxd
service-control --start vmware-content-library
service-control --start vmware-content-library
- Ensure the changes to vpxd.cfg from Step 5 have been removed.
Feedback
Yes
No
Powered by
