vMotion of virtual machines fails with the error: "[msg.namespaceDb.badVersion] Incompatible version -1 (expect 2)"
Article ID: 328227
Updated On:
Products
VMware vSphere ESXi
VMware NSX Networking
Issue/Introduction
Symptoms:
In a vSphere 6.5 U1, or vSphere 6.7 environment with NSX for vSphere 6.4.1 and above with Guest Introspection, you experience these symptoms:
In a vSphere 6.5 U1, or vSphere 6.7 environment with NSX for vSphere 6.4.1 and above with Guest Introspection, you experience these symptoms:
- Migrating any new VMs may be impacted if the destination host has more than 170 VMs.
- The vmware.log file contains the following entries similar to:
2018-07-18T02:41:32.035Z| vmx| I125: MigrateSetState: Transitioning from state 11 to 12.
2018-07-18T02:41:32.035Z| vmx| I125: Migrate: Caching migration error message list:
2018-07-18T02:41:32.035Z| vmx| I125: [msg.checkpoint.migration.failedReceive] Failed to receive migration.
2018-07-18T02:41:32.035Z| vmx| I125: [msg.namespaceDb.badVersion] Incompatible version -1 (expect 2).
2018-07-18T02:41:32.035Z| vmx| I125: [msg.checkpoint.mrestoregroup.failed] An error occurred restoring the virtual machine state during migration.
2018-07-18T02:41:32.035Z| vmx| I125: Migrate: Caching migration error message list:
2018-07-18T02:41:32.035Z| vmx| I125: [msg.checkpoint.migration.failedReceive] Failed to receive migration.
2018-07-18T02:41:32.035Z| vmx| I125: [msg.namespaceDb.badVersion] Incompatible version -1 (expect 2).
2018-07-18T02:41:32.035Z| vmx| I125: [msg.checkpoint.mrestoregroup.failed] An error occurred restoring the virtual machine state during migration.
- In the /var/log/vmkernel.log file on the ESXi host, you see entries similar to:
2018-07-18T02:32:43.011Z cpu5:66134)WARNING: Heap: 3534: Heap fcntlHeap-1 already at its maximum size. Cannot expand.
2018-07-18T02:41:35.613Z cpu2:66134)WARNING: Heap: 4169: Heap_Align(fcntlHeap-1, 200/200 bytes, 8 align) failed. caller: 0x41800aaca9a3
2018-07-18T02:41:35.613Z cpu2:66134)WARNING: Heap: 4169: Heap_Align(fcntlHeap-1, 200/200 bytes, 8 align) failed. caller: 0x41800aaca9a3
Cause
The vMotions are failing because the fcntlHeap has been exhausted.
- On NSX for vSphere 6.4.1 and above, the user world component of Guest Introspection called EPSecMux aka Endpoint Multiplexor or Mux uses Namespace database to store its VM to solution (anti-virus) mapping.
- Namespaces will be created whenever a VM is created/moved to a host and deleted whenever it is removed/moved from a host. Also, whenever a VM connects, namespace database is read to restore protection to the VM immediately.
- Currently the namespace database is not big enough to handle large number of VMs moving to the host in a short interval of time such as during bulk vMotion.
Resolution
This issue is resolved for ESXi 6.5 in ESXi650-201811002, available at VMware Patch Downloads.
This issue is resolved for ESXi 6.7 in ESXi 6.7u2, available at VMware Downloads
This issue is resolved for ESXi 6.7 in ESXi 6.7u2, available at VMware Downloads
Feedback
Yes
No
Powered by
