3rd party Hyper Converged Infrastructure controller virtual machines (CVM) running on ESXi 6.7, 7.0.x and 8.0.x may experience soft lockups and experience unexpected reboots
search cancel

3rd party Hyper Converged Infrastructure controller virtual machines (CVM) running on ESXi 6.7, 7.0.x and 8.0.x may experience soft lockups and experience unexpected reboots

book

Article ID: 318511

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction


 


Symptoms:
  • 3rd party Hyper Converged Infrastructure controller virtual machines (CVM) running on ESXi 6.7/7.0 and 8.0 may experience soft lockups and experience unexpected reboots.
  • The OS log files will have messages similar to the below when these reboots occur.        "NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s!"
  • "vmware.log" for 3rd party Hyper Converged Infrastructure  controller VM's, will have messages similar to below:

 2023-01-29T08:07:15.232Z In(05) vcpu-0 - Vix: [vmxCommands.c:7182]:                 VMAutomation_HandleCLIHLTEvent. Do nothing.
 2023-01-29T08:07:15.232Z In(05) vcpu-0 - MsgHint: msg.monitorevent.halt
 2023-01-29T08:07:15.232Z In(05)+ vcpu-0 - The CPU has been disabled by the guest      operating system. Power off or reset the virtual machine.
 2023-01-29T08:07:15.232Z In(05)+ vcpu-0 - ---------------------------------------
 2023-01-29T08:07:15.235Z In(05) vcpu-0 - VigorTransportProcessClientPayload:     opID=vim-cmd-6d-8fa4 seq=49288: Receiving Bootstrap.MessageReply request.

NOTE: This KB is not be used for any other VM soft lockup issue.
NOTE:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice or statements made on third-party websites.Inclusion of such links does not imply that VMware endorses,recommends or accepts any responsibility for the content of such sites.


Environment

VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 8.0.x

Cause

There is a circular dependency between a resource provided by a VM and the ESXi host consuming this resource (NFS mount).

Resolution

This issue is mitigated in VMware vSphere ESXi 7.0 U3n and ESXi 8.0 U1c patches. To download go to the Customer Connect Patch Downloads page.
 


Workaround:

To workaround use one of the below steps  to disable VMX logging for 3rd party Hyper Converged Infrastructure controller VM's.

1. Disabling virtual machine logging (For 3rd party Hyper Converged Infrastructure controller VM's) using SSH. 

  • Power off the CVM virtual machine.
  • Navigate to  VM folder using command cd /vmfs/volumes/datastore/ServiceVM_Centos/
           Edit the VMX file using VI editor.  Add the following line in the .vmx file.
            logging = FALSE
  •  Power on the VM.


2. From GUI - To configure the disable logging option:

  • Login to VMware vSphere Client or ESXi host client and power off the virtual machine.
  • Right click the virtual machine and click Edit Settings.
  • Select the VM Options tab.
  • Click Advanced > General.
  • Under Settings, Uncheck the Enable logging option. (De-select the Enable logging option to disable.)
  • Power on the virtual machine.

Example:
 image.png


Additional Information

Impact/Risks:
The workaround will disable logging to vmware.log .But it has no noticeable impact on the VM behavior, especially no negative performance impact.

Powered by Wolken Software