Configuration file has unexpected changes and during backup proxy virtual machine terminates
search cancel

Configuration file has unexpected changes and during backup proxy virtual machine terminates

book

Article ID: 317644

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides workaround to avoid the Proxy VM from entering into an invalid state.

Symptoms:
Backup proxy goes to invalid state during backup.
  • In the hostd.log file, you see entries similar to: 
2018-06-08T10:33:14.150Z info hostd[15A03B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-a861cb5a-1083-e83935c5e190/smf-xxxproxy-007/smf-xxxproxy-007.vmx opID=5953cf5e-3-a90a user=vpxuser:ADMIN\nwadmin_svc] State Transition (VM_STATE_ON -> VM_STATE_RECONFIGURING)
...
2018-06-08T10:33:14.167Z error hostd[15640B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-a861cb5a-1083-e83935c5e190/smf-xxxproxy-007/smf-xxxproxy-007.vmx opID=5953cf5e-3-a90a user=vpxuser:ADMIN\nwadmin_svc] Could not apply pre-reconfigure domain changes: Failed to add file policies to domain :171: world ID :0:Cannot allocate memory
...
2018-06-08T10:33:14.826Z info hostd[15640B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-a861cb5a-1083-e83935c5e190/smf-xxxproxy-007/smf-xxxproxy-007.vmx opID=5953cf5e-3-a90a user=vpxuser:ADMIN\nwadmin_svc] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_ON)
...
2018-06-08T10:35:53.120Z error hostd[15A44B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-a861cb5a-1083-e83935c5e190/smf-xxxproxy-007/smf-xxxproxy-007.vmx] Expected permission (3) for /vmfs/volumes/5ac38771-f9105baf-f368-c25883f002d0/smf-app-comm-dev-054 not found in domain 171
  • In the vmkernel.log file, you see entries similar to:
2018-06-08T05:40:53.264Z cpu49:68763 opID=4c6a367c)World: 12235: VC opID d03205b-b9-17ed maps to vmkernel opID 4c6a367c
2018-06-08T05:40:53.264Z cpu49:68763 opID=4c6a367c)WARNING: Heap: 3534: Heap domainHeap-54 already at its maximum size. Cannot expand.
  • Increasing the FilePolicyMaxPolicies to 500 from default 256 returns the same issue:             vsish -e set /config/VmkAccess/intOpts/FilePolicyMaxPolicies 500 
  • Virtual Machine terminates with the error message similar to:

    Configuration file has unexpected changes. Virtual machine has been terminated. Unregister and re-register virtual machine if the configuration file is correct.
 
  • To bring this VM up, you should restart the ESXi management agents or un-register and re-register the VM back to inventory in VC.
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Cause

A logical error leads to accumulation of entries in a kernel heap of a fixed size. For a proxy VM that performs numerous disk hot-add, eventually the heap space is exhausted. This results with incorrect policy for the VM and causes the management software to terminate the VM.

Resolution

This issue is resolved in ESXi 6.5 P03 and ESXi 6.7 U1. To download go to Customer Connect download


Workaround:
To work around this issue:
  1. Migrate the proxy VM periodically to clear up stale security policy rules.
  2. Reload the proxy VM after it is migrated by running the following command in the ESXi shell:
vim-cmd vmsvc/reload <vmid>

Where, <vmid> is the managed object ID of the VM on the host, and can be obtained using the vim-cmd vmsvc/getallvms command.


Powered by Wolken Software