Bootbanks mounts to /tmp folder and configuration changes are lost after rebooting of ESXi 6.0 Server

Products

VMware

Issue/Introduction

Symptoms:

Configuration changes on the ESXi is lost after rebooting the server.
Bootbank and altbootbank points to /tmp folder instead of pointing to the ESXi installed partition (this can be confirmed by running ls -l on root ("/") folder)

Cause

This issue happens if there is a delay in Fabric Login during the ESXi Boot process. By default, there is 3 Second timeout for Fabric login.

If Fabric Login takes more than 3 seconds, claiming the devices and discovering the LUNs happens parallelly and some LUNs will not be claimed on time during the boot process. If the boot LUN is not claimed before mounting the bootbank, ESXi boot process will mount bootbank and altbootbank to /tmp.

Resolution

VMware introduced a new boot configuration parameter on ESXi 6.0 Patch 4 to handle the delay in Fabric logins.

To resolve this issue, follow the steps:

To mount the bootbank to proper LUN, run this command on ESXi:
localcli --plugin-dir=/usr/lib/vmware/esxcli/int/ boot system restore --bootbanks
Upgrade ESXi to Patch 4 (Build number 4600944) available VMware Downloads.
Connect to the Server KVM Console.
During the ESXi boot process, press Shift+O and append the boot parameter devListStabilityCount=10 (its 10 Seconds, can be different value depending on flogi delay in the environment).
After the boot process, append the devListStabilityCount=10 entry in the /bootbank/boot.cfg file.

Note: This step is to ensure that change is persistent.

Additional Information

Following log from ESXi boot, HBA Link is UP and it took more than 3 seconds for the Fabric login completion (flogi succeeded) and Claim rules are enabled after 3rd Second :

2016-05-13T11:07:45.610Z cpu24:33363)<7>fnic : 1 :: link up

2016-05-13T11:07:45.610Z cpu24:33363)<6>host1: libfc: Link up on port ( 0)

2016-05-13T11:07:45.752Z cpu15:33365)<7>fnic : 2 :: link up

2016-05-13T11:07:45.752Z cpu15:33365)<6>host2: libfc: Link up on port ( 0)

2016-05-13T11:07:46.754Z cpu0:33369)<6>usb 1-1: suspended

2016-05-13T11:07:48.019Z cpu0:33315)ScsiClaimrule: 2463: Enabling claimrules for MPplugins.

2016-05-13T11:07:49.465Z cpu0:33351)<6>usb usb1: suspended

2016-05-13T11:07:49.620Z cpu26:33353)<6>host1: Assigned Port ID 16369

2016-05-13T11:07:49.620Z cpu26:33353)<7>fnic : 1 :: set port_id 16369 fp 0x439e41ccebc8

2016-05-13T11:07:49.620Z cpu26:33353)<6>host1: fip: received FLOGI LS_ACC using non-FIP mode

2016-05-13T11:07:49.620Z cpu26:33353)<7>fnic : 1 :: update_mac 0x439e41ccec41M

2016-05-13T11:07:49.620Z cpu26:33353)<7>fnic : 1 :: FLOGI reg issued fcid 16369 map 0 dest 0x43915249bcfaM

2016-05-13T11:07:49.622Z cpu11:33158)<7>fnic : 1 :: flog reg succeeded

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Build numbers and versions of VMware ESXi/ESX