This issue is resolved in vSphere 4.1 Update3.
For prior releases, this issue occurs when the boot-time file system check utility (
fsck) for EXT-3 file systems cannot resolve a file system (by UUID) defined in
/etc/fstab.
Issues that can result in this may include:
- The default roll-back option is left enabled when a subsequent upgrade is being performed.
- The device not present during system boot.
- The unresolvable EXT file systems appear to reside on disks/devices that are initialized later during system boot (e.g. the last LUN).
Note: If you are experiencing an outage with virtual machines down, consider resolving the situation in a timely manner through the re-installation of VMware ESX. Troubleshooting may take more time than a re-installation, which is in the order of approximately 20 minutes.
Otherwise refer to instructions below for submission of information to VMware Technical Support for technical analysis.
Further troubleshooting is available in the shell:
- Confirm the UUIDs which were not resolvable, and remain so, by running fsck again without additional arguments. This information (or similar information) is displayed:
# fsck
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
esx-root: clean, 32953/641280 files, 414801/1281175 blocks
e2fsck 1.39 (29-May-2006)
/dev/sdt1: clean, 35/140832 files, 25323/281596 blocks
fsck.ext3: Unable to resolve 'UUID=34d192db-17eb-442e-9613-c5c24c6fa9fa'
e2fsck 1.39 (29-May-2006)
/dev/sdt6: clean, 31/250368 files, 27851/500220 blocks
e2fsck 1.39 (29-May-2006)
/dev/sdt7: clean, 22/250368 files, 16815/500220 blocks
- Record the UUID or UUIDs which failed to resolve. You may take a screen shot of your System Management Interface, take a picture, or write the values down.
- Confirm these same values in the /etc/fstab file.
# cat /etc/fstab
UUID=79815890-f11c-4907-80fe-d1cd6bf061f8 / ext3 defaults 1 1
UUID=45460133-027b-40b6-8b4d-e52aaf4c417f /boot ext3 defaults 1 2
None /dev/pts devpts defaults 0 0
/dev/cdrom /mnt/cdrom udf,iso9660 noauto,owner,kudzu,ro 0 0
/dev/fd0 /mnt/floppy auto noauto,owner,kudzu 0 0
None /proc proc defaults 0 0
None /sys sysfs defaults 0 0
UUID= 34d192db-17eb-442e-9613-c5c24c6fa9fa /var/log ext3 defaults,errors=panic 1 2
UUID=e32ec5f4-d795-414a-8d73-a2bb3ea86342 swap swap defaults 0 0
Note: Highlighted in blue is the mount point for the respective unresolvable UUID, in red.
- Verify what UUIDs the system is currently aware of by running the command:
# ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 Nov 9 14:36 45460133-027b-40b6-8b4d-e52aaf4c417f -> ../../sdm1
lrwxrwxrwx 1 root root 10 Nov 9 14:36 e32ec5f4-d795-414a-8d73-a2bb3ea86342 -> ../../sdr1
lrwxrwxrwx 1 root root 10 Nov 9 14:36 34d192db-17eb-442e-9613-c5c24c6fa9fa -> ../../sdr2
lrwxrwxrwx 1 root root 10 Nov 9 14:36 79815890-f11c-4907-80fe-d1cd6bf061f8 -> ../../sdr5
Notes: - This output reveals the UUID-to-partition relationship for all discovered EXT partitions in the system. Affected mount points or content can be associated using the previous step.
- It is possible in some environments that none of the known partitions reported by listing /dev/disk/by-uuid match the unresolved UUID. This is correctable; for additional instructions, proceed to the following sections and correct the content of the /etc/fstab file.
- Sometimes " ls -l /dev/disk/by-uuid" does not list the instance of the UUID available in /etc/fstab and you may find one device which has a UUID but no mount/label in /etc/fstab (e.g. /dev/sdb#). It may be the /tmp partition which is not resolved by ESXi host. After editing the /etc/fstab file to correctly reflecting the mount point for /tmp partition to the actual device instance ( /dev/sdb#), you can list that partition correctly with proper UUID.
(Note: Here in the example we referred /dev/sdb# but you need to verify the output of /etc/fstab to see the actual device instance with the number ( #) at the end.)
Workarounds
There are 2 recommended workarounds. Both workarounds involve the modification of the
/etc/fstab file. You can either:
- Generate a new UUID for the affected file system(s) and update /etc/fstab to match the new value(s).
- Update /etc/fstab to incorporate the correct UUID from the file system.
Applying a new UUID
Apply a new UUID to the EXT-3 file systems which fail to resolve and update the
/etc/fstab file.
- Run tune2fs against each Linux partition on the suspected disk device. For example:
# tune2fs -l /dev/sdr2 | grep UUID
Filesystem UUID: 34d192db-17eb-442e-9613-c5c24c6fa9fa
# tune2fs -U random /dev/sdr2
tune2fs 1.39 (29-May-2006)
# tune2fs -l /dev/sdr2 | grep UUID
Filesystem UUID: 25a18c70-ffcb-4b15-9d2d-1cfab1754d86
- Update /etc/fstab with the updated UUID. From earlier steps, /dev/sdr2 partition was determined to be the /var/log mount point:
- Remount the root partition in read-write mode:
# mount / -o remount,rw
- Open the /etc/fstab file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
- Search for, and change, the original UUID to the newly-generated UUID from earlier steps, above.
- Save the file and remount the root partition in read-only mode:
# mount / -o remount,ro
- Reboot the server:
# shutdown -r now
Applying a UUID to /etc/fstab
-
Remount the filesystem to a read/write state with the command:
# mount / -o remount,rw
- Open the /etc/fstab file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
- Comment out the line referring to the older ESX installation by putting a hash symbol (#) before it.
- Save the file and exit the editor.
- Reboot the ESX host.
- To prevent this issue from occurring again, after the system is booted, follow the steps in Clean Up the ESX Bootloader Menu After Upgrade in the vSphere Upgrade Guide.
Removing residual roll-back installation UUIDs in /etc/fstab
Perform these steps to remove past installations from the /etc/fstab file. This may be necessary if roll-back data is preventing the ESX host from starting successfully.
- Log into the server using the root password.
-
Remount the root filesystem in read-write mode with the command:
# mount -o remount,rw /
- Open the /etc/fstab file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
- Comment out or remove the line referring to the previous ESX installation by inserting a hash symbol (#) at the beginning of the line.
- Save the file and exit the editor.
- Reboot the ESX host.
- To prevent this issue from recurring, follow the steps in Clean Up the ESX Bootloader Menu After Upgrade in the vSphere Upgrade Guide after rebooting.