ESX 4.x host fails to boot after power operation with the error: fsck.ext3: Unable to resolve UUID
search cancel

ESX 4.x host fails to boot after power operation with the error: fsck.ext3: Unable to resolve UUID

book

Article ID: 305991

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • After power-cycling or rebooting an ESX 4.x server, this error is produced during boot:

    fsck.ext3: Unable to resolve 'UUID=34d192db-17eb-442e-9613-c5c24c6fa9fa'

    And

    *** An error occurred during the file system check.
    *** Dropping you to a shell; the system will reboot
    *** when you leave the shell.

  • After encountering this error, you are unable to boot into ESX or Troubleshooting mode.
  • The unresolvable EXT file systems or partitions most commonly later appear to have mount points such as /var, /opt and /tmp.


Environment

VMware ESX 4.0.x
VMware ESX 4.1.x

Resolution

This issue is resolved in vSphere 4.1 Update3.

For prior releases, this issue occurs when the boot-time file system check utility ( fsck) for EXT-3 file systems cannot resolve a file system (by UUID) defined in /etc/fstab.

Issues that can result in this may include:
  • The default roll-back option is left enabled when a subsequent upgrade is being performed.
  • The device not present during system boot.
  • The unresolvable EXT file systems appear to reside on disks/devices that are initialized later during system boot (e.g. the last LUN).
Note: If you are experiencing an outage with virtual machines down, consider resolving the situation in a timely manner through the re-installation of VMware ESX. Troubleshooting may take more time than a re-installation, which is in the order of approximately 20 minutes.

Otherwise refer to instructions below for submission of information to VMware Technical Support for technical analysis.

Further troubleshooting is available in the shell:
  1. Confirm the UUIDs which were not resolvable, and remain so, by running fsck again without additional arguments. This information (or similar information) is displayed:

    # fsck

    fsck 1.39 (29-May-2006)

    e2fsck 1.39 (29-May-2006)
    esx-root: clean, 32953/641280 files, 414801/1281175 blocks
    e2fsck 1.39 (29-May-2006)
    /dev/sdt1: clean, 35/140832 files, 25323/281596 blocks
    fsck.ext3: Unable to resolve 'UUID=34d192db-17eb-442e-9613-c5c24c6fa9fa'
    e2fsck 1.39 (29-May-2006)
    /dev/sdt6: clean, 31/250368 files, 27851/500220 blocks
    e2fsck 1.39 (29-May-2006)
    /dev/sdt7: clean, 22/250368 files, 16815/500220 blocks

  2. Record the UUID or UUIDs which failed to resolve. You may take a screen shot of your System Management Interface, take a picture, or write the values down.

  3. Confirm these same values in the /etc/fstab file.

    # cat /etc/fstab

    UUID=79815890-f11c-4907-80fe-d1cd6bf061f8 / ext3 defaults 1 1
    UUID=45460133-027b-40b6-8b4d-e52aaf4c417f /boot ext3 defaults 1 2
    None /dev/pts devpts defaults 0 0
    /dev/cdrom /mnt/cdrom udf,iso9660 noauto,owner,kudzu,ro 0 0
    /dev/fd0 /mnt/floppy auto noauto,owner,kudzu 0 0
    None /proc proc defaults 0 0
    None /sys sysfs defaults 0 0
    UUID= 34d192db-17eb-442e-9613-c5c24c6fa9fa /var/log ext3 defaults,errors=panic 1 2
    UUID=e32ec5f4-d795-414a-8d73-a2bb3ea86342 swap swap defaults 0 0

    Note: Highlighted in blue is the mount point for the respective unresolvable UUID, in red.

  4. Verify what UUIDs the system is currently aware of by running the command:

    # ls -l /dev/disk/by-uuid

    total 0
    lrwxrwxrwx 1 root root 10 Nov 9 14:36 45460133-027b-40b6-8b4d-e52aaf4c417f -> ../../sdm1
    lrwxrwxrwx 1 root root 10 Nov 9 14:36 e32ec5f4-d795-414a-8d73-a2bb3ea86342 -> ../../sdr1
    lrwxrwxrwx 1 root root 10 Nov 9 14:36 34d192db-17eb-442e-9613-c5c24c6fa9fa -> ../../sdr2
    lrwxrwxrwx 1 root root 10 Nov 9 14:36 79815890-f11c-4907-80fe-d1cd6bf061f8 -> ../../sdr5

    Notes:
    • This output reveals the UUID-to-partition relationship for all discovered EXT partitions in the system. Affected mount points or content can be associated using the previous step.
    • It is possible in some environments that none of the known partitions reported by listing /dev/disk/by-uuid match the unresolved UUID. This is correctable; for additional instructions, proceed to the following sections and correct the content of the /etc/fstab file.
    • Sometimes " ls -l /dev/disk/by-uuid" does not list the instance of the UUID available in /etc/fstab and you may find one device which has a UUID but no mount/label in /etc/fstab (e.g. /dev/sdb#). It may be the /tmp partition which is not resolved by ESXi host. After editing the /etc/fstab file to correctly reflecting the mount point for /tmp partition to the actual device instance ( /dev/sdb#), you can list that partition correctly with proper UUID.

      (Note: Here in the example we referred /dev/sdb# but you need to verify the output of /etc/fstab to see the actual device instance with the number ( #) at the end.)

Workarounds

There are 2 recommended workarounds. Both workarounds involve the modification of the /etc/fstab file. You can either:
  • Generate a new UUID for the affected file system(s) and update /etc/fstab to match the new value(s).
  • Update /etc/fstab to incorporate the correct UUID from the file system.

Applying a new UUID

Apply a new UUID to the EXT-3 file systems which fail to resolve and update the /etc/fstab file.
  1. Run tune2fs against each Linux partition on the suspected disk device. For example:

    # tune2fs -l /dev/sdr2 | grep UUID
    Filesystem UUID: 34d192db-17eb-442e-9613-c5c24c6fa9fa

    # tune2fs -U random /dev/sdr2
    tune2fs 1.39 (29-May-2006)

    # tune2fs -l /dev/sdr2 | grep UUID
    Filesystem UUID: 25a18c70-ffcb-4b15-9d2d-1cfab1754d86

  2. Update /etc/fstab with the updated UUID. From earlier steps, /dev/sdr2 partition was determined to be the /var/log mount point:

    1. Remount the root partition in read-write mode:

      # mount / -o remount,rw

    2. Open the /etc/fstab file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
    3. Search for, and change, the original UUID to the newly-generated UUID from earlier steps, above.
    4. Save the file and remount the root partition in read-only mode:

      # mount / -o remount,ro

    5. Reboot the server:

# shutdown -r now

Applying a UUID to /etc/fstab

  1. Remount the filesystem to a read/write state with the command:

    # mount / -o remount,rw

  2. Open the /etc/fstab file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
  3. Comment out the line referring to the older ESX installation by putting a hash symbol (#) before it.
  4. Save the file and exit the editor.
  5. Reboot the ESX host.
  6. To prevent this issue from occurring again, after the system is booted, follow the steps in Clean Up the ESX Bootloader Menu After Upgrade in the vSphere Upgrade Guide.

Removing residual roll-back installation UUIDs in /etc/fstab

Perform these steps to remove past installations from the /etc/fstab file. This may be necessary if roll-back data is preventing the ESX host from starting successfully.

  1. Log into the server using the root password.
  2. Remount the root filesystem in read-write mode with the command:

    # mount -o remount,rw /

  3. Open the /etc/fstab file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
  4. Comment out or remove the line referring to the previous ESX installation by inserting a hash symbol (#) at the beginning of the line.
  5. Save the file and exit the editor.
  6. Reboot the ESX host.
  7. To prevent this issue from recurring, follow the steps in Clean Up the ESX Bootloader Menu After Upgrade in the vSphere Upgrade Guide after rebooting.


Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

To list partitions in the system, run:

# fdisk -l | less

Note: All known partitions are shown. In many cases where this issue has occurred, the resolved and unresolved partitions reside on the same block or disk device. Specifically, they most commonly reside on the same device as the ESX Console OS or the ESX host's system disk. For example:

System Disk
Disk /dev/sdm: 64.4 GB, 64424509440 bytes
64 heads, 32 sectors/track, 61440 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdm1 * 1 1100 1126384 83 Linux
/dev/sdm2 1101 1210 112640 fc VMware VMKCORE
/dev/sdm3 1211 61440 61675520 5 Extended
/dev/sdm5 1211 40960 40703984 fb VMware VMFS

Console OS Disk
Disk /dev/sdr: 7973 MB, 7973371904 bytes
255 heads, 63 sectors/track, 969 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdr1 1 76 610438+ 82 Linux swap / Solaris
/dev/sdr2 77 331 2048287+ 83 Linux
/dev/sdr3 332 969 5124735 5 Extended
/dev/sdr5 332 969 5124703+ 83 Linux

Notes:
  • In these steps, the chosen disks/partitions should typically be ones known to be utilized by the ESX host operating system; RDMs used by Linux virtual machines can be skipped.
  • ESX host system disks typically contain type FB (VMFS) or FC (VMKCore) partitions.
  • ESX Console OS disks usually appear to be relatively small, to the order of 8 GB or slightly larger.
  • Press q to quit the less utility when you are done reviewing partitioning information.
Enabling serial-line logging for an ESXi/ESXi host
Editing configuration files in VMware ESXi and ESX
El host ESX 4.x falla su proceso de inicio con el error: fsck.ext3: Unable to resolve UUID
How to file a Support Request in Customer Connect
電源操作後、ESX 4.x ホストを次のエラーで起動できない: fsck.ext3: Unable to resolve UUID
ESX 4.x 主机在电源操作后无法引导,并显示以下错误:fsck.ext3: 无法解析 UUID