Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

ESX 4.0 may generate a purple diagnostics screen or kernel panic after an upgrade due to an insufficient disk space (1016511)

Symptoms

  • ESX 4.0 generates a purple diagnostics screen or kernel panic during reboot after upgrading to ESX 4.0 Update 1A when using VMware Update Manager or esxupdate.
  • The purple diagnostics screen contains the text:

    No place on disk to dump data

  • The esxcfg-boot log may contain entries similar to:

    2009-12-08 11:35:15 (32358) INFO : Replacing /etc/vmware/simple.map file...
    2009-12-08 11:35:15 (32358) INFO : "/sbin/vmware-mkinitrd -v -f '/tmp/vmware.0.tmp'"
    2009-12-08 11:35:15 (32358) INFO : [01;31m!!! ERROR:[00;00m Not enough space to create ramdisk
    2009-12-08 11:35:15 (32358) INFO : Command returned with non-zero status: exit status 1
    2009-12-08 11:35:15 (32358) ERROR: Unable to regenerate initrd.
    2009-12-08 11:35:15 (32358) INFO : "/bin/sync"
    2009-12-08 11:35:16 (32358) ERROR: /usr/sbin/esxcfg-boot failed, examine /var/log/vmware/esxcfg-boot.log for details.
  • The esxupdate.log may contain entries similar to:

    [2009-12-08 11:35:10] INFO: vibcache: Running [/usr/sbin/vmkmod-install.sh]...
    [2009-12-08 11:35:16] INFO: vibcache: 2009-12-08 11:35:15 (32358) ERROR: Unable to regenerate initrd.
    Unable to regenerate initrd, system in unknown state. Please check log files for errors and correct before rebooting.2009-12-08 11:35:16 (32358) ERROR: /usr/sbin/esxcfg-boot failed, examine /var/log/vmware/esxcfg-boot.log for details.
    [2009-12-08 11:35:16] ERROR: esxupdate: An esxupdate error exception was caught:
    Traceback (most recent call last):
      File "/usr/sbin/esxupdate", line 242, in ?
        cmd.Run()
      File "/usr/lib/vmware/python2.4/site-packages/vmware/esx4update/cmdline.py", line 382, in Run
        installer.Install(self.iface)
      File "/usr/lib/vmware/python2.4/site-packages/vmware/esx4update/VibCach

Resolution

The default installation of ESX 4 has a 5GB root partition ( / ). Occasionally, depending on service console activities, some new files might be uploaded to the service console's file system in addition to the default installation files, consuming more space. If there is not enough space left on the / partition, the upgrade fails.
 
The host system should have the following space available to ensure space for the installation:
  • A minimum of 36MB for the /tmp if it on a separated partition
  • A minimum of 24MB /boot directories.
  • A minimum of 100MB for the /root directory if it on a separated partition.
In general, the installation requires free space equivalent to approximately thrice that required for the downloaded bulletins. If you store the downloaded bulletin file on the service console, you need additional free space equivalent to thrice that of the bulletin.

For example, the ESX 4.0 Update 1 download is 774.4 MB in size, and requires 1.5 GB of free space to successfully complete the installation – or 1.75 GB total free space on the root (/) partition if you are storing the zip file on the local service console file system.
The ESX 4.1 Update 2 download is 670MB in size and requires 2012MB of disk space to install the update. ESX 4.1 Update 2 download requires more than twice the disk space required for the downloaded bulletins.

Upgrading From ESX 3.5 to ESX 4

If you are upgrading from ESX 3.5 to ESX 4, the service console might contain more date than the default ESX 4.0 service console can accommodate. To recover from this situation, you need to boot the system into troubleshooting mode and roll back to ESX 3.5. For more information on rolling back, see the vSphere Upgrade Guide. After roll back, verify the initrd image is deleted from /boot.

When you are able to successfully boot into ESX 3.5, see Performing an offline upgrade from ESX 3.x to 4.x (1009440) to perform an offline upgrade to ESX 4 with –v parameter (note v is lower case) to specify additional space for the ESX 4 vmdk file (service console). You may also choose to clean up any unnecessary files in the ESX 3.5 service console to avoid having to use the –v parameter during the off line upgrade. For more information, see the Freeing disk space on an ESX host in this article.

Upgrading from ESX 4 to ESX 4 Update 1

If you originally upgraded from ESX 3.5 to ESX 4.0, and have subsequently upgraded to ESX 4.0 U1, you may still have the option to roll back to ESX 3.5. In this instance, you can choose to perform a direct upgrade install from ESX 3.5 to ESX 4 Update 1. To do this you can boot into troubleshooting mode and roll back to ESX 3.5. For more information, see the vSphere Upgrade Guide. After the roll back, verify the initrd image is deleted from /boot.

When you have successfully booted into ESX 3.5, you can perform an Off Line Upgrade to ESX 4.0 U1. For more information, see Performing an offline upgrade from ESX 3.x to 4.x (1009440). During this upgrade you can choose to use the -v (note lower case) parameter to specify more space for the ESX 4 vmdk file (service console).

If you have booted into troubleshooting mode, and do not have the option to roll back to ESX 3.5, you can verify the space left on the partitions of your ESX host by running the df -h command in the service console as user root.

The output appears similar to:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb5 4.9G 1.7G 2.9G 37% /
/dev/cciss/c0d0p1 1.1G 75M 952M 8% /boot
/dev/sdb2 2.0G 54M 1.8G 3% /var/log
 
If there is not enough space available to perform the upgrade, a service console disk clean up is needed. For more information, see the Freeing disk space on an ESX host in this article.

After you have cleaned up sufficient space on the service console, you can place the ESX 4.0 Update 1 bundle on an http patch repository or burn it to a DVD and mount it in the service console.

Re-run the installation with the following command:

esxupdate --bundle=ESX-4.0.0-update01a.zip --nocache update

During installation you may receive the message, Skipping bulletin ESX400-XXXXX, it is installed or obsolete. This message can be safely ignored.

Reboot your host using the reboot command when the installation has completed.

If the ESX host still fails to boot successfully, you need to re-install ESX 4 Update 1 from DVD. When you are re-installing ESX 4 Update 1, consider using the partition table included in the Preventing the issue section of this article.

Freeing disk space on an ESX host

Freeing disk space can become a large task if there have not been recent changes to the ESX host.
Run the df -h command in the service console as a root user to determine what the status of your disk space is:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb5 4.9G 1.7G 2.9G 37% /
/dev/cciss/c0d0p1 1.1G 75M 952M 8% /boot
/dev/sdb2 2.0G 54M 1.8G 3% /var/log

There are a few common reasons that partitions become full:
  • Informational files, such as a virtual machine or an ISO, were copied to the service console file system
  • Log files have not yet been rotated after a substantial amount of information was written to them
  • A backup file was created by using third party backup application
There are also several common places to start for determining what is taking up the space on the volume:
  • The /vmimages folder. This folder is used to store operating system install files such as the VMware Tools or other ISO files.
  • The /var/core folder. This folder is used to store crash files for processes on the service console.
  • The /var/log folder. This folder stores the majority of the logs for the ESX host.
  • The /esx3-installation folder. This folder contains upgrade logs and old configuration files after a successful upgrade from ESX 3.5 to 4.0. For more information about post upgrade clean up, see the vSphere Upgrade Guide.
If you are unable to determine where the space is being used, you can run the command:
 
[root@server]# find / -size +10240000c -exec du -h {} \; | less
 
This command provides a list of all files that are larger than 10MB in size.
 
After you have determined what is taking up the free space you can use the rm command to free up space by deleting files.

Warning: When you delete a file there is no way to recover it, so use caution when deleting anything.
 
The following are a list of files that are safe to delete:
  • Old vm-support logs, usually under /usr/lib/vmware/hostd/docroot/downloads/
  • Virtual machines that are not needed and were copied to service console for storage
  • ISO files that were copied to the system. In many cases under the /vmimages folder
Note: Do not delete the VMware Tools ISO files.
 
If you are unsure about deleting a specific file, file a Support Request with VMware Support. If a system file is removed inadvertently this may cause damage to your ESX host that can require a re-installation of the software. For more information on contacting VMware Support, see Filing a Support Request in My VMware (2006985).

Preventing the issue

To prevent this issue from happening in the future, you can use the following partition table if you decide to perform a fresh install of ESX 4 Update 1:
Primary:
/ - 7168MB
/boot - 300MB
Swap - 1600MB

Extended Partition:
/var - 4096MB
/home - 2048MB
/opt - 2048MB
/tmp - 2048MB
 
The vmkcore partition is automatically created.
 
Note: This partition table recommendation does not mean it is better than the default partition table. It means that if there are a lot of service console activities, this partition table may prevent the insufficient disk space issue.

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 8 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 8 Ratings
Actions
KB: