Search the VMware Knowledge Base (KB)
View by Article ID

Determining why an ESXi/ESX host was powered off or restarted (1019238)

  • 44 Ratings

Symptoms

  • An ESXi/ESX host is disabled (grayed out) and displays as Not Responding.
  • An ESXi/ESX host is disabled (grayed out) and displays as Disconnected.
  • Clients connected to services running in one or more virtual machines are no longer accessible.
  • Applications dependent on services running in one or more virtual machines are reporting errors.
  • One or more virtual machines are no longer responding to network connections.

Purpose

This article provides steps to determine if an ESX or ESXi host was powered off or restarted.

Resolution

Note: This article assumes that you have completed the steps described in Troubleshooting an unresponsive host and multiple Disconnected virtual machines (1019082).

ESX 4.x

To determine the reason for abrupt shut down or reboot an ESX host:
  1. If the host is currently turned off, turn the host back on.

  2. Ensure that there are no hardware lights that may indicate a hardware issue. For more information, engage the hardware vendor.

  3. Log in to the host at the console as the root user.

  4. Run the command:

    # cat /var/log/vmksummary

  5. Determine if the ESX host was deliberately rebooted. When a user or script reboots a VMware ESX host, it generates a series of events under /var/log/vmksummary similar to:

    localhost logger: (1265803308) hb: vmk loaded, 1746.98, 1745.148, 0, 208167, 208167, 0, vmware-h-59580, sfcbd-7660, sfcbd-3524
    localhost vmkhalt: (1268148282) Rebooting system...
    localhost vmkhalt: (1268148374) Starting system...
    localhost logger: (1268148407) loaded VMkernel

    Hostd: [<YYYY-MM-DD> <TIME>.284 27D13B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.HostSystem.reboot-50


    If your ESX host is deliberately restarted, review the vCenter Server logs to identify any recent tasks that may have made the ESX host to reboot. These are a list of other resources that help determine the reason for reboot of an ESX host:

  6. Determine if the VMware ESX host was deliberately shut down. When a user or script shuts down a VMware ESX host, it generates a series of events similar to:

    localhost logger: (1265803308) hb: vmk loaded, 1746.98, 1745.148, 0, 208167, 208167, 0, vmware-h-59580, sfcbd-7660, sfcbd-3524
    localhost vmkhalt: (1268149354) Halting system...
    localhost vmkhalt: (1268149486) Starting system...
    localhost logger: (1268149540) loaded VMkernel


    If your VMware ESX host is deliberately shut down, review the vCenter Server logs to identify any recent tasks that may have told the VMware ESX host to reboot. Use this list of other resources to help determine the reason for shutting down of VMware ESX host:

    • For information on tracking user login and activities, see Tracking ESX host user logins and activities (1010026).
    • Third-party products that reside in the service console or using the VMware vSphere API may are able to manipulate the functionality of the VMware ESX host. For more information about third-party software in the service console, see Third-Party Software in the Service console.
    • If a server hardware watchdog timer is enabled, it may automatically reboot the ESX host if it detects that the operating system is unresponsive. For more details about your server hardware watchdog timer, consult the applicable software documentation and support. For more information about Hewlett Packard's server hardware watchdog, see HP Automatic Server Recovery in a VMware ESX Environment (1010842) and if necessary engage Hewlett Packard documentation and support.
    • Sometime virtual power shutdown/restart using an iLO on HP server can be the reason if the ESX server rebooted or shutdown.

  7. Determine if the ESX host experienced a kernel error. When an ESX host experiences a kernel error, it generates a series of events similar to:

    vsphere5 logger: (1251788469) hb: vmk loaded, 3597562.98, 3597450.113, 13, 164009, 164009, 356, vmware-h-79976, vpxa-54148, sfcbd-12600
    vsphere5 vmkhalt: (1251797195) Starting system...
    vsphere5 logger: (1251797206) VMkernel error
    vsphere5 logger: (1251797261) loaded VMkernel


    If your ESX host has experienced a kernel error, see Interpreting an ESXi/ESX host purple diagnostic screen (1004250).

  8. Run this command to check if the ESXi host is configured to automatically reboot after a purple diagnostic screen:

    esxcfg-advcfg -g /Misc/BlueScreenTimeout

    If the value is different than 0, then the ESXi host reboots automatically after a purple diagnostic screen.

    For more information, see Configuring an ESX/ESXi host to restart after becoming unresponsive with a purple diagnostic screen (2042500).

    When the host is rebooted after a failure and if the core dump is successful, the /var/log/vmksummary.log shows that a core dump is found.

    For example:

    <YYYY-MM-DD>T<TIME>Z bootstop: Host has booted
    <YYYY-MM-DD>T<TIME>Z bootstop: file core dump found


    Note: The preceding information indicates the ESXi host failure rather than indicating the ESXi host is restarted automatically.

  9. Determine if the VMware ESX host hardware abruptly rebooted. When the VMware ESX host hardware abruptly reboots, it generates a series of events similar to:

    localhost logger: (1265803308) hb: vmk loaded, 1746.98, 1745.148, 0, 208167, 208167, 0, vmware-h-59580, sfcbd-7660, sfcbd-3524
    localhost vmkhalt: (1268149486) Starting system...
    localhost logger: (1268149540) loaded VMkernel


    If your VMware ESX host has experienced an outage and it was not the result of a kernel error, deliberate reboot, or shut down, then the physical hardware may have abruptly restarted on its own. Hardware is known to reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor.

  10. Alternatively, the outage may have been deliberately triggered by an administrator by physically pressing the power button to turn off the hardware or using the hardware tools such as iLO, DRAC, RAS, etc. This occurrence may generate this event in the /var/log/vmkernel log of the ESX host:

    VMKAcpi: 1865: In PowerButton Helper

  11. If your VMware ESX host experiences an outage that is not the result of a kernel error, deliberate reboot, or shut down, then the physical hardware may have abruptly restarted on its own. Hardware may reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor.

    Alternatively, if an administrator has physically turned off or restarted the physical hardware because the console is not responding to user interaction, see Determining why an ESXi/ESX host does not respond to user interaction at the console (1017135).

    Notes:
    • This message is also logged when the server is powered down through the System Management Interface (such as HP iLO).
    • If the server is powered off by pressing the power button and the button is held for more than 10 seconds, this event is not logged.

    If an administrator has physically turned off or restarted the physical hardware because the console was not responding to user interaction, see Determining why an ESXi/ESX host does not respond to user interaction at the console (1017135)

ESXi 4.x/5.x/6.x

  1. To determine the reason for abrupt shut down or reboot of a VMware ESXi host:

    Note: By default, VMware ESXi logs do not persist upon a reboot. If a VMware ESXi host experiences an abrupt reboot due to reasons other than a VMkernel error, the logs do not persist and you do not have access to the logs prior to the reboot to determine the cause. The steps in this section assume that the VMware ESXi host is configured to redirect the logs to a location where the logs persist. For more information on how to configure a VMware ESXi host to redirect the logs to an alternate location, see Configure Syslog on ESXi Hosts in the Basic Administration Guide for your version of ESXi.

    1. If the ESXi host is currently turned off, turn the host back on.

    2. Ensure that there are no hardware lights that may indicate a hardware issue. For more information, engage the hardware vendor.

    3. Determine where the logs are being redirected to:

      1. Open vSphere Client.
      2. Connect to the ESXi host or vCenter Server managing the ESXi host.
      3. Provide the credentials of an administrative user.
      4. Select the ESXi host in the Inventory.
      5. Click the Configuration tab.
      6. Click Advanced Settings.
      7. In the Advanced Settings dialog, verify the location where the log files are being redirected:

        Note: If either of these settings are not properly configured, then logs do not persist upon a reboot and may limit the amount of information that can be gathered for troubleshooting.

        • Syslog > Local > Syslog.Local.DatastorePath contains the location of the logs if they are redirected to a VMFS volume.
        • Syslog > Remote > Syslog.Remote.Hostname contains the IP address or hostname of the syslog server that houses the logs for this host.

    4. Navigate to the location of the log files, and based on the modified date of the files, open the log file using your preferred editor.

    5. Determine if the ESXi host was deliberately restarted. If an ESXi host was restarted deliberately, the /var/log/hostd.log file will contain events similar to these:

      • Hostd: [12:51:54.284 27D13B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.HostSystem.reboot-50

        or

      • DCUI: reboot

      Note: In ESXi 5.5, these entries will be in /var/log/shell.log.

      If your host is deliberately shut down, review the vCenter Server logs to identify any recent tasks that may have made the host to power off.

    6. Determine if the ESXi host was deliberately shut down. If an ESXi server was shut down deliberately, it contains an event similar to:

      • Hostd: [<YYYY-MM-DD> <TIME>.550 2FEDEB90 info 'TaskManager'] Task Created : haTask-ha-host-vim.HostSystem.shutdown-78

        or

      • DCUI: poweroff

      If your host is deliberately shut down, review the vCenter Server logs to identify any recent tasks that may have made the host to power off.

      ESXi 5.x may also include PowerButton Helper events in the vmkernel.log file, similar to:

      T02:04:13.069Z cpu6:8222)VMKAcpi: 217: In PowerButton Helper

    7. Verify whether the virtual machine or ESXi host has generated a core dump:

      1. Log in to Tech Support mode. For more information, see Tech Support Mode for Emergency Support (1003677).

      2. ESXi hosts do not automatically collect the core dumps. To collect the core dump, manually run the esxcfg-dumppart command. For more information, see Extracting a core dump file from the VMKCore diagnostic partition following a purple diagnostic screen error (1002769).

        Note: Not configuring a core dump partition could interfere with the analysis of the abrupt reboots. For information on setting up a core dump partition, see Configuring an ESXi/ESX host to capture a VMkernel coredump from a purple diagnostic screen (1000328).

      3. If your VMware ESXi host has experienced a kernel error, see Interpreting an ESX host purple diagnostic screen (1004250).

    8. Check if ESXi is configured to automatically reboot after a purple screen by executing this command:

      esxcfg-advcfg -g /Misc/BlueScreenTimeout

      If the value is different than 0, then ESXi reboots automatically after the purple screen.

      For more details: Configuring an ESX/ESXi host to restart after becoming unresponsive with a purple diagnostic screen (2042500)

      When the host is rebooted after a crash and if the core dump was successful, the /var/log/vmksummary.log shows that a core dump is found.

      For example:
      <YYYY-MM-DD>T<TIME>Z bootstop: Host has booted
      <YYYY-MM-DD>T<TIME>Z bootstop: file core dump found


      Note: This information does not necessarily means that ESXi restarted automatically but gives an indication when ESXi crashed.

    9. If your VMware ESXi host experiences an outage that is not the result of a kernel error, deliberate reboot, or shut down, then the physical hardware may have abruptly restarted on its own. Hardware may reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor.

      Alternatively, if an administrator has physically turned off or restarted the physical hardware because the console is not responding to user interaction, see Determining why an ESXi/ESX host does not respond to user interaction at the console (1017135).

    10. The ESXi 5.x log file /var/log/vmksummary.log contains information regarding ESXi host startup and shutdown and an hourly heartbeat with uptime and other metrics. For more information, see Format of the ESXi 5.0 vmksummary log file (2004566).

Additional Information

For all other ESXi 5.x log file locations, see Location of ESXi 5.0 log files (2004201).

See Also

This Article Replaces

1004594, 1004102

Update History

07/16/2012 - Updated the ESX section with the hostd.log error

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 44 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 44 Ratings
Actions
KB: