VMware ESX Server Host Agent is already running

Products

VMware vSphere ESXi

Issue/Introduction

The mgmt-vmware service is the service that starts hostd. Instead of starting hostd directly, mgmt-vmware runs a script called vmware-watchdog which in turn starts hostd as a child process. After vmware-watchdog starts hostd, it waits for (monitors) it and restarts hostd if hostd fails or generates debug/support info after a number of failures.

When mgmt-vmware starts successfully and hostd is running normally, ps shows the following processes:

The watchdog (parent process):

/bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u

The hostd (child process):

/usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u

Before mgmt-vmware tells vmware-watchdog to start hostd, it first tells vmware-watchdog to check if hostd is already running. mgmt-vmware does not tell vmware-watchdog to start hostd if vmware-watchdog reports that hostd is already running.

vmware-watchdog performs the following checks to see if hostd is already running:

Is there something in the /var/run/vmware/watchdog-hostd.PID file? When vmware-watchdog starts hostd, it writes the PID of the watchdog process to this file.
Is there a vmware-watchdog hostd monitoring process already running? It checks the ps output.

When mgmt-vmware service or hostd is stopped, vmware-watchdog deletes the watchdog-hostd.PID file and exits. Sometimes, if hostd or vmware-watchdog fails to exit cleanly, hostd terminates but vmware-watchdog may still be running or may not delete the watchdog-hostd.PID, therefore preventing the mgmt-vmware service from starting again.

Symptoms:

The mgmt-vmware service does not start because it incorrectly thinks that the Host Agent (hostd) is already running.
The following dialog appears when trying to start mgmt-vmware:

VMware ESX Server Host Agent is already running
Listing the running processes (ps aux) shows that hostd is not running.

Environment

VMware ESX Server 3.5.x
VMware ESX Server 3.0.x

Resolution

Note: Before you begin, refer to Restarting the Management agents on an ESX Server (1003490) for important information on restarting the mgmt-vmware service.

This fix only applies if hostd is not running.

Run the following command to confirm that hostd is not running:

ps -auxwww | grep vmware-hostd | egrep -v "watchdog|grep"

If hostd is not running, but mgmt-vmware (vmware-watchdog) thinks it is running:

Run the following commands to stop the vmware-watchdog hostd monitoring process:
- ps -auxwww | grep "watchdog -s hostd" | grep -v grep
- kill <PID returned from above command>
Run the following command to delete the watchdog-hostd.PID file:

rm /var/run/vmware/watchdog-hostd.PID
Run the following command to start hostd:

service mgmt-vmware start

Additional Information

To check if vmware-watchdog is monitoring a process you can use a command of the form: vmware-watchdog -r <tag>

# vmware-watchdog -r hostd

To stop a vmware-watchdog monitor process (does not stop the process being monitored) use a command of the form: vmware-watchdog -k <tag>

# vmware-watchdog -k hostd

Note: If the PID number stored in /var/run/vmware/watchdog-hostd.PID is not running, this command fails and does not delete the PID file. The file must then be deleted manually.

Note: This information can be applied to any process started by vmware-watchdog, including VPXA (vmware-vpxa service).