Troubleshooting vSphere ESX Agent Manager (EAM) with NSX (2122392)
- Deploying vSphere Installation Bundles (VIBs) using VMware ESX Agent Manager (EAM) fails.
- EAM fails to access VIBs from NSX Manager.
- EAM showing error, VIB module for agent is not installed on host.
- EAM displays a Status of Alert.
- EAM agency is missing an already prepared cluster.
- Under Networking and Security, Installation Status showing Not Ready on the Clusters and Hosts, clicking on Resolve does not fix the issue.
- ESXi host VXLAN VIB installation fails.
- ESXi host fails to access VIBs from the vCenter Server. In the VC system events table, you see:
Event Message:'Installation of deployment unit failed, please check if ovf/vib urls are accessible, in correct format and all the properties in ovf environment have been configured in service attributes. Please check logs for details.', Module:'Security Fabric'
- In the /var/log/vmkernel.log file of the ESXi hosts, you see entries similar to:
An esxupdate error exception was caught:
esxupdate: ERROR: MetadataDownloadError
- In the /var/log/vmkernel.log file on the ESXi host, you see entries similar to:
<2015-09-14>T<10:15> .833Z cpu16:1138415)WARNING: vxlan: VDL2PortPropSet:672: Failed to set control plane property for port[0x4000016] on VDS[DvsPortset-0] : Would block
<2015-09-14>T<10:15> .833Z cpu16:1138415)WARNING: NetPort: 1431: failed to enable port 0x4000016: Would block
Note: The Would block string is the most definitive symptom of this issue. For more information, see Network connectivity issues after upgrading or repreparing ESXi hosts in a VMware NSX/VCNS environment (2107951)
- In the /var/log/netcpa.log file on the ESXi host, you see entries similar to:
Netcpa error: error netcpa[FFCFFB70] [Originator@6876 sub=Default] Failed to get host id, returned , len 0
- In the Host Preparation tab of the Installation section of NSX UI, the Installation Status of clusters is showing as Not Ready after the NSX Manager or EAM service is restarted.
- You see a spinning wheel for hours or days and the installation process never completes.
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
OverviewvSphere ESX Agent Manager (EAM) is an application that sits between NSX and vCenter. NSX Manager requests EAM prepares a given NSX Cluster by creating an Agency with Scope that covers that Cluster. EAM automates the process of deploying and managing vSphere ESX agents.
The services that ESX Agent Manager provides include out-of-the-box integration of agents with vSphere features such as DRS, AddHost, High Availability, DRM, and maintenance mode. All of these features can be difficult to integrate with manually. ESX Agent Manager also allows users to monitor the health of ESX agents, and blocks users from performing certain operations on ESX agents that might affect the virtual machines that use them. For example, ESX Agent Manager can prevent an ESX agent virtual machine from being powered off or moved from an ESX host that contains other virtual machines that use that agent.
ESX Agent Manager adds a Management tab to the three standard Solutions Manager tabs. The Management tab shows information about running agencies, lists any orphaned ESX agents, and shows logging information about the ESX agents that ESX Agent Manager manages.
- Agency: In EAM terms, defines what enhancements should be done on all hosts/clusters in the scope.
- Enhancements: In EAM terms, comprise installing an agent VM and/or VIB on each ESXi host in the agency scope.
- Agent: In EAM terms, is an enhancement on a given ESXi host defined by an Agency. There might be more than one Agency that cover the same host. In this case each one of them correspond to a different Agent on that particular host.
Host PreparationThere are 3 components for NSX ESXi host preparation:
- vCenter Server
- NSX Manager
- EAM (ESX Agent Manager)
The vCenter Server manages the Compute Infrastructure and is tightly connected to the NSX Manager. There is a 1 to 1 relationship between the NSX Manager and the vCenter Server.
The NSX Manager is responsible for the ESXi host preparation. It installs on the ESXi hypervisor the various vSphere Installation Bundles (VIBs) to enable VXLAN, Distributed Routing, Distributed Firewall(DFW) and a user world agent(UWA) used to communicate at the control plane level.
VIBs are packages of files that get installed in the Kernel space of the ESXi hypervisor.
Software components that provide Control Plane communication from the ESXi hypervisor to the NSX Manager and the Controller Cluster nodes are:
- RabbitMQ Message bus provides communication between the RMQ client (vsfwd process on the ESXi hypervisor) and the RMQ Server process hosted on the NSX Manager.
- User World Agent (UWA) process (netcpa on the ESXi hypervisor) establishes TCP over SSL communication channels to the Controller Cluster nodes. Controllers use this channel to populate the local MAC,ARP and VTEP tables to determine where workloads are connected in the deployed logical networks.
Host Preparation Workflow
- Connect NSX Manager to vCenter Sever/VCVA.
- Deploy NSX Controllers (If using Unicast,Hybrid mode or LDR).
- Host Prep.
- NSX Manager > EAM > vCenter Server > ESXi host
- Agency is created and EAM calls vCenter Server to install VIBs
- EAM initiates VIBs scanning on each agency create/update/delete and on ESXi host restart (for the sake of installing vibs on stateless hosts). The actual scanning is done by vSphere Update Manager. EAM calls VUM providing NSX VIBs. VUM determines the delta between the provided VIB metadata and the actual VIBs installed on the ESXi host. The result is used by EAM to request VIB install from VUM.
- VIBs will be accessed from the NSX Manager through the following URL:
- When agents and Agency are successfully deployed EAM reports GREEN health status.
EAM Agency/Agents Health Status
An agency and its agents each maintains a status field which can be either Red, Yellow, or Green.
- RED State: The RED health status is used to indicate that the solution must somehow intervene for EAM to proceed.
- YELLOW State: The YELLOW health status indicates that EAM is actively working on reaching a given goal state. The goal state can be one of Enabled, Disabled, or Undeployed . E.g. when a solution is first registered its status is YELLOW until EAM has deployed the solution's agents to all the specified compute resources. A solution does not need to intervene when EAM reports its EAM health status as YELLOW.
- GREEN State: The GREEN health status is used to indicate that a solution and all its agents have reached the desired goal state.
For an overview of the architecture of ESX Agent Manager and how it integrates with vCenter Server, see the vSphere ESX Agent Manager API Reference Documentation.
- Check the NSX for vSphere release notes for current releases to see if the problem has been resolved in a bug fix. Look under Install and Upgrade Issues. For more information, see the VMware NSX for vSphere Documentation page. In additional information, see the vSphere Release Notes.
- Ensure that DNS is configured correctly on the ESXi hosts and the environment. The VMware ESX Agent Manager (EAM) may fail to deploy all required VIBs during installation of network virtualization components on vSphere hosts during Host Preparation in NSX for vSphere 6.x. For more information, see Installation Status shows as Not Ready on the Host Preparation Tab under Installation object in NSX (2075600).
- Verify that all required ports are open between ESXi hosts, vCenter Server and NSX Manager, and are not being blocked by a Firewall. For more information, see Network Port Requirements for VMware NSX for vSphere (2079386). Also, see TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network components (1012382).
- Verify that there are no issues with the NSX Message Bus. For more information, see Understanding and troubleshooting Message Bus in VMware NSX for vSphere 6.x (2133897).
- If you are using VMware Update Manager, ensure that VUM is available and running. For more information, see Installing VXLAN Agent fails with ESX Agent Manager displaying the error: Agent VIB module not installed (2053782).
- Verify that the URL http://VC_IP/eam/mob/ is accessible. For more information, see ESXi hosts in NSX for vSphere 6.x fails to configure VXLAN Tunnel End Point when rebooted (2095767).
- Ensure that the EAM is able to access the VIBs from NSX Manager. Review the eam.log file and look for VIB inaccessible messages. Also check the vCenter Server registration in NSX Manager. For more information, see Configuring the SSO Lookup Service fails (2102041).
- Ensure that the ESXi hosts are able to access VIBs from the vCenter Server. You see this error: esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known in the esxupdate.log file. This can occur if the ESXi host is unable to access the vCenter Server's Fully Qualified Domain Name (FQDN). For more information, see Verifying the VMware vCenter Server Managed IP Address (1008030).
- Ensure that there is sufficient space in the bootbank. Some third-party custom image size is relatively big in size. In addition to the base image, extra VIBs are automatically installed on the environment base on the services used. As a result, if the total size of an image exceeds the maximum size of the bootbank, then the VIB install fails. For more information, see Installing/upgrading NSX VIBs fails with the error: ERROR: The pending transaction requires xxx MB free space, however, the maximum supported size is xxx MB (2144200).
- Verify that the cluster contains the correct agencies. If an EAM agency is missing on an already prepared cluster, new hosts can fail to prepare. To work around this issue, create a dummy cluster and prepare it. This forces NSX/EAM to update the configuration of all existing clusters, creating a new EAM Agency for the problematic cluster.
- Ensure there are no duplicate agencies. For more information, see Resolving duplicate ESX Agent Manager (EAM) agency for VMware NSX for vSphere 6.x (2111232).
Note: Starting with VMware NSX for vSphere 6.1.3, an alarm is raised when no agency is detected.
- Ensure that the ESXi hosts are rebooted after an upgrade. Failure to manually reboot the ESXi host marks them as red in the Installation Status.
Note: In a DRS enabled cluster, an ESXi host reboot can be initiated by clicking the Resolve option on the cluster.
- If you see errorCode =99 on the eam.log file, this means that EAM incorrectly mark Host Preparation as successful. For more information, see Network connectivity issues after upgrade in NSX/VCNS environment (2107951).
In VMware vSphere 6.0 Update 1, host scan operation by EAM fails but NSX host preparation status shows as green incorrectly as the returned unhandled error 99. In releases with a fix for this issue, EAM correctly reports that the host scan failed and raises a vibNotInstalled state. The following example message from the /var/log/eam.log file on the affected ESXi host shows the error code 99.
INFO | 2015-02-25 11:41:32,221 | host-4 | VcPatchManager.java | 356 | Scan result on VcHostSystem(host-383):
errorCode = 99,
vibUrl = 'https://172.16.224.232/bin/vdn/vibs/5.5/vxlan.zip',
responseXml = <unset>,
bulletins = ,
- If you see the message Failed to complete VIB task on the eam.log file, this means that the vSphere Update Manager (VUM) is having issues installing NSX VIBs. To resolve this issue, restart the vSphere Update Manager service. For more information, see Stopping, starting, or restarting the vSphere Update Manager service (1039328).
- If you see the error esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known) in the esxupdate.log file, this may happen if the ESXi host cannot reach the vCenter Server Fully Qualified Domain Name (FQDN). To resolve this issue, set the correct the vCenter Server Managed IP address. For more information, see Verifying the VMware vCenter Server Managed IP Address (1008030).
- Sometimes after restarting NSX Manager, EAM or vCenter Server, you notice the cluster status reported Not Ready in Host Preparation tab, in Installation section of NSX Manager. This is a false-positive status that is a result of restarting one of the components. To get the state updated, click the Resolve All button.
- Ensure that the EAM service is aware when the vCenter Server certificate has been replaced. When the EAM service is not aware of the new vCenter Server certificate , it is not able to properly log in and displays the error similar to:
2016-06-03T16:53:02.772-07:00 | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password. For more information, see After replacing the vCenter Server certificates in VMware vSphere 6.0, the ESX Agent Manager solution user fails to log in (2112577).
- If installation never completes after a few hours or days, verify that the EAM and Web Management Services are running. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895). Also, see Stopping, starting or restarting VMware vCenter Server Appliance 6.0 services (2109887).
- VMware vSphere 6.0 supports VIB downloads over port 443 (instead of port 80). This port is opened and closed dynamically. The intermediate devices between the ESXi hosts and vCenter Server must allow traffic using this port.
- Ensure that a manual reboot is done on the ESXi hosts after an upgrade to the VIBs. Failure to manually rebooting the ESXi host will mark it as RED under the Installation Status. In a DRS enabled cluster, a host reboot can be initiated by clicking the Resolve option on the cluster. For more information, see Network connectivity issues after upgrading or repreparing ESXi hosts in a VMware NSX/VCNS environment (2107951).
- For Stateless ESXi host, the VIBS are included as part of Auto Deploy image. For more information, see In VMware NSX for vSphere 6.x, unpreparing Stateless ESXi host fails with the error: Agent VIB module is not installed. Cause : 15 The installation transaction failed. The transaction is not supported (2144732).
If problem still persists after performing the steps in this article, see:
- Gather the VMware Support Script Data. For more information, see Collecting diagnostic information for VMware products (1008524).
- File a support request with VMware Support and note this KB Article ID (2122392) in the problem description. For more information, see How to file a Support Request in My VMware (2006985).
vSphere Installation Bundles (VIBs)When you prepare an ESXi host for VMware NSX for vSphere, vSphere Installation Bundles (VIBs) are automatically pushed by the NSX Manager through VMware vSphere ESX Agency Manager (EAM). On the ESXi hosts, you see these VIBs installed:
- Starting with VMware NSX for vSphere 6.2.0, the esx-dvfilter-switch-security VIB is now included in esx-vxlan VIB. For more information, see Deploying VXLAN through Auto Deploy and VMware NSX for vSphere 6.x (2092871).
- VIBs URL will change for every release of NSX Manager.
EAM Log location
VMware vSphere 5.1.x/5.5.x (EAM is a part of the common tomcat server):
- C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\eam.log
- C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd-*.log - VC logs
- C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\localhost_access_log_*.log - common tomcat logs
Same as Windows 2003, but the VC log dir is C:\ProgramData\VMware\VMware VirtualCenter\Logs\
vCenter Server Virtual Appliance (VCVA):
- /storage/log/vmware/vpx/vpxd-*.log - VC logs
- /storage/log/vmware/vpx/tomcat/logs/localhost_access_log_*.log - common tomcat logs
VMware vSphere 6.x (EAM is a standalone service and has embedded tcserver):
- C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\eam\eam.log
- C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\eam\wrapper.log - tanuki Java service wrapper log
- C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\eam\web - embedded tcserver logs
- C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\firstboot\eam_firstboot.py_*.log - EAM firstboot script logs
- C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\vmware-vpx\vpx\vpxd-*.log - VC logs
Same as Windows 2003, but the VC log dir is C:\ProgramData\VMware\VMware VirtualCenter\Logs\
- /storage/log/vmware/eam/wrapper.log - tanuki Java service wrapper log
- /storage/log/vmware/eam/web - embedded tcserver logs
- /var/log/firstboot/eam_firstboot.py_*.log - EAM firstboot script logs
- /storage/log/vmware/vpx - VC logs
Tomcat/tc-server locationVMware vSphere 5.1.x/5.5.x:
VMware vCenter server Virtual Appliance (VCVA) :
EAM Service locationVMware vSphere 5.1.x/5.5.x
VMware vSphere 6.x, RPM: vmware-eam
How to remove EAM service from common tomcat/tcservervSphere 5.1.x/5.5.x
- Move EAM web application from TOMCAT_DIR\webapps to SOME_BACKUP_DIR which is not under TOMCAT_DIR\webapps
- Stop tcserver instance : TOMCAT_DIR\bin\tcruntime-ctl.bat stop tomcat
- Start tcserver instance : TOMCAT_DIR\bin\tcruntime-ctl.bat start tomcat
- Move EAM web application from TOMCAT_DIR\webapps to SOME_BACKUP_DIR which is not under <TOMCAT_DIR>\webapps
- Stop tcserver instance : TOMCAT_DIR/bin/tcruntime-ctl.sh stop tomcat
- Start tcserver instance : TOMCAT_DIR\bin\tcruntime-ctl.bat start tomcat
EAM logs should be checked when installation of the VIBs module fails:
/storage/log/vmware/vpx/eam.log on Linux vCenter
ProgramData/VMware/VMware VirtualCenter/Logs/ on Windows vCenter