Search the VMware Knowledge Base (KB)
View by Article ID

Troubleshooting vSphere ESX Agent Manager (EAM) with NSX (2122392)

  • 14 Ratings
Language Editions

Symptoms

  • Deploying vSphere Installation Bundles (VIBs) using VMware ESX Agent Manager (EAM) fails.
  • EAM fails to access VIBs from NSX Manager.
  • EAM showing error, VIB module for agent is not installed on host.
  • EAM displays a Status of Alert.
  • EAM agency is missing an already prepared cluster.
  • Under Networking and Security, Installation Status showing Not Ready on the Clusters and Hosts, clicking on Resolve does not fix the issue.
  • ESXi host VXLAN VIB installation fails.
  • ESXi host fails to access VIBs from the vCenter Server. In the VC system events table, you see:

Event Message:'Installation of deployment unit failed, please check if ovf/vib urls are accessible, in correct format and all the properties in ovf environment have been configured in service attributes. Please check logs for details.', Module:'Security Fabric'

  • In the /var/log/vmkernel.log file of the ESXi hosts, you see entries similar to:

    An esxupdate error exception was caught:
    esxupdate: ERROR: MetadataDownloadError
  • In the /var/log/vmkernel.log file on the ESXi host, you see entries similar to:

    <2015-09-14>T<10:15> .833Z cpu16:1138415)WARNING: vxlan: VDL2PortPropSet:672: Failed to set control plane property for port[0x4000016] on VDS[DvsPortset-0] : Would block
    <2015-09-14>T<10:15> .833Z cpu16:1138415)WARNING: NetPort: 1431: failed to enable port 0x4000016: Would block

    Note: The Would block string is the most definitive symptom of this issue. For more information, see Network connectivity issues after upgrading or repreparing ESXi hosts in a VMware NSX/VCNS environment (2107951)
  • In the /var/log/netcpa.log file on the ESXi host, you see entries similar to:

    Netcpa error: error netcpa[FFCFFB70] [Originator@6876 sub=Default] Failed to get host id, returned , len 0
  • In the Host Preparation tab of the Installation section of NSX UI, the Installation Status of clusters is showing as Not Ready after the NSX Manager or EAM service is restarted.
  • You see a spinning wheel for hours or days and the installation process never completes.

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Purpose

This article provides information on VMware vSphere ESX Agent Manager (EAM).

Overview

vSphere ESX Agent Manager (EAM) is an application that sits between NSX and vCenter. NSX Manager requests EAM prepares a given NSX Cluster by creating an Agency with Scope that covers that Cluster. EAM automates the process of deploying and managing vSphere ESX agents.

The services that ESX Agent Manager provides include out-of-the-box integration of agents with vSphere features such as DRS, AddHost, High Availability, DRM, and maintenance mode. All of these features can be difficult to integrate with manually. ESX Agent Manager also allows users to monitor the health of ESX agents, and blocks users from performing certain operations on ESX agents that might affect the virtual machines that use them. For example, ESX Agent Manager can prevent an ESX agent virtual machine from being powered off or moved from an ESX host that contains other virtual machines that use that agent.

ESX Agent Manager adds a Management tab to the three standard Solutions Manager tabs. The Management tab shows information about running agencies, lists any orphaned ESX agents, and shows logging information about the ESX agents that ESX Agent Manager manages.
  • Agency: In EAM terms, defines what enhancements should be done on all hosts/clusters in the scope.
  • Enhancements: In EAM terms, comprise installing an agent VM and/or VIB on each ESXi host in the agency scope.
  • Agent: In EAM terms, is an enhancement on a given ESXi host defined by an Agency. There might be more than one Agency that cover the same host. In this case each one of them correspond to a different Agent on that particular host.
To summarize, an Agency is comprised of agents, one per ESXi host from the scope.

Host Preparation

There are 3 components for NSX ESXi host preparation:
  • vCenter Server
  • NSX Manager
  • EAM (ESX Agent Manager)

The vCenter Server manages the Compute Infrastructure and is tightly connected to the NSX Manager. There is a 1 to 1 relationship between the NSX Manager and the vCenter Server.

The NSX Manager is responsible for the ESXi host preparation. It installs on the ESXi hypervisor the various vSphere Installation Bundles (VIBs) to enable VXLAN, Distributed Routing, Distributed Firewall(DFW) and a user world agent(UWA) used to communicate at the control plane level.

VIBs are packages of files that get installed in the Kernel space of the ESXi hypervisor.

Software components that provide Control Plane communication from the ESXi hypervisor to the NSX Manager and the Controller Cluster nodes are:

  • RabbitMQ Message bus provides communication between the RMQ client (vsfwd process on the ESXi hypervisor) and the RMQ Server process hosted on the NSX Manager.
  • User World Agent (UWA) process (netcpa on the ESXi hypervisor) establishes TCP over SSL communication channels to the Controller Cluster nodes. Controllers use this channel to populate the local MAC,ARP and VTEP tables to determine where workloads are connected in the deployed logical networks.

Host Preparation Workflow

  1. Connect NSX Manager to vCenter Sever/VCVA.
  2. Deploy NSX Controllers (If using Unicast,Hybrid mode or LDR).
  3. Host Prep.
  4. NSX Manager > EAM > vCenter Server > ESXi host
  5. Agency is created and EAM calls vCenter Server to install VIBs
  6. EAM initiates VIBs scanning on each agency create/update/delete and on ESXi host restart (for the sake of installing vibs on stateless hosts). The actual scanning is done by vSphere Update Manager. EAM calls VUM providing NSX VIBs. VUM determines the delta between the provided VIB metadata and the actual VIBs installed on the ESXi host. The result is used by EAM to request VIB install from VUM.
  7. VIBs will be accessed from the NSX Manager through the following URL:

    https://<NSXMGR_IP>/bin/vdn/vibs/5.5/vxlan.zip.
  8. When agents and Agency are successfully deployed EAM reports GREEN health status.

EAM Agency/Agents Health Status

An agency and its agents each maintains a status field which can be either Red, Yellow, or Green.

  • RED State: The RED health status is used to indicate that the solution must somehow intervene for EAM to proceed.
  • YELLOW State: The YELLOW health status indicates that EAM is actively working on reaching a given goal state. The goal state can be one of Enabled, Disabled, or Undeployed . E.g. when a solution is first registered its status is YELLOW until EAM has deployed the solution's agents to all the specified compute resources. A solution does not need to intervene when EAM reports its EAM health status as YELLOW.
  • GREEN State: The GREEN health status is used to indicate that a solution and all its agents have reached the desired goal state.

For an overview of the architecture of ESX Agent Manager and how it integrates with vCenter Server, see the vSphere ESX Agent Manager API Reference Documentation.

Resolution

IMPORTANT: Please note that this knowledge base article is no longer being updated. For the most up-to-date information, see the latest version of the NSX Troubleshooting Guide.

Validate that each troubleshooting step is true for your environment. Each step provides instructions or a link to an article to eliminate possible causes and take corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Please do not skip any step.
  1. Check the NSX for vSphere release notes for current releases to see if the problem has been resolved in a bug fix. Look under Install and Upgrade Issues. For more information, see the VMware NSX for vSphere Documentation page. In additional information, see the vSphere Release Notes.
  2. Ensure that DNS is configured correctly on the ESXi hosts and the environment. The VMware ESX Agent Manager (EAM) may fail to deploy all required VIBs during installation of network virtualization components on vSphere hosts during Host Preparation in NSX for vSphere 6.x. For more information, see Installation Status shows as Not Ready on the Host Preparation Tab under Installation object in NSX (2075600).
  3. Verify that all required ports are open between ESXi hosts, vCenter Server and NSX Manager, and are not being blocked by a Firewall. For more information, see Network Port Requirements for VMware NSX for vSphere (2079386). Also, see TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network components (1012382).
  4. Verify that there are no issues with the NSX Message Bus. For more information, see Understanding and troubleshooting Message Bus in VMware NSX for vSphere 6.x (2133897).
  5. If you are using VMware Update Manager, ensure that VUM is available and running. For more information, see Installing VXLAN Agent fails with ESX Agent Manager displaying the error: Agent VIB module not installed (2053782).
  6. Verify that the URL http://VC_IP/eam/mob/ is accessible. For more information, see ESXi hosts in NSX for vSphere 6.x fails to configure VXLAN Tunnel End Point when rebooted (2095767).
  7. Ensure that the EAM is able to access the VIBs from NSX Manager. Review the eam.log file and look for VIB inaccessible messages. Also check the vCenter Server registration in NSX Manager. For more information, see Configuring the SSO Lookup Service fails (2102041).
  8. Ensure that the ESXi hosts are able to access VIBs from the vCenter Server. You see this error: esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known in the esxupdate.log file. This can occur if the ESXi host is unable to access the vCenter Server's Fully Qualified Domain Name (FQDN). For more information, see Verifying the VMware vCenter Server Managed IP Address (1008030).
  9. Ensure that there is sufficient space in the bootbank. Some third-party custom image size is relatively big in size. In addition to the base image, extra VIBs are automatically installed on the environment base on the services used. As a result, if the total size of an image exceeds the maximum size of the bootbank, then the VIB install fails. For more information, see Installing/upgrading NSX VIBs fails with the error: ERROR: The pending transaction requires xxx MB free space, however, the maximum supported size is xxx MB (2144200).
  10. Verify that the cluster contains the correct agencies. If an EAM agency is missing on an already prepared cluster, new hosts can fail to prepare. To work around this issue, create a dummy cluster and prepare it. This forces NSX/EAM to update the configuration of all existing clusters, creating a new EAM Agency for the problematic cluster.
  11. Ensure there are no duplicate agencies. For more information, see Resolving duplicate ESX Agent Manager (EAM) agency for VMware NSX for vSphere 6.x (2111232).

    Note: Starting with VMware NSX for vSphere 6.1.3, an alarm is raised when no agency is detected.
        
  12. Ensure that the ESXi hosts are rebooted after an upgrade. Failure to manually reboot the ESXi host marks them as red in the Installation Status.
    Note: In a DRS enabled cluster, an ESXi host reboot can be initiated by clicking the Resolve option on the cluster.

  13. If you see errorCode =99 on the eam.log file, this means that EAM incorrectly mark Host Preparation as successful. For more information, see Network connectivity issues after upgrade in NSX/VCNS environment (2107951).

    In VMware vSphere 6.0 Update 1, host scan operation by EAM fails but NSX host preparation status shows as green incorrectly as the returned unhandled error 99. In releases with a fix for this issue, EAM correctly reports that the host scan failed and raises a vibNotInstalled state. The following example message from the /var/log/eam.log file on the affected ESXi host shows the error code 99.

    INFO | 2015-02-25 11:41:32,221 | host-4 | VcPatchManager.java | 356 | Scan result on VcHostSystem(host-383):
    ScanResult {
    errorCode = 99,
    vibUrl = 'https://172.16.224.232/bin/vdn/vibs/5.5/vxlan.zip',
    responseXml = <unset>,
    bulletins = [],


  14. If you see the message Failed to complete VIB task on the eam.log file, this means that the vSphere Update Manager (VUM) is having issues installing NSX VIBs. To resolve this issue, restart the vSphere Update Manager service. For more information, see Stopping, starting, or restarting the vSphere Update Manager service (1039328).
  15. If you see the error esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known) in the esxupdate.log file, this may happen if the ESXi host cannot reach the vCenter Server Fully Qualified Domain Name (FQDN). To resolve this issue, set the correct the vCenter Server Managed IP address. For more information, see Verifying the VMware vCenter Server Managed IP Address (1008030).
  16. Sometimes after restarting NSX Manager, EAM or vCenter Server, you notice the cluster status reported Not Ready in Host Preparation tab, in Installation section of NSX Manager. This is a false-positive status that is a result of restarting one of the components. To get the state updated, click the Resolve All button.
  17. Ensure that the EAM service is aware when the vCenter Server certificate has been replaced. When the EAM service is not aware of the new vCenter Server certificate , it is not able to properly log in and displays the error similar to:

    2016-06-03T16:53:02.772-07:00 | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password. For more information, see After replacing the vCenter Server certificates in VMware vSphere 6.0, the ESX Agent Manager solution user fails to log in (2112577).

  18. If installation never completes after a few hours or days, verify that the EAM and Web Management Services are running. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895). Also, see Stopping, starting or restarting VMware vCenter Server Appliance 6.0 services (2109887).

Notes:

If problem still persists after performing the steps in this article, see:

Additional Information

vSphere Installation Bundles (VIBs)

When you prepare an ESXi host for VMware NSX for vSphere, vSphere Installation Bundles (VIBs) are automatically pushed by the NSX Manager through VMware vSphere ESX Agency Manager (EAM). On the ESXi hosts, you see these VIBs installed:
  • esx-dvfilter-switch-security
  • esx-vsip
  • esx-vxlan

Notes:

    EAM Log location

    VMware vSphere 5.1.x/5.5.x (EAM is a part of the common tomcat server):

    Windows 2003:

    • C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\eam.log
    • C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd-*.log - VC logs
    • C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\localhost_access_log_*.log - common tomcat logs

    Windows 2008:

    Same as Windows 2003, but the VC log dir is C:\ProgramData\VMware\VMware VirtualCenter\Logs\

    vCenter Server Virtual Appliance (VCVA):

    • /storage/log/vmware/vpx/eam.log
    • /storage/log/vmware/vpx/vpxd-*.log - VC logs
    • /storage/log/vmware/vpx/tomcat/logs/localhost_access_log_*.log - common tomcat logs

    VMware vSphere 6.x (EAM is a standalone service and has embedded tcserver):

    Windows 2003:

    • C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\eam\eam.log
    • C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\eam\wrapper.log - tanuki Java service wrapper log
    • C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\eam\web - embedded tcserver logs
    • C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\firstboot\eam_firstboot.py_*.log - EAM firstboot script logs
    • C:\Documents and Settings\All Users\Application Data\VMware\CIS\logs\vmware-vpx\vpx\vpxd-*.log - VC logs

    Windows 2008:

    Same as Windows 2003, but the VC log dir is C:\ProgramData\VMware\VMware VirtualCenter\Logs\

    CloudVM:

    • /storage/log/vmware/eam/eam.log
    • /storage/log/vmware/eam/wrapper.log - tanuki Java service wrapper log
    • /storage/log/vmware/eam/web - embedded tcserver logs
    • /var/log/firstboot/eam_firstboot.py_*.log - EAM firstboot script logs
    • /storage/log/vmware/vpx - VC logs

    Tomcat/tc-server location

    VMware vSphere 5.1.x/5.5.x:

    Windows:

    C:\Program Files\VMware\Infrastructure\tomcat

    VMware vCenter server Virtual Appliance (VCVA) :

    /usr/lib/vmware-vpx/tomcat.

    EAM Service location

    VMware vSphere 5.1.x/5.5.x

    Windows:

    TOMCAT_DIR\webapps\eam

    VCVA:

    TOMCAT_DIR/webapps/eam

    VMware vSphere 6.x, RPM: vmware-eam

    Windows:

    C:\Program Files\VMware\CIS\eam

    VCVA:

    /usr/lib/vmware-eam

    How to remove EAM service from common tomcat/tcserver

    vSphere 5.1.x/5.5.x

    Windows:
    1. Move EAM web application from TOMCAT_DIR\webapps to SOME_BACKUP_DIR which is not under TOMCAT_DIR\webapps
    2. Stop tcserver instance : TOMCAT_DIR\bin\tcruntime-ctl.bat stop tomcat
    3. Start tcserver instance : TOMCAT_DIR\bin\tcruntime-ctl.bat start tomcat
    VCVA:
    1. Move EAM web application from TOMCAT_DIR\webapps to SOME_BACKUP_DIR which is not under <TOMCAT_DIR>\webapps
    2. Stop tcserver instance : TOMCAT_DIR/bin/tcruntime-ctl.sh stop tomcat
    3. Start tcserver instance : TOMCAT_DIR\bin\tcruntime-ctl.bat start tomcat
    Note: Due to some reason, after the tcserver instance stops it starts automatically again. Therefore, first move the EAM web application folder.

    EAM logs should be checked when installation of the VIBs module fails:

    /storage/log/vmware/vpx/eam.log on Linux vCenter
    ProgramData/VMware/VMware VirtualCenter/Logs/ on Windows vCenter

    EAM logs are included as part of the vCenter Server Log Bundle. For more information, see Collecting diagnostic information for VMware vCenter Server 4.x, 5.x and 6.0 (1011641).

    See Also

    Request a Product Feature

    To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

    Feedback

    • 14 Ratings

    Did this article help you?
    This article resolved my issue.
    This article did not resolve my issue.
    This article helped but additional information was required to resolve my issue.

    What can we do to improve this information? (4000 or fewer characters)




    Please enter the Captcha code before clicking Submit.
    • 14 Ratings
    Actions
    KB: