Search the VMware Knowledge Base (KB)
View by Article ID

Unable to add Active Directory users or groups to vCenter Server Appliance or vRealize Automation permissions (2127213)

  • 11 Ratings

Symptoms

When using the Active Directory (Integrated Windows Authentication) identity source from the vCenter Single Sign-On 5.5 (SSO), Platform Services Controller 6.0 (PSC), or vRealize Automation Identity Appliance, you experience these symptoms:
  • Attempting to browse and add users to the vCenter Server permissions (Local Permission: Hosts and Clusters > vCenter > Manage > Permissions, Global Permissions: Administration > Global Permissions) fails with the error:

    Cannot load the users for the selected domain

  • Attempting to browse users from your Active Directory Domain under the Users tab (Administration > Users and Groups) in the vCenter Server fails with the error:

    com.vmware.identity.idm.IDMException: Failed to establish server connection

  • Attempting to browse and add users to the vRealize Automation Center permissions fails with the error:

    System Exception

  • In the /var/log/vmware/sso/vmware-sts-idmd.log file on the vCenter Single Sign-On or Platform Services Controller, you see entries similar to:
[YYYY-MM-DDT<Time>Z vsphere.local        3572c5f8-e776-4049-8487-d94f68634a2f WARN ] [ServerUtils] cannot bind connection: [ldap://<Active Directory Domain Controller FQDN>, null]
[YYYY-MM-DDT<Time>Z vsphere.local        3572c5f8-e776-4049-8487-d94f68634a2f ERROR] [ServerUtils] cannot establish connection with uri: [ldap://<Active Directory Domain Controller FQDN>]
[YYYY-MM-DDT<Time>Z vsphere.local        3572c5f8-e776-4049-8487-d94f68634a2f INFO ] [ActiveDirectoryProvider] removeDcInfo - domain [<Active Directory Domain Name>], domainFQDN [<Active Directory Domain Controller FQDN>], domainIpAddress [<Active Directory Domain Controller IP]
[YYYY-MM-DDT<Time>Z vsphere.local        3572c5f8-e776-4049-8487-d94f68634a2f ERROR] [ActiveDirectoryProvider] Failed to get non-GC connection to domain <Active Directory Domain> - domain controller might be offline
com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40287][LW_ERROR_LDAP_LOCAL_ERROR][]
        at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.LdapSaslBind(LinuxIdmNativeAdapter.java:345)
        at com.vmware.identity.interop.ldap.LinuxLdapClientLibrary.ldap_sasl_bind_s(LinuxLdapClientLibrary.java:676)
        at com.vmware.identity.interop.ldap.LdapConnection.bindSaslConnection(LdapConnection.java:158)
        at com.vmware.identity.idm.server.ServerUtils.getLdapConnection(ServerUtils.java:297)
        at com.vmware.identity.idm.server.ServerUtils.getLdapConnectionByURIs(ServerUtils.java:215)

...
                                                                      [YYYY-MM-DDT<Time>Z vsphere.local        b77dc08e-9d6b-4386-af56-eee92feae7c6 WARN ] [ServerUtils] cannot bind connection: [ldap://<Active Directory Domain Controller FQDN>, null] 
                                                                      [YYYY-MM-DDT<Time>Z vsphere.local        b77dc08e-9d6b-4386-af56-eee92feae7c6 ERROR] [ServerUtils] cannot establish connection with uri: [ldap://<Active Directory Domain Controller FQDN>] 
                                                                      [YYYY-MM-DDT<Time>Z vsphere.local        b77dc08e-9d6b-4386-af56-eee92feae7c6 ERROR] [ActiveDirectoryProvider] Failed to get non-GC connection to domain <Active Directory Domain Name> in retry 
                                                                      com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40287][LW_ERROR_LDAP_LOCAL_ERROR][]
                                                                      at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.LdapSaslBind(LinuxIdmNativeAdapter.java:345)
                                                                      at com.vmware.identity.interop.ldap.LinuxLdapClientLibrary.ldap_sasl_bind_s(LinuxLdapClientLibrary.java:676)
                                                                      at com.vmware.identity.interop.ldap.LdapConnection.bindSaslConnection(LdapConnection.java:158)
                                                                      at com.vmware.identity.idm.server.ServerUtils.getLdapConnection(ServerUtils.java:297)

                                                                      ...

                                                                      [
                                                                      YYYY-MM-DDT<Time>Z vsphere.local        b77dc08e-9d6b-4386-af56-eee92feae7c6 ERROR] [IdentityManager] Failed to find person users [Criteria : searchString=, domain=markit.partners] in tenant [vsphere.local] 
                                                                      [YYYY-MM-DDT<Time>Z vsphere.local        b77dc08e-9d6b-4386-af56-eee92feae7c6 ERROR] [ServerUtils] Exception 'com.vmware.identity.idm.IDMException: Failed to establish server connection' 
                                                                      com.vmware.identity.idm.IDMException: Failed to establish server connection
                                                                      ...
                                                                      Caused by: com.vmware.identity.idm.IDMException: Failed to get non-GC connection to domain <Active Directory Domain Name> in retry
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.getAdConnection(ActiveDirectoryProvider.java:2250)
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.getNonGcConnToDomain(ActiveDirectoryProvider.java:2218)
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.findUsersInternal(ActiveDirectoryProvider.java:2599)
                                                                      ... 18 more
                                                                      • In the /var/log/vmware/sso/vmware-sts-idmd.log file on the vRealize Automation Identity Appliance, you see entries similar to:
                                                                      [YYYY-MM-DD <Time> vsphere.local        9439b581-c839-4765-b8e7-39d3af448747 WARN ] [ServerUtils] cannot bind connection: [ldap://<Active Directory Domain Controller FQDN>, null] 
                                                                      [YYYY-MM-DD <Time> vsphere.local        9439b581-c839-4765-b8e7-39d3af448747 ERROR] [ServerUtils] cannot establish connection with uri: [ldap://<Active Directory Domain Controller FQDN>
                                                                      [YYYY-MM-DD <Time> vsphere.local        9439b581-c839-4765-b8e7-39d3af448747 ERROR] [IdentityManager] Failed to find person users [Criteria : searchString=po.tenant, domain=<Active Directory Domain Name>] in tenant [vsphere.local] 
                                                                      [YYYY-MM-DD <Time> vsphere.local        9439b581-c839-4765-b8e7-39d3af448747 ERROR] [ServerUtils] Exception 'com.vmware.identity.idm.IDMException: Failed to establish server connection' 
                                                                      com.vmware.identity.idm.IDMException: Failed to establish server connection
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.findUsersInternal(ActiveDirectoryProvider.java:2550)
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.findUsers(ActiveDirectoryProvider.java:465)
                                                                      at com.vmware.identity.idm.server.IdentityManager.findPersonUsers(IdentityManager.java:3405)
                                                                      at com.vmware.identity.idm.server.IdentityManager.findPersonUsers(IdentityManager.java:8415)
                                                                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                                                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                                                                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                                                      at java.lang.reflect.Method.invoke(Method.java:606)
                                                                      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
                                                                      at sun.rmi.transport.Transport$2.run(Transport.java:202)
                                                                      at sun.rmi.transport.Transport$2.run(Transport.java:199)
                                                                      at java.security.AccessController.doPrivileged(Native Method)
                                                                      at sun.rmi.transport.Transport.serviceCall(Transport.java:198)
                                                                      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)
                                                                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
                                                                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)
                                                                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)
                                                                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)
                                                                      at java.security.AccessController.doPrivileged(Native Method)
                                                                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)
                                                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                                                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                                                                      at java.lang.Thread.run(Thread.java:745)
                                                                      Caused by: com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40287][LW_ERROR_LDAP_LOCAL_ERROR][]
                                                                      at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.LdapSaslBind(LinuxIdmNativeAdapter.java:345)
                                                                      at com.vmware.identity.interop.ldap.LinuxLdapClientLibrary.ldap_sasl_bind_s(LinuxLdapClientLibrary.java:391)
                                                                      at com.vmware.identity.interop.ldap.LdapConnection.bindSaslConnection(LdapConnection.java:150)
                                                                      at com.vmware.identity.idm.server.ServerUtils.getLdapConnection(ServerUtils.java:299)
                                                                      at com.vmware.identity.idm.server.ServerUtils.getLdapConnectionByURIs(ServerUtils.java:217)
                                                                      at com.vmware.identity.idm.server.provider.BaseLdapProvider.getConnection(BaseLdapProvider.java:290)
                                                                      at com.vmware.identity.idm.server.provider.BaseLdapProvider.getConnection(BaseLdapProvider.java:76)
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.getNonGcConnToDomain(ActiveDirectoryProvider.java:2190)
                                                                      at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.findUsersInternal(ActiveDirectoryProvider.java:2545)
                                                                      ... 22 more

                                                                      Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

                                                                      Cause

                                                                      This issue occurs because the Likewise Kerberos stack requires all DNS servers to be configured with the Reverse Lookup Zone and that all Active Directory Domain Controller (AD DC) Pointer (PTR) records are available. The Likewise Kerberos stack in the Appliances use both Forward and Reverse Name Lookup to canonically organize hostnames for use in service principal names.

                                                                      Resolution

                                                                      To work around this issue, ensure that all DNS servers have the Reverse Lookup Zone configured and Active Directory Domain Controller (AD DC) Pointer (PTR) records present.

                                                                      Determining the DNS servers of vCenter Server or vRealize Automation Appliance

                                                                      1. Initiate an SSH connection to the vCenter Server or vRealize Automation Appliance.
                                                                      2. Enter the root username and password when prompted.

                                                                        Note: If you are using vSphere 6.0, run these commands to switch to the Bash shell:

                                                                        shell.set --enable True
                                                                        shell


                                                                      3. Run this command to review the DNS servers configured for the vCenter Server or vRealize Automation Appliance:

                                                                        less /etc/resolv.conf

                                                                        For example:

                                                                        nameserver 10.100.10.213
                                                                        nameserver 10.10.10.252


                                                                      Checking Active Directory Trust Enumeration

                                                                      To determine all trusts that are enumerated by the SSO 5.5, PSC 6.0, or Identity Appliance 6.x:
                                                                      1. Initiate an SSH connection to the SSO, PSC, or Identity Appliance.
                                                                      2. Enter the root user name and password when prompted.

                                                                        Note: If using vSphere 6.0, run the following command to switch to the Bash shell:

                                                                        shell.set --enable True
                                                                        shell


                                                                      3. Run this command to review all of the enumerated trusts from the Likewise Kerberos stack on the SSO, PSC, or Identity Appliance Appliance:

                                                                        less /var/lib/likewise/krb5-affinity.conf

                                                                        Note: This will output all of the trusts currently accessible from the SSO, PSC, or Identity Appliance. 

                                                                        You see output similar to:

                                                                        [realms]

                                                                            DomainA.local = {
                                                                                kdc = 10.10.10.213
                                                                            }
                                                                            DomainB.local = {
                                                                                kdc = 10.10.10.81
                                                                            }
                                                                            ChildDomainA.DomainB.local = {
                                                                                kdc = 10.10.10.85
                                                                            }
                                                                            ChildDomainB.DomainB.Local = {
                                                                                kdc = 10.10.10.83
                                                                            }
                                                                            DomainC.local = {
                                                                                kdc = 10.10.10.252
                                                                                kdc = 10.10.10.250
                                                                            }
                                                                            ChildDomainC.DomainB.local = {
                                                                                kdc = 10.10.10.247
                                                                                kdc = 10.10.10.82
                                                                            }

                                                                      4. Run this command to view a list of domain controllers that are not accessible from the Appliance:

                                                                        grep "cannot establish connection with uri:" /var/log/vmware/sso/vmware-sts-idmd.log | cut -d'[' -f4 | sort -nr | uniq -c

                                                                        Or

                                                                        grep "cannot establish connection with uri:" /var/log/vmware/sso/vmware-sts-idmd.log | cut -d'[' -f4 | uniq

                                                                        You see output similar to:

                                                                        ldap://localhost:389]
                                                                        ldap://dc2-root.DomainA.local]
                                                                        ldap://Vigrid.local]
                                                                        ldap://DC-4.DomainB.local]
                                                                        ldap://dc-us.DomainC.local]
                                                                        ldap://dc2-nh.DomainB.local]
                                                                        ldap://sqa-dc-3.DomainB.local]
                                                                        ldap://dc2-root.DomainA.local]
                                                                        ldap://DC-4.DomainB.local]


                                                                      Checking Active Directory Domain Controller DNS Resolution:

                                                                      1. Initiate an SSH connection to the Appliance.
                                                                      2. Enter the root username and password when prompted.

                                                                        Note: If you are using vSphere 6.0, run this command to switch to the Bash shell:

                                                                        shell.set --enable True
                                                                        shell


                                                                      3. Using nslookup from the Appliance, run this command to ensure there is DNS resolution for Forward Lookup for the Domain Controllers determined from the Checking Trust Enumeration section:

                                                                        nslookup dc2-root.DomainA.local

                                                                        Note: This command displays the IP address of the Domain Controller.

                                                                        You see output similar to:

                                                                        nslookup dc2-root.DomainB.local
                                                                        Server:         10.100.10.213
                                                                        Address:        10.100.10.213#53

                                                                        Non-authoritative answer:
                                                                        Name:   dc2-root.DomainB.local
                                                                        Address: 10.10.10.81


                                                                      4. To ensure that there is DNS resolution for Reverse Lookup for the domain controllers, run this command:

                                                                        nslookup 10.10.10.81

                                                                        If the Reverse Lookup is incorrect or missing, you will see output similar to:

                                                                        nslookup 10.10.10.81
                                                                        Server:         10.100.10.213
                                                                        Address:        10.100.10.213#53

                                                                        Non-authoritative answer:
                                                                        81.10.10.10.in-addr.arpa       name = <Incorrect FQDN>.

                                                                        Authoritative answers can be found from:

                                                                      5. Repeat Steps 1 to 4 for any additional Active Directory Domain Controllers to determine the records that are missing or incorrect.

                                                                      6. To resolve the issue when there are missing or incorrect records, use one of these options:
                                                                        • Option 1: Create or update the PTR record(s) for the Active Directory Domain Controller(s) on the listed DNS Servers from the Determining the Appliance's DNS Servers section.
                                                                        • Option 2: Update the DNS servers configured on the appliance to use DNS servers containing the correct PTR for your Active Directory Domain Controllers records. For more information, see Edit the DNS and IP Address Settings of the vCenter Server Appliance section in the vCenter Server Appliance Configuration guide.
                                                                        • Option 3: Add the missing Reverse Lookup records for the Active Directory Domain Controller(s) to the Appliance's /etc/hosts. For more information, see Editing files on an ESX host using vi or nano (1020302).

                                                                          Entries added to /etc/hosts file on the Appliance should be in the following format:

                                                                          IP_Address FQDN_of_Domain_Controller Short_Name_of_Domain_Controller

                                                                          For example:

                                                                          10.10.10.81 dc2-root.DomainB.local dc2-root

                                                                      See Also

                                                                      Request a Product Feature

                                                                      To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

                                                                      Feedback

                                                                      • 11 Ratings

                                                                      Did this article help you?
                                                                      This article resolved my issue.
                                                                      This article did not resolve my issue.
                                                                      This article helped but additional information was required to resolve my issue.

                                                                      What can we do to improve this information? (4000 or fewer characters)




                                                                      Please enter the Captcha code before clicking Submit.
                                                                      • 11 Ratings
                                                                      Actions
                                                                      KB: