"LDAP Error Code 49"/Error (49) error in vmdird logs in vCenter Server
search cancel

"LDAP Error Code 49"/Error (49) error in vmdird logs in vCenter Server

book

Article ID: 319348

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

How to reset the machine account password

Symptoms:
  • In the vmdird-syslog.log file, you see entries similar to:

    2016-09-21T18:47:48.024511+00:00 err vmdird t@140107551946496: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
    2016-09-21T18:47:48.024533+00:00 err vmdird t@140107551946496: VmDirSendLdapResult: Request (96), Error (49), Message ((49)(SASL step failed.)), (0) socket ([17] 10.105.217.85:389<-10.105.212.102:54753)
    2016-09-21T18:47:48.024538+00:00 err vmdird t@140107551946496: Bind Request Failed ([17] 10.105.217.85:389<-10.105.212.102:54753) error 49: Protocol version: 3, Bind DN: "cn=accountname,ou=Computers,dc=vsphere,dc=local", Method: 163


    Note: The vmdird-syslog.log file is located at:
    • vCenter Server Appliance with embedded PSC/PSC: /var/log/vmware/vmdird/vmdird-syslog.log
    • Windows installed vCenter Server with embedded PSC/PSC: "%VMWARE_LOG_DIR%"\vmdird\vmdir.log
Note: The vmdir log is not present in vCenters that do not have an embedded PSC.
 
Note: From 6.5 onwards inventory services is not available, For LDAP errors see /var/log/vmware/sso/vmware-sts-idmd.log  or /var/log/vmware/vmdird/vmdird-syslog.log.
          This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.
  • vmdir replication may not be working between vCenter with Embedded PSCs/External PSCs - nodes may be XXX changes behind from replication partners' point of view
Replication can be checked via below command (must be run on each VC/PSC in the SSO domain to accurately reflect the situation):

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

If a partner is changes behind, review the vmdird-syslog.log of both nodes for ldap 49 errors against those machines


Cause

This issue occurs when a machine loses its trust due to a password mismatch in the vmdird for the account listed in the vmdird-syslog.log file.
 
This can occur if the vCenter Server or PSC is restored to an earlier version from backups or an older snapshot.

Resolution

To resolve this issue, reset the password for the user account listed in the vmdird-syslog.log file.

Example:

If the vmdird-syslog.log entries for "error 49" were to look like this:
2016-09-21T18:47:48.024538+00:00 err vmdird t@140107551946496: Bind Request Failed ([17] 10.105.217.85:389<-10.105.212.102:54753) error 49: Protocol version: 3, Bind DN: "cn=VCVA01.vmware.local,ou=Computers,dc=vsphere,dc=local", Method: 163

For PSC nodes, the bind DN will appear in this format: Bind DN: "cn=VCVA01.vmware.local,ou=Domain Controllers,dc=vsphere,dc=local"

Then VCVA01.vmware.local would be the affected account.
The SSO domain, which we can also see in the error message, is vsphere.local

Therefore, the entire machine account name in this example will look like: 
[email protected]

 

1) Reset the machine account password manually:

  1. In Appliance 
  2. In Windows
 
Warning:
For vdadmintool correct default settings in the SSO password policies are required, VMware currently does not support to set the maximum password length above 20 characters.
 

 Appliance

  1. Create a snapshot of vCenter Server and Platform Services Controller. If you have multiple vCenters with embedded PSCs/PSCs in the SSO domain; ensure all machines in the SSO domain are offline, snapshot them all and if restoring; restore all before powering on any machines to ensure consistency in the case of a revert.
  2. Connect to the vCenter/PSC which has the ldap 49 error in it's vmdird-syslog.log with an SSH session and root credentials.
  3. Run this command to enable access the Bash shell:

    shell.set –enabled true
     
  4. Type shell and press Enter.
  5. Run this command to open the vdcadmintool:

    /usr/lib/vmware-vmdir/bin/vdcadmintool

    You can see these options:
    ================================
    Please select:
    0. exit
    1. Test LDAP connectivity
    2. Force start replication cycle
    3. Reset account password
    4. Set log level and mask
    5. Set vmdir state
    ================================

     
  6. Select option 3.
  7. Enter the user account listed in the vmdird-syslog.log file.

    Note: This is the machine account in the format FQDN@SSO DOmain.

    For example:

    [email protected]

    Note: The tool does not filter out invalid characters from the generated password such as:
    & (ampersand)
    ; (semicolon)
    " (double quotation mark)
    ' (single quotation mark)
    ^ (circumflex)
    \ (backslash)
    % (percentage)

    You may have to keep running option 3 several times until you get a valid password.
     
  8. Make a note of the new auto-generated password.
  9. Connect to vCenter Server Appliance or PSC with an SSH session and root credentials.
  10. Run this command to enable access the Bash shell:

    shell.set –enabled true
     
  11. Type shell and press Enter.
  12. Run these commands to update the password:

    /opt/likewise/bin/lwregshell
    cd HKEY_THIS_MACHINE\services\vmdir\
    set_value dcAccountPassword "new password"
    quit

     
  13. Restart the vCenter Server Appliance services. For more information, see Stopping, starting, or restarting VMware vCenter Server Appliance 6.x services (2109887).

Windows installed vCenter Server

  1. Create a snapshot of the vCenter Server and Platform Services Controller. If you have multiple vCenters with embedded PSCs/PSCs in the SSO domain; ensure all machines in the SSO domain are offline, snapshot them all and if restoring; restore all before powering on any machines to ensure consistency in the case of a revert.
  2. Open the elevated command prompt on the vCenter with Embedded PSC/PSC.
  3. Run this command:

    %VMWARE_CIS_HOME%\vmdird\vdcadmintool.exe

    You see these options:

    ================================
    Please select:
    0. exit
    1. Test LDAP connectivity
    2. Force start replication cycle
    3. Reset account password
    4. Set log level and mask
    5. Set vmdir state
    ================================

     
  4. Select option 3.
  5. Enter the user account listed in the vmdir.log file.

    Note: This is the machine account in the format FQDN@SSO DOmain.

    For example:

    [email protected]

    As above, you may have to try option 3 several times before you get a password without invalid special characters.

     
  6. Make note of the generated password.
  7. Connect to the vCenter Server or PSC and open regedit.

    Note: Before making any registry modifications, ensure that you have a current and valid backup of the registry and the virtual machine. For more information on backing up and restoring the registry, see the Microsoft article 136393.
     
  8. Navigate to HLKM\System\CurrentControlset\Services\VMwareDirectoryService\ location.
     
  9. Update the password for the key dcAccountPassword.
  10. Save the changes and exit.
  11. Restart the vCenter Server services. For more information, see Stopping, starting, or restarting VMware vCenter Server 6.x services (2109881).

2) Reset machine account password using dir-cli (versions 6.5 onwards)

Note: Ensure offline snapshots of all vCenters and PSCs are in place before running. This means to power off all vCenters and PSCs in the SSO domain, login to the ESXi hosts they're placed on and snapshot them when down. If reverting, revert all machines before powering any on. This is to ensure consistency in the SSO domain.

Appliance:
  1. Login to the machine noted in the vmdird-syslog.log as root user via SSH
  2. Run this command to enable access the Bash shell:

    shell.set –enabled true
     
  3. Type shell and press Enter.
  4. Run below command where <Platform Services Controller FQDN> is the FQDN of the vCenter with embedded PSC or PSC of the machine with the error 49 in it's vmdird-syslog.log log:
/usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password <[email protected] password>

Windows:
  1. Login to the machine noted in the vmdir.log 
  2. Open the elevated command prompt on the vCenter with Embedded PSC/PSC
  3. Run below command where <Platform Services Controller FQDN> is the FQDN of the vCenter with embedded PSC or PSC of the machine with the error 49 in it's vmdir.log:
 
%VMWARE_CIS_HOME%\vmafdd\dir-cli.exe computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password <[email protected] password>
 

3) Reset machine account password using shell script (Appliance only)

LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script (70756)
 

4) Reset using reset_machine_pw.sh shell script (Built in for version 7.0 only)

  1. Take offline snapshots of all vCenters in the SSO domain before proceeding. This means to power off all vCenters in the SSO domain, connect to the ESXi hosts they're placed on and snapshot each of them while in powered off state. If reverting; restore each to snapshot before powering any on. This ensure consistency of the SSO domain.
  2. Connect to the vCenter over SSH with root user and type shell to access the bash shell
  3. Run the script using the command below - you'll be prompted for the FQDN of the replication partners (vCenters) that you wish to reset the machine account password for and also prompted for SSO admin credentials:
/usr/lib/vmware-vmdir/vmdir-tool/reset_machine_pw.sh


Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
How to stop, start, or restart vCenter Server 6.x services
Stopping, starting, or restarting VMware vCenter Server Appliance 6.x services
VCenter Server fails to start with "Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)", After vCenter Server is restored from backup or snapshot

Read this article in different languages here:
在 vCenter Server 6.x 中启动 Inventory Service 时出现“凭据无效 LDAP 错误 49 (invalid credentials LDAP Error 49)”错误
vCenter Server 6.x で Inventory Service を開始すると「無効な認証情報 LDAP エラー 49(invalid credentials LDAP Error 49)」エラーが発生する