VMdir enters failure state after upgrading vCenter Server to 8.0 U1.

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

The vCenter Server started at version 6.5 or below, and has now been upgraded to 8.0U1.
Messages in /var/log/vmware/vmdird/vmdird-syslog.log show vmdir changing to an unrecoverable state following a reboot or service restart.

2023-05-01T16:13:49.154844-05:00 err vmdird t@140008367298304: _VmDirConsumePartner: Did not succesfully perform any updates after full pull. Moving vmdir to an unrecoverable state
2023-05-01T16:13:49.155184-05:00 info vmdird t@140008367298304: VmDir State (5)
2023-05-01T16:13:49.155230-05:00 err vmdird t@140008367298304: vdirReplicationThrFun: Replication has failed with unrecoverable error.
2023-05-01T16:13:49.157537-05:00 err vmdird t@140008241473280: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
2023-05-01T16:13:49.157585-05:00 err vmdird t@140008241473280: VmDirSendLdapResult: Request (Search), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (10.10.10.10)

There are also messages that indicate a replication conflict for the LegacyAliasMappings cn.

2023-05-01T16:13:48.990984-05:00 err vmdird t@140008367298304: InternalDeleteEntry: VdirExecutePostDeleteCommitPlugins - code(9117)
2023-05-01T16:13:48.991013-05:00 warning vmdird t@140008367298304: ReplDeleteEntry/VmDirInternalDeleteEntry: 66 (Operation not allowed on non-leaf). DN: cn=LegacyAliasMappings,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,DC=vsphere,DC=local, first attribute: cn, it's meta data: '659195:2:abdefg-3891-435f-7afc-6b9636240bb3:20230429035650.714:426961'. NOT resolving this possible replication CONFLICT. For this object, system may not converge. Partner USN 0

Note: There is a small chance that the same replication conflict may occur for entries that are not LegacyAliasMapping. This will cause vmdir to go into the same failure mode. The action plan will be the same in these cases.

The domain functional level (DFL) of the vCenter is not "4".

To retrieve the DFL of vCenter, use the following command.

/usr/lib/vmware-vmafd/bin/dir-cli domain-functional-level get

Environment

VMware vCenter Server 8.0.1

Cause

This occurs when the domain functional level of the vCenter has an unexpected value other than 4. vCenters that have been upgraded since version 6.5 will have a DFL of 1. vCenter servers of version 7.0+ should have a DFL value of 4.

Resolution

This issue is resolved in vCenter Server 8.0 Update 2.

Workaround:
To work around this issue:

Set the DFL of the affected node to 4 with the following command.

/usr/lib/vmware-vmafd/bin/dir-cli domain-functional-level set --level 4 --login [email protected] --domain-name vsphere.local

Note: Update vsphere.local to match your SSO domain name.

Restart the vmdir service on all linked vCenter nodes.

service-control --restart vmdird

Note: Restart vmdir on all nodes only after updating the DFL of all the nodes in the ELM topology. Otherwise, vmdir will fail to start on the nodes which have a higher DFL than their partners.