"Leave fed not complete, tries left 0", Convergence of vCenter Server in Enhanced Linked Mode fails due to Replication Partner VMDIR in Standalone Mode
Article ID: 322233
Updated On:
Products
VMware vCenter Server
VMware vSphere ESXi
Issue/Introduction
Symptoms:
- You are running vCenter Server in Enhanced Linked Mode and Convergence of very first vCenter Server Completed successfully.
- Convergence of subsequent vCenter Server in ELM pointing to the first Converged vCenter Server hangs at 8% and eventually fails.
- Converge log (/var/log/vmware/converge/converge.log) of failed vCenter Server shows similar to below entries,
2019-07-09T14:14:07.135Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:14:07.146Z INFO converge Leave fed not complete, tries left 29
2019-07-09T14:14:37.177Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:14:37.190Z INFO converge Leave fed not complete, tries left 28
2019-07-09T14:27:38.177Z INFO converge Leave fed not complete, tries left 2
2019-07-09T14:28:08.180Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:28:08.192Z INFO converge Leave fed not complete, tries left 1
2019-07-09T14:28:38.192Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:28:38.204Z INFO converge Leave fed not complete, tries left 0
2019-07-09T14:29:08.235Z INFO converge Leave fed complete status: False
2019-07-09T14:29:08.241Z INFO converge Unsetting the converge running flag
2019-07-09T14:14:07.146Z INFO converge Leave fed not complete, tries left 29
2019-07-09T14:14:37.177Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:14:37.190Z INFO converge Leave fed not complete, tries left 28
2019-07-09T14:27:38.177Z INFO converge Leave fed not complete, tries left 2
2019-07-09T14:28:08.180Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:28:08.192Z INFO converge Leave fed not complete, tries left 1
2019-07-09T14:28:38.192Z INFO converge Collected replication_flag:True as a part of Telemetry
2019-07-09T14:28:38.204Z INFO converge Leave fed not complete, tries left 0
2019-07-09T14:29:08.235Z INFO converge Leave fed complete status: False
2019-07-09T14:29:08.241Z INFO converge Unsetting the converge running flag
- VMware Directory Service Log (/var/log/vmware/vmdird/vmdird-syslog.log) on the Converged vCenter (Replication Partner) shows similar to below entries,
2019-07-09T11:29:01.120342+01:00 info vmdird t@139733669390080: VmKdcInit: done!
2019-07-09T11:29:01.120517+01:00 info vmdird t@139733669390080: Lotus Vmkdcd: running...
2019-07-09T11:29:01.120739+01:00 info vmdird t@139733669390080: VmDir State (8)
2019-07-09T11:29:01.120926+01:00 info vmdird t@139733669390080: Lotus Vmdird: running... state (8)
2019-07-09T11:29:01.122121+01:00 info vmdird t@139711444145920: Vmkdc: initializing directory
2019-07-09T11:29:01.122366+01:00 info vmdird t@139711444145920: Vmkdc: VmKdcdStateSet(1)
2019-07-09T11:29:01.122566+01:00 info vmdird Starting VMware Directory Servicedone
2019-07-09T11:29:18.712393+01:00 err vmdird t@139710823380736: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
2019-07-09T11:29:18.712755+01:00 err vmdird t@139710823380736: VmDirSendLdapResult: Request (Search), Error (53), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (<IP Address>)
2019-07-09T11:29:48.776604+01:00 err vmdird t@139710143899392: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
2019-07-09T11:29:48.777148+01:00 err vmdird t@139710143899392: VmDirSendLdapResult: Request (Search), Error (53), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (<IP Address>)
2019-07-09T11:29:01.120517+01:00 info vmdird t@139733669390080: Lotus Vmkdcd: running...
2019-07-09T11:29:01.120739+01:00 info vmdird t@139733669390080: VmDir State (8)
2019-07-09T11:29:01.120926+01:00 info vmdird t@139733669390080: Lotus Vmdird: running... state (8)
2019-07-09T11:29:01.122121+01:00 info vmdird t@139711444145920: Vmkdc: initializing directory
2019-07-09T11:29:01.122366+01:00 info vmdird t@139711444145920: Vmkdc: VmKdcdStateSet(1)
2019-07-09T11:29:01.122566+01:00 info vmdird Starting VMware Directory Servicedone
2019-07-09T11:29:18.712393+01:00 err vmdird t@139710823380736: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
2019-07-09T11:29:18.712755+01:00 err vmdird t@139710823380736: VmDirSendLdapResult: Request (Search), Error (53), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (<IP Address>)
2019-07-09T11:29:48.776604+01:00 err vmdird t@139710143899392: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
2019-07-09T11:29:48.777148+01:00 err vmdird t@139710143899392: VmDirSendLdapResult: Request (Search), Error (53), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (<IP Address>)
Environment
VMware vSphere ESXi 6.7
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.7.x
Cause
During Converge process, VMware Directory Service DB will be set in Standalone mode and exits from Standalone mode at the end of process. However, likewise Service Manager where the Standalone arguments for VMDIRD service are stored is not refreshed post Convergence. Whenever, VMDIRD service is restarted post Convergence it will restart in Standalone Mode and due to that Convergence of second or subsequent vCenter Server will fail as the replication will not happen to the first node which was already converged.
Resolution
This issue is resolved in VMware vCenter Server 6.7 U3b, available at VMware Downloads.
Workaround:
Before Starting the Convergence process of vCenter Server in ELM, verify the VMDIRD Status of the Replication Partner. Status of VMDIRD should be NORMAL. Follow below steps to verify and set the status:
Workaround:
Before Starting the Convergence process of vCenter Server in ELM, verify the VMDIRD Status of the Replication Partner. Status of VMDIRD should be NORMAL. Follow below steps to verify and set the status:
- Login to the Replication Partner VC using SSH Client (Embedded vCenter Server).
- Change Shell to Bash running the command below,
shell
- Verify the current status of VMDIR (it will prompt for SSO Admin Credentials) running the command below,
/usr/lib/vmware-vmafd/bin/dir-cli state get
Example:
/usr/lib/vmware-vmafd/bin/dir-cli state get
Enter password for [email protected]:
Directory Server State: Standalone (8)
Example:
/usr/lib/vmware-vmafd/bin/dir-cli state get
Enter password for [email protected]:
Directory Server State: Standalone (8)
- If the State is Standalone, execute below command to Set VMDIR in NORMAL State (If it is already in NORMAL state, skip this step and proceed with the final step to Refresh Likewise Service Manager),
/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL
Enter password for [email protected]:
Directory Server State set to: NORMAL (3)
Enter password for [email protected]:
Directory Server State set to: NORMAL (3)
- Run below command to verify the status to confirm the change,
/usr/lib/vmware-vmafd/bin/dir-cli state get
Enter password for [email protected]:
Directory Server State: Normal (3)
Enter password for [email protected]:
Directory Server State: Normal (3)
- Refresh Likewise Service Manager by executing below command,
/opt/likewise/bin/lwsm refresh
- Proceed with Convergence of vCenter Server.
Feedback
Yes
No
Powered by
