SRM or VR is unable to connect to vCenter after PSC is converged to Embedded PSC or vCenter SSL changed
search cancel

SRM or VR is unable to connect to vCenter after PSC is converged to Embedded PSC or vCenter SSL changed

book

Article ID: 339651

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Background. 

 

Announced in November 2018, the External Platform Services Controller (PSC) is being deprecated. Previously, many customers have deployed the External PSC to take advantage of features such as Enhanced Linked Mode (ELM) or PSC HA. With the removal of the External PSC, VMware has provided a Converge Tool to be able to migrate from an External Deployment to an Embedded deployment. The Converge Tool does require you to be on vSphere 6.5 Update 2d or greater or vSphere 6.7 Update 1 or greater and be running the vCenter Server Appliance (VCSA).

https://blogs.vmware.com/vsphere/2019/02/understanding-the-vcenter-server-converge-tool.html

 

image1.jpg

 

Before proceeding with the convergence ensure to have a screenshot of the SRM/VR configuration and an export of the configuration with the export import tool.



Symptoms:
  • PSC Convergence has been done and attempts have been done to run Modify/Install on SRM Windows OR Reconfigure on SRM Appliance

  • Error 'null' appears on the SRM UI or it just fails to load

  • -Site Recovery Manager and vSphere Replication services may crash eventually after start up.
              -You may get an invalid thumbprint error when attempting to repair site connections or may receive an SDK timeout.
              -Reconfigure or Modify/Install fails.



Environment

VMware vSphere Replication 8.x

Cause

  • This is due to stale entries of external PSC of source and target site still present in the database and needs to be updated.
  • SRM unable to pair sites due to VMdir mismatch.
  • PSCs rolled back to different points in time.
  • Network connection loss between sites.
  • PSC Replication status failure.

Resolution

Resolution. 

Prerequisites.

  • Check replication status of vCenter has no changes behind and has an updated status.  Ref: https://kb.vmware.com/s/article/2127057

    psc1:~ # /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator.  If replication status is not matching on both vCenters in linked mode please log a new ticket under vCenter Product Support.   Continue to keep SRM ticket open until SRM is fully operational. 
  • Snapshot and backup all nodes SRM /VR /VC and backup the database of SRM. 
  • Run lsdoctor -l to check SSL certs on PSC and lsdoctor -t to run fix. https://kb.vmware.com/s/article/80469
  • Ensure vCenter certificate on port 5480 matches the vCenter certificate on port 443.  If not follow this kb. 
  • https://kb.vmware.com/s/article/2136693
  1. Modify install / Save and Restart (VR) / Reconfigure 8.4 Appliances
  2. Reconnect site pairings. 
  3. Logging a Support Request. 
 

Modify Install SRM. 

If SRM is windows, copy the version of the SRM installer to a writable directory and run the installer as administrator.  Choose the Modify Option Only(In the event this fails choose the uninstall option keeping existing data and run installer again).   Entering the new FQDN of the embedded vCenter during the PSC step. 

 

SRM is an appliance

Login to https://<SRM_IP:5480 and choose the reconfigure option and follow the prompts. 

If still failing, check thumbprints in the SRMDB are correct.  

####Insert the correct thumbprint SRM tables.

####>pd_sslthumbprintstore

####>pds_remotesite

####>pds_solutionuser

Simple script to check. 

#/bin/bash!

 

####Insert the correct thumbprint SRM tables.

####>pd_sslthumbprintstore

####>pds_remotesite

####>pds_solutionuser



 

echo please enter PSC?

 

read psc

 

echo $psc

 

pscthumbprint=`openssl s_client -connect $psc:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -sha256 -noout -in /dev/stdin`

 

echo $pscthumbprint | awk -F "=" '{print $2}'



 


Main issues affecting SRM/VR after PSC converge process

1. New PSC URL or thumbprints have not been updated in the SRM Db tables
pd_localsite; pds_remotesite; pds_solutionuser; pd_sslthumbprintstore
  
    New PSC URL or thumbprints have not been updated in the VR Db tables
localvcentity; hmsserverentity; vmomiserverentity

Issue: Old PSC name and thumbprints not correctly updated
Solution: Edit the SRM & VR DB tables to correct



SRM Db Tables:
# SELECT * from pd_localsite;
# SELECT * pds_remotesite;

# SELECT * from pds_solutionuser;
# SELECT *  from pd_sslthumbprintstore;

VR Db Tables:
VR8.2/8.3/8.4 [localvcentity]
# cat  /opt/vmware/hms/conf/embedded_db.cfg | grep -i EMB_DB_PASSWORD; echo -e "\n\n" && echo "select dbid,address,sitename,thumbprint  from localvcentity;" | /opt/vmware/vpostgres/current/bin/psql -U vrmsdb

VR8.2 or lower [hmsserverentity AND vmomiserverentity]
#echo "select dbid, address, sitename, thumbprint FROM vmomiserverentity;" | /opt/vmware/vpostgres/current/bin/psql -U vrmsdb
# echo "select pairedvcsitename, pairedvcuuid FROM hmsserverentity;" |  /opt/vmware/vpostgres/current/bin/psql -U  vrmsdb       

VR8.3/8.4 [hmsserverentity NOT vmomiserverentity]
# cat  /opt/vmware/hms/conf/embedded_db.cfg | grep -i EMB_DB_PASSWORD; echo -e "\n\n" && echo "select pairedvcsitename, pairedvcuuid, vsrv_thumbprint   from hmsserverentity;" | /opt/vmware/vpostgres/current/bin/psql -U vrmsdb

* Restart services
* Reconfigure



 

VR 8.4 or later

 

Login to https://<SRM_IP:5480 and choose the reconfigure option and follow the prompts and enter the new FQDN of embedded vCenter. 

 

In case of failure follow the same steps as outlined for VR 8.3 if that fails then please log a support ticket as mentioned later in the KB. 


SRM 8.5 Fix Post PSC convergence. 
 

The first issue is the pd_localsite will need to be updated. Usually this will hold the old PSC address.  We want to change this to the vCenter address.

echo "select * from pd_localsite;" | PGPASSWORD="$pass" /opt/vmware/vpostgres/current/bin/psql -U srmdb -d srmdb -p 5432


To update.

echo "update pd_localsite set siteurl = 'https://myVC.vmware.com:443/lookupservice/sdk' where db_id =1 ;" | PGPASSWORD="$pass" /opt/vmware/vpostgres/current/bin/psql -U srmdb -d srmdb -p 5432 

Important next step is to reboot SRM.

Then try and reregister.   If it fails likely 1 or 2 services are not in the correct site ID location.


Step 2: 

The next steps are to download both of these ldap tools jxplorer and LDAPADMIN.



https://sourceforge.net/projects/jxplorer/
 
https://sourceforge.net/projects/ldapadmin/
 

username =  cn=administrator,cn=users,dc=vsphere,dc=local


Step 3: 


You may see some entries here from old Site ID name and another Default Site ID name after convergence.  In Old Site ID you may see both SRM/VR registrations along with
authentication VC registration.  Use lsdoctor to rebuild these services with the -r option. 


Step 4: 

For Error code 61 when registering SRM remove the com.vmware.vcDr plugin from the vCenter mob. 

Go to https://vCenterAddress/mob > Login with [email protected]
>Extension Manager> Go to unregister extenstion> com.vmware.vcDr and click unregister twice.

And do a reconfigure/modify install which should be successful.  



Step 5: 

Reconnect site pairings.  


Additional Information

Impact/Risks:
May require a complete reinstall of SRM / VR environment or may lead to long downtime resolving the registration issues.