Symptoms:

• After configuring a vSphere Replication Management Server (VRMS) server in vCenter Site Recovery Manager (SRM) 5.0, the VRMS server status appears as disconnected.
  
  
• When registering the VRMS Server to vCenter Server fails, you see the this SSL certificate warning:
  
  Unacceptable signature algorithm: MD5withRSA
  
  
• In the /opt/vmware/hms/logs/HMS.log file on the VRMS server, you see entries similar to:
  
  2012-01-30 15:03:51.846 ERROR jvsl.sessions [main] (..net.impl.PersistentConnection) | Failed to connect to server vcenter.xxx.xxx:80
java.util.concurrent.ExecutionException: javax.net.ssl.SSLException: Unacceptable signature algorithm: MD5withRSA
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:70)
at com.vmware.jvsl.sessions.net.impl.vc.ConnectionHandlerImpl.doConnect(ConnectionHandlerImpl.java:88)
at com.vmware.jvsl.sessions.net.impl.vc.ConnectionHandlerImpl.doConnect(ConnectionHandlerImpl.java:36)
at com.vmware.jvsl.sessions.net.impl.PersistentConnection.connect(PersistentConnection.java:517)
at com.vmware.jvsl.sessions.net.impl.PersistentConnection.start(PersistentConnection.java:131)
at com.vmware.jvsl.sessions.net.impl.vc.ServerViewImpl.start(ServerViewImpl.java:74)
at com.vmware.jvsl.sessions.net.impl.AbstractServer.start(AbstractServer.java:83)
at com.vmware.jvsl.sessions.net.ServerRegistry.getLocalVcServer(ServerRegistry.java:223)
at com.vmware.hms.monitor.host.HostInventoryListenerImpl.synchronizeInventory(HostInventoryListenerImpl.java:468)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
  
  
• You receive an error similar to:
  
  Login of SRM Server '<SRM_Server_Instance>' into vCenter Server at '<vCenter_FQDN:80>' failed. SRM server '<SRM_Server_Instance>' cannot validate SSL certificate from server at '<vCenter_FQDN:80>'. The remote host certificate has these problems: Unknown SSL certificate error.

This issue can occur if vCenter Server is upgraded from version 2.5 to version 4.x, and finally to version 5.0 using the default certificates that come with vCenter Server (the default vCenter Server 2.5 certificate is signed using an MD5 certificate). vCenter Server 5.0 is configured with a custom certificate using the MD5 algorithm.

VRMS servers do not support MD5 certificates and refuse any connection to remote servers (in this case, vCenter Server) with MD5 certificates.

The MD5 algorithm for the certificate key is a vCenter Server aspect. The VRMS server extension registration fails on the vCenter Server because the VRMS server cannot accept the vCenter Server certificate key which is encrypted using an MD5 signature algorithm.

Site Recovery Manager itself is unrelated to this issue, as it caused by the certificate issue between the VRMS server and vCenter Server. The pairing fails because the local VRMS server shows as not connected to the local vCenter Server.

The VRMS server does not support vCenter Server certificate public key if this key is encrypted using the MD5 algorithm.

The vCenter Server certificate public key must be encrypted using the RSA SHA1 signature algorithm.

To resolve this issue, follow these steps:

1. If vCenter Server was installed using default certificates, and if it was upgraded from version 2.5 to version 4.x, and then to version 5.x, it is using a certificate public key that is encrypted using the MD5 algorithm. In this case, the vCenter Server default certificate and the certificate key must regenerated using the SHA1 algorithm. To accomplish this, see Method-1 in Regenerating expired SSL certificates after 2 years in VMware vCenter Server 4.x / 5.x (1009092).
   
   Note: If the vCenter 4.x or 5.x Server was a new install, the certificates use the SHA1 signature algorithm by default for the public key. In this case, the issue does not occur.
   
   
2. If vCenter Server 5.x (regardless of whether it was upgraded from a previous version or a new install) is using custom CA signed certificates, you must verify the signature algorithm of the certificate public key:
   
   
   1. Locate the vCenter Server certificate. (Because this is a custom certificate, the location will vary depending on where you placed the rui.crt file on your vCenter Server.)
   2. Right-click the .crt file and click Open.
   3. Click the Details tab.
   4. Check the value of the Signature Hash Algorithm property:
      
      
      • If the value is SHA1, you do not have to regenerate the certificate key.
        
      • If the signature hash algorithm is MD5, the vCenter Server custom CA signed certificate and the certificate key must regenerated using the RSA SHA1 algorithm:
        
        
        1. Recreate a certificate signing request for your vCenter Server and submit that to your designated root certification authority.
        2. Once you obtain the new signed certificate, you must reload it into your vCenter Server. For more information, see Replacing vCenter Server 4.1 and 5.0 SSL certificates using the vpxd -p command fails with the error: failed to do early initialization (1030661).
           
           Note: Some CA authorities require that you use 2048-bit certificates.

• Register the new vCenter Server certificate with the SRM server:
  
  
  1. Start the SRM server installation again and select the Modify option. This allows SRM to accept the vCenter Server's new certificates.
     
  2. Un-register and re-register the VRMS server with the vCenter Server.
     
     Note: You may have to recreate the VRMS server database instance to achieve this. This would be the last resort. The creation of a new VRMS DB is a disruptive step which results loosing replication information and will require to re-replicate all protected virtual machine disks.
     
     
  3. When you are prompted to accept the new vCenter Server certificate, this indicates that the VRMS server will successfully register itself with vCenter Server.
     
  4. Verify that the new certificate is being used and accept it to complete the registration process.

For more information, see:

• Site Recovery Manager site pairing fails with custom certificates (1016175)
  
• Requirements when using trusted certificates with VMware Site Recovery Manager 1.0.x/4.0.x/4.1.x/5.x (1008390)
  

Requirements when using trusted certificates with VMware Site Recovery Manager 1.0.x to 5.0.x
Regenerating expired SSL certificates after 2 years in VMware vCenter Server 4.x / 5.0.x
VMware vCenter Site Recovery Manager 4.x /5.x site pairing fails with custom certificates
Replacing vCenter Server 4.1 and 5.0 SSL certificates using the vpxd -p command fails with the error: failed to do early initialization