• After changing the SSL certificate on ESX 3.5 hosts and VirtualCenter 2.5, you cannot connect to the hosts using VirtualCenter
• You see the error:
  
  failed to install virtual center agent services, login failed due to bad user name and password
  
• You can access the hosts directly using VMware Infrastructure (VI) Client
• /var/log/vmware/vpx/vpxa.log has errors similar to:
  
  [2010-03-23 15:31:38.272 0xf7bda6c0 error 'App'] [VpxdCertificate] Failed: unrecognized file format: /etc/vmware/ssl/rui.crt
  

1. Login as root to the ESX host using an SSH client.
2. In the /var/log/vmware/vpx/vpxa.log and look for errors similar to:
   
   unrecognized file format: /etc/vmware/ssl/rui.crt
   
   These errors indicate that VirtualCenter does not recognize the rui.crt file.

To resolve this issue, you must create a new rui.crt file.

1. Open the /etc/vmware/ssl/rui.crt using a text editor and copy its contents from:
   
   -----BEGIN CERTIFICATE-----
   
   TO
   
   -----END CERTIFICATE-----
   
   Note: Do not copy any text that comes before -----BEGIN CERTIFICATE-----.
   
   
2. Create a new file named rui1.crt and paste content you copied in step 1 into the file.
3. Run the following commands:
   
   # mv /etc/vmware/ssl/rui.crt /etc/vmware/ssl/rui.crt.old
   # mv /etc/vmware/ssl/rui1.crt /etc/vmware/ssl/rui.crt
   # service vmware-vpxa start
   
4. Start the VirtualCenter Agent Service. For more info see Restarting the Management agents on an ESX Server (1003490).
5. Add the ESX host to Inventory.

• This issue also occurs when the rui.crt file has windows line terminators (CRLF) in it. This can be checked by running the command:
  
  cat -v /etc/vmware/ssl/rui.crt
  
• Windows line terminators (CRLF) which will show up as ^M characters can be removed running the command:
  
  tr -d '\r' < /etc/vmware/ssl/rui.crt > /etc/vmware/ssl/rui-fixed.crt && mv /etc/vmware/ssl/rui-fixed.crt /etc/vmware/ssl/rui.crt

To view the characters that prevented VirtualCenter from recognizing the old rui.crt file, run the diff command and compare the new file with the old file:

The output appears similar to: