• vCenter Server 6.5 Windows services under VMware Service life cycle Manager (vmon) fails to start.
• Accessing WebClient or UI Client will throw below error message:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb444041040] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

• Running the Service-control --start --all command times out with an error similar to:
  
  #services-control --start --all
  .....
  .........
  2017-02-28T14:52:20.394Z Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.
  2017-02-28T14:52:20.394Z State for service vmware-psc-client: STOPPED
  Successfully started service vmware-psc-client
  Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start eam, vapi-endpoint, vsphere-ui, vsphere-client, vpxd-svcs, vmware-vpostgres services. Error: Operation timed out
   
• In the %ALLUSERSPROFILE%\VMware/vCenterServer/logs/vmon/vMon.log file, you see this entry indicating the service start failure:
  
  Logon failure: the user has not been granted the requested logon type at this computer.
   
• In the Security Event Viewer, you see an entry for audit failure for the respective users and below properties:
  
  Logon failure: the user has not been granted the requested logon type at this computer
  Log on type : 4
   
• In the svcs.log file, you see entries similar to,
  
  .cis.core.kv.impl.Exception.KVInternalException: Error creating VCDBProviderFactory: at com.vmware.cis.core.kv.impl.VCDBKVManagerImpl.init(VCDBKVManagerImpl.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1706) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1645) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ... 84 more Caused by: com.vmware.cis.core.kv.impl.Exception.KVExecutionException: SQL Error: at com.vmware.cis.core.kv.impl.KVUtil.createAndLogExecutionException(KVUtil.java:153) at com.vmware.cis.core.kv.impl.Provider.VCDBProviderFactory.getConnection(VCDBProviderFactory.java:122) at com.vmware.cis.core.kv.impl.Provider.VCDBProviderFactory.<init>(VCDBProviderFactory.java:83) at com.vmware.cis.core.kv.impl.VCDBKVManagerImpl.init(VCDBKVManagerImpl.java:72) ... 91 more Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) at com.vmware.cis.core.kv.impl.Provider.VCDBProviderFactory.getConnection(VCDBProviderFactory.java:120) ... 93 more Caused by: org.postgresql.util.PSQLException: Connection refused. Check that
  
   

This issue occurs when vCenter Server 6.5 uses multiple service accounts to initialize services under VMware Service life cycle Manager (vmon). These accounts are added to local security policy Log on as a batch job policy. Any group policy overriding this settings can cause these symptoms.

To resolve this issue:

1. Identify the Group policy which is overriding the local policy using the rsop.msc tool.
    
2. Add these accounts to the Log on as a batch job policy of effective GPO. To add the accounts, navigate to Computer Configuration > Windows Settings >Security Settings >Local Policies > User Rights Assignment > Log on as a batch job
   Note: Add the account without the browse option.
   • cm
   • content-library
   • eam
   • imagebuilder
   • mbcs
   • netdumper
   • perfcharts
   • rbd
   • vapiEndpoint
   • vmware-vpostgres
   • vsan-health
   • vsm
   • vsphere-client
   • vsphere-ui
     
     Note: You wont be able to make these changes on the vCenter itself, it will have to be done at their top level AD domain.
      
3. Push the new policies to the vCenter Server.
    
   1. Open Command Prompt on vCenter Server.
       
   2. Run the command gpupdate/force to push the new policies. You see output similar to:
      
      #gpupdate/force
      Updating Policy...
      
      User Policy update has completed successfully.
      Compute Policy update has completed successfully.
       
   3. Reboot the vCenter Server.
       
   4. Verify that new active policies are pushed down and you are able to see them in the batch job policy.
       
4. Start the vCenter services:

cd C:\Program Files\VMware\vCenter Server\bin

service-control --start --all