• Upgrading vCenter Server to 7.0 fails with WCP firstboot with error "Failed to create service account for workload storage" "VMware directory error[9127]"
• You will see the log entries similar to the below snippet in /var/log/firstboot

wcp-firstboot.py_xxxxxx_stderr.log
2021-02-02T13:21.231Z WCP firstboot failed

wcp-firstboot.py_xxxx_stdout.log
2020-12-02T13:21:35.279Z INFO wcp-firstboot WCP storage user does not exists, create the user.
2020-12-02T13:21:35.279Z INFO wcp-firstboot Creating ServiceAccount client...
2020-12-02T13:21:35.368Z Further filtering retrieved service registration list on hostname : vcenter.domain.local
2020-12-02T13:21:35.376Z INFO wcp-firstboot Creating service account...
2020-12-02T13:21:35.377Z INFO wcp-firstboot Initializing ServiceAccount session...
2020-12-02T13:21:43.727Z ERROR wcp-firstboot Unexpected error creating ServiceAccount {messages : [LocalizableMessage(id='com.vmware.vcenter.svcaccountmgmt.error', default_message='Exception found (Internal Server Error, VMware directory error[9127])', args=['Internal Server Error, VMware directory error[9127]'], params=None, localized=None)], data : None, error_type : None}
2020-12-02T13:21:43.727Z ERROR wcp-firstboot Failed to create service account for workload storage
Traceback (most recent call last):
  File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 298, in _create_storage_user
    password = svcacctmgmt_client.create_svc_account(self._user_name)
  File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 90, in create_svc_account
    raise er
  File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 84, in create_svc_account
    svcacct_pwd_out = svcacct_client.create(create_spec)
  File "/usr/lib/vmware-wcp/py-modules/vapi-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 368, in create
    'create_spec': create_spec,
  File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke
    self._rest_converter_mode)
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.vcenter.svcaccountmgmt.error', default_message='Exception found (Internal Server Error, VMware directory error[9127])', args=['Internal Server Error, VMware directory error[9127]'], params=None, localized=None)], data : None, error_type : None}

This issue is caused due to the mismatch between DC account and the SAM account in the vCenter Server.

VMware Engineering Team is aware of this issue and working towards the fix.

1. Check the SAM Account for the VC using below command,

2. Check the pnid as well using below command
   • /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
   • vcenter.domain.local
3. In the above scenario the DC account is vcenter2.domain.local and SAM account is vcenter.domain.local
4. Initiate the VC Upgrade process, post stage 1 take a snapshot of the newly deployed VC.
5. Enable putty/ssh session on the new vCenter server.
6. Open putty session on new VC and add an entry in the /etc/hosts
   • Add  <Source VC IP Address> <CN Name>
   • Example:192.168.10.13 vcenter2.domain.local
7. Continue the upgrade from the Stage 2 or re-initiate the upgrade from VAMI page and use the temporary IP address. You should be able to complete the VC upgrade successfully.