Replacing vCenter Server when vSAN Encryption is configured
Article ID: 326797
Updated On:
Products
VMware vSAN
Issue/Introduction
Symptoms:
After migrating hosts to a new vCenter, the hosts work normally. However, on reboot, the hosts will show drives as unmounted.
Running the command “vdq -q” from one of the hosts, shows drives as not mounted and are ineligible for use by vSAN. In the UI the devices show as encrypted and locked because the encryption key is not available.
After migrating hosts to a new vCenter, the hosts work normally. However, on reboot, the hosts will show drives as unmounted.
Running the command “vdq -q” from one of the hosts, shows drives as not mounted and are ineligible for use by vSAN. In the UI the devices show as encrypted and locked because the encryption key is not available.
Environment
VMware vSAN 6.7.x
VMware vSAN 6.6.x
VMware vSAN 6.6.x
Cause
The issue occurs because the KMS cluster was not added to the new vCenter or it was added under a different KMS Cluster ID.
Resolution
In order to recover from this issue, it is necessary to create a new KMS cluster with the same exact KMS cluster ID as the previous vCenter.
It is imperative that the same KMS cluster ID remains in order for the recovery feature to work. Although the old vCenter is gone, the hosts still have the information and keys from the KMS cluster, after connecting to the same KMS cluster with the same cluster ID, the hosts will be able to retrieve the key (assuming the key still exists and was not deleted). The KMS credentials will be re-applied to all hosts so that hosts can connect to KMS to get the keys.
Since the decommissioned vCenter is no longer available, the KMS cluster ID cannot be retrieved from the vCenter KMS config screen.
In order to obtain the kmip cluster ID, it is necessary to look for such info in the esx.conf file of the hosts.
Open the esx.conf using cat or grep to look at the kmipServer information. Look for kmipClusterId, name(alias), etc. Ensure the KMS cluster on the new vCenter is configured exactly as it was before.
Notice that the KmipClusterId is “HTKC_KMS_Cluster”. When creating the new KMS cluster in the new vCenter use the same KmipClusterId with the required information. KMS information including port varies by KMS vendor. In this case, HyTrust uses port 5696.
After the KMS cluster has been added to the new vCenter as it was configured in the old vCenter, there is no need for reboots. During reconfiguration, the new credentials will be sent to all hosts and such hosts should reload keys for all disks in a few minutes.
Note: As of ESXi 7.0 we switched from esx.conf to configstorecli. To get the KMS info from a host running 7.0 or higher using a Native Key Provider run the below command:
[root@esxi01:~] configstorecli config current get -c esx -g trusted_infrastructure -k "kms_providers"
For 3rd party KMS run configstorecli config current get -c vsan -g system -k "vsan"|less
It is imperative that the same KMS cluster ID remains in order for the recovery feature to work. Although the old vCenter is gone, the hosts still have the information and keys from the KMS cluster, after connecting to the same KMS cluster with the same cluster ID, the hosts will be able to retrieve the key (assuming the key still exists and was not deleted). The KMS credentials will be re-applied to all hosts so that hosts can connect to KMS to get the keys.
Since the decommissioned vCenter is no longer available, the KMS cluster ID cannot be retrieved from the vCenter KMS config screen.
In order to obtain the kmip cluster ID, it is necessary to look for such info in the esx.conf file of the hosts.
Open the esx.conf using cat or grep to look at the kmipServer information. Look for kmipClusterId, name(alias), etc. Ensure the KMS cluster on the new vCenter is configured exactly as it was before.
cat /etc/vmware/esx.conf
or
grep “/vsan/kmipServer/” /etc/vmware/esx.conf
/vsan/kmipServer/child[0000]/old = "false"
/vsan/kmipServer/child[0000]/port = "5696"
/vsan/kmipServer/child[0000]/address = "x.x.x.x"
/vsan/kmipServer/child[0000]/name = "HTKC_1.vsanpe.vmware.com"
/vsan/kmipServer/child[0000]/kmipClusterId = "HTKC_KMS_Cluster"
/vsan/kmipServer/child[0000]/kmskey = "HTKC_KMS_Cluster/HTKC_1.vsanpe.vmware.com"
/vsan/kmipServer/child[0003]/kmskey = "HTKC_KMS_Cluster/HTKC_2.vsanpe.vmware.com"
/vsan/kmipServer/child[0003]/kmipClusterId = "HTKC_KMS_Cluster"
/vsan/kmipServer/child[0003]/name = "HTKC_2.vsanpe.vmware.com"
/vsan/kmipServer/child[0003]/address = "x.x.x.x"
/vsan/kmipServer/child[0003]/port = "5696"
/vsan/kmipServer/child[0003]/old = "false"
or
grep “/vsan/kmipServer/” /etc/vmware/esx.conf
/vsan/kmipServer/child[0000]/old = "false"
/vsan/kmipServer/child[0000]/port = "5696"
/vsan/kmipServer/child[0000]/address = "x.x.x.x"
/vsan/kmipServer/child[0000]/name = "HTKC_1.vsanpe.vmware.com"
/vsan/kmipServer/child[0000]/kmipClusterId = "HTKC_KMS_Cluster"
/vsan/kmipServer/child[0000]/kmskey = "HTKC_KMS_Cluster/HTKC_1.vsanpe.vmware.com"
/vsan/kmipServer/child[0003]/kmskey = "HTKC_KMS_Cluster/HTKC_2.vsanpe.vmware.com"
/vsan/kmipServer/child[0003]/kmipClusterId = "HTKC_KMS_Cluster"
/vsan/kmipServer/child[0003]/name = "HTKC_2.vsanpe.vmware.com"
/vsan/kmipServer/child[0003]/address = "x.x.x.x"
/vsan/kmipServer/child[0003]/port = "5696"
/vsan/kmipServer/child[0003]/old = "false"
Notice that the KmipClusterId is “HTKC_KMS_Cluster”. When creating the new KMS cluster in the new vCenter use the same KmipClusterId with the required information. KMS information including port varies by KMS vendor. In this case, HyTrust uses port 5696.
After the KMS cluster has been added to the new vCenter as it was configured in the old vCenter, there is no need for reboots. During reconfiguration, the new credentials will be sent to all hosts and such hosts should reload keys for all disks in a few minutes.
Note: As of ESXi 7.0 we switched from esx.conf to configstorecli. To get the KMS info from a host running 7.0 or higher using a Native Key Provider run the below command:
[root@esxi01:~] configstorecli config current get -c esx -g trusted_infrastructure -k "kms_providers"
For 3rd party KMS run configstorecli config current get -c vsan -g system -k "vsan"|less
Additional Information
Impact/Risks:
The drives will not become available for vSAN to use until the hosts have reached crypto safe mode.
The drives will not become available for vSAN to use until the hosts have reached crypto safe mode.
Feedback
Yes
No
Powered by
