Moving a vSAN cluster from one vCenter Server to another
search cancel

Moving a vSAN cluster from one vCenter Server to another

book

Article ID: 326849

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

This article explains how to move a vSAN cluster from one vCenter Server to another vCenter Server.

Environment

VMware vSAN 8.0.x
VMware vSAN 7.0.x
VMware vSAN 6.x

Resolution

The following steps can be followed when migrating vSAN clusters from one vCenter to another vCenter and when the original vCenter is offline permanently or unable to be restored from backup.

Notes:

  • Always maintain current and tested VM backups of critical Virtual Machine workloads.
  • Review this document in its entirety before proceeding with the migration.
  • For stretched clusters, you must Disable stretched cluster from Fault Domains (Cluster > Configure > vSAN > Fault Domains > Stretched Cluster > Disable) prior to migrating the cluster to the new vCenter. Once the cluster has been fully migrated then you can enable stretched cluster again.
  • If you forgot to disable stretched cluster in Fault Domains configuration or vCenter failed and had to be redeployed see KB 77163


Base Steps:
1. Build a new vCenter server with a build version equal or greater than the version of the ESXi Hosts being migrated.
Version differences may affect the Skyline vSAN Health services in rare cases. 

2. Prior to migrating the cluster, please be sure to test the vSAN VMkernel, Host Management, vMotion, and any additional dependent network routing schemes.

3. Determine if the host networking is configured with vDS, vSS, or NSX-T NVDS virtual switches.

- When using vSphere Standard Virtual Switch no additional steps are required other than ensuring the physical networking routes are working as designed.
- When using vSphere Distributed Switch, please reference further material/instruction KB links below to prepare the vDS for migration.

Note: While importing vDS settings, do not select preserve original distributed switch port group identifiers as this could negatively impact the migration.
- When using NSX-T NVDS with standalone host configuration, no additional steps are required other than ensuring the physical networking routes are working as designed.        
Note: If using Transport Node Profile for your NSX-T NVDS, then you need to migrate to standalone host configuration.

For more information on the migration of Hosts, see Moving an ESX/ESXi host with vDS from one vCenter Server to another (1029498)
For more information on migration of vDS see  Exporting/importing/restoring Distributed Switch configs using vSphere Web Client (2034602)

Backing Up and Restoring a vSphere Distributed Switch Configuration

For more information on vSphere see https://docs.vmware.com/

- Once the cluster networking configuration requirements are satisfied you may proceed with the next steps of the migration.

4. Determine if vSAN Data at Rest Encryption is in use and make certain that keys and the KMS configurations are well known. See the guide for further guidance in preparing the cluster for migration. 
Also, See further details in this instruction on step 7a.
Document: Using Encryption on a vSAN Cluster

5. Create a new cluster with DRS, vSphere HA, and EVC Disabled.  These options should only be enabled after the cluster is moved and stable on the new vCenter.

6. Create a new cluster Inventory object.

7a. Enable vSAN along with required services matching the original cluster parameters such as Encryption and or Deduplication and Compression. 
- If the original vCenter is not available to compare configurations, the following command can be used to list the vSAN services in use.


localcli vsan storage list |grep -i 'Device\|Is ssd\|Is Capacity Tier\|DEDUPLICATION\|COMPRESSION\|In CMMDS\|ENCRYPTION:' |sed 'N;N;N;N;;N;N;s/\n//g' | sort -k9;
       

 

# localcli vsan storage list |grep -i 'Device\|Is ssd\|Is Capacity Tier\|DEDUPLICATION\|COMPRESSION\|In CMMDS\|ENCRYPTION:' |sed 'N;N;N;N;;N;N;s/\n//g' | sort -k9;
                                        Device: mpx.vmhba0:C0:T1:L0   Is SSD: true   In CMMDS: true   Deduplication: false   Compression: false   Is Capacity Tier: false   Encryption: true
                                        Device: mpx.vmhba0:C0:T2:L0   Is SSD: true   In CMMDS: true   Deduplication: false   Compression: false   Is Capacity Tier: true   Encryption: true


7b. When KMS Encryption service is in use -
Make certain to add the KMS servers to the newly built vCenter server with the same KMS-Cluster names and complete all required steps to add the KMS servers into the new vCenter server.

  • If you are not sure about the KMS cluster procedures, please reference KMS documentation and/or contact VMware Global Support to consult.
  • If you are unsure about the KMS cluster name and the original cannot be recovered, you can obtain the KMS details about KMS-cluster info by using "esxcli vsan encryption kms list" and  "esxcli vsan encryption info get" on any of the hosts in the cluster, where vSAN encryption was enabled. Other additional information such as host-key and certs can be fetched using "esxcli vsan encryption hostkey get" and "esxcli vsan encryption cert get".

     
Example :

 

# esxcli vsan encryption kms list
        Cluster               Name   KMS Address        Proxy Address  Proxy Port  Username
        --------------------  -----  -----------------  -------------  ----------  --------
        Hy-Trust-KMS-Cluster  KMS-1  10.109.44.40:5696  N/A            N/A         N/A
     
# esxcli vsan encryption info get
        Attribute            Value
        -------------------  -----
        kekId                bfdfad26-ba41-4bec-8593-d16014858278
        dekGenerationId      1
        enabled              True
        hostKeyId            835cc701-1734-4411-bf2c-7df3ec524536
        eraseDisksBeforeUse  False
        changing             False


8. Create the Virtual Machine vSAN storage policies that match the vSAN policies from the old cluster. Ensure that rulesets have matching policies. If the VMs are migrated without matching policies, vSAN may need to perform a resync to ensure the VMs and objects are compliant.

If the policy names and policy attributes were unknown due to old vCenter unavailability, the following commands can be used to obtain the policy settings:

"esxcli vsan debug object list | grep spbmProfileId | sort | uniq" or "esxcli vsan debug object list | grep spbmProfileName | sort | uniq"
Review the attributes for each object using the policies by running "esxcli vsan debug object list |less" and search by the policy name (found above).

Example :

# esxcli vsan debug object list | grep spbmProfileId | sort | uniq
                                    spbmProfileId: 3128b75d-6790-4f7f-a25d-4d7889850940
                                    spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
  # esxcli vsan debug object list | grep spbmProfileName | sort | uniq
                                    spbmProfileName: VM Storage Policy-DN
                                    spbmProfileName: vSAN Default Storage Policy

An example is provided below of the policy attributes for the spbmProfileName: vSAN Default Storage Policy

            Run Command : esxcli vsan debug object list | less
Example :

 

            Object UUID: b5bf275e-5ba7-5d01-be96-ecf4bbec6050
            Version: 10
            Health: healthy
            Owner: ESXI1.lab.com
            Size: 15.00 GB
            Used: 0.89 GB
            Policy: ** CONTENT BELOW INDICATES POLICY ATTRIBUTES**
                cacheReservation: 0
                forceProvisioning: 0
                spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
                proportionalCapacity: 0
                spbmProfileGenerationNumber: 0
                stripeWidth: 1
                spbmProfileName: vSAN Default Storage Policy
                CSN: 16
                hostFailuresToTolerate: 1

            Configuration:

                RAID_1
                    Component: b5bf275e-6125-aa03-efcc-ecf4bbec6050
                    Component State: ACTIVE,  Address Space(B): 16106127360 (15.00GB),  Disk UUID: 52208d4c-7f55-578a-cf89-21b65b58e3a3,  Disk Name: naa.5002538c4044d87f:2
                    Votes: 1,  Capacity Used(B): 486539264 (0.45GB),  Physical Capacity Used(B): 478150656 (0.45GB),  Host Name: ESXI1.lab.com
                    Component: b5bf275e-8389-ab03-58a0-ecf4bbec6050
                    Component State: ACTIVE,  Address Space(B): 16106127360 (15.00GB),  Disk UUID: 5225900e-2911-3548-1c57-3fb38e948a00,  Disk Name: naa.5002538c4044d881:2
                    Votes: 1,  Capacity Used(B): 486539264 (0.45GB),  Physical Capacity Used(B): 478150656 (0.45GB),  Host Name: ESXI2.lab.com
                Witness: b5bf275e-0168-ac03-7612-ecf4bbec6050
                    Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 525452e3-e6c0-884a-a6ea-edc03570a490,  Disk Name: naa.5002538c4044d6ab:2
                    Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: ESXI3.lab.com

            Type: vdisk
            Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/b1bf275e-24f9-b5ff-5860-ecf4bbec6050/VCSA70U1_6.vmdk (Exists)
            Group UUID: b1bf275e-24f9-b5ff-5860-ecf4bbec6050
            Directory Name: N/A


        
        
9. In vSAN 6.6 and above, it is critical to maintain vSAN virtual networking unicast communication between hosts on the vSAN network.  Please see KB for configuration details.
Configuring vSAN Unicast networking from the command line (2150303)


10. Set the following configuration via CLI on each ESXi vSAN Cluster Member Host to be migrated.
esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates

This will ensure when the hosts are disconnected from the original vCenter, they will not lose their unicast configurations and therefore their connectivity between hosts. If this step is skipped, the unicast address table will be cleared and data availability will be affected temporarily until the table is manually repopulated.


11. Run the command "esxcli vsan cluster get" from any hosts in the cluster and take note of the host count to ensure the count matches before and after the migration.


12. Disconnect one host from vCenter by right clicking the host and using the disconnect option. (**Not Applicable if old vCenter is unavailable)


13. Remove the host from inventory. Make sure to follow KB 1029498 if using a vDS  (**Not Applicable if old vCenter is unavailable)


14. Add the hosts only to the DataCenter level of the new vCenter inventory. Do NOT add the hosts directly to the new vSAN enabled cluster otherwise, the host will be forced into Maintenance Mode. (Avoiding Maintenance Mode is key for a seamless downtime migration.)


15. Once the host is registered in the new vCenter inventory, under the DataCenter Inventory Object, it can safely be dragged and dropped into the new vSAN enabled cluster from within the vCenter UI.


16. Before moving on to the next host, verify there is host-to-host vSAN communication. Run the command "esxcli vsan cluster get" from ESXi CLI and take note of the host count to ensure the count matches.

It is good to practice this command each time a host is disconnected, moved, and connected. If the host is not communicating, i.e. the host count is less than expected then it is highly likely a critical step has been missed. Please review the steps and ensure each host's unicast tables are complete and host-to-host communication on the vSAN network is fully functional.
To verify unicast agent table, you can use below command -

esxcli vsan cluster unicastagent list


17. Once the first host is moved and connectivity checks are performed you may move forward with the remaining hosts. Please be certain to follow the steps carefully and check the cluster membership along the way to prevent any data availably impacts.


18. Once all hosts have been added to the new cluster and access to the VMs has been verified, revert the hosts' unicast table configuration. This procedure should be completed at the completion of the migration, and before the hosts are rebooted next. Host rebooting is not necessary for the procedure's success.
esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates
 

19. Again verify host and VM connectivity again by pinging the new vCenter from the management interface on each host.


20. Apply the vSAN storage policy to the VMs. 
Click OK. Assuming the policy is consistent with the old policy and all the hosts have remained in a healthy state, a resync shouldn't be triggered. It is recommended to perform this action, one at a time. The goal is to move gradually to prevent creating a major resync in the cluster.

       


21. Repeat this action for the remaining VMs. When performed correctly, there will be no reconfigurations/resyncs. If the VMs start resyncing then there is an inconsistency in the ruleset from the original to the new storage policies. Please allow the resync to complete before proceeding. 


22. Verify by browsing to Menu, Policies and Profiles, VM Storage Policies, desired storage policy, and VM Compliance.  All the VMs should be shown.


Additional Information

Impact/Risks:
If performed within the parameters listed below, the migration will be seamless. However, if performed incorrectly, data availability may be temporarily affected until missing or misconfigured parameters are corrected.

Note:
Moving a vSAN File Service enabled cluster from one vCenter to another is not supported, since the data associated with vSAN File Service cannot be migrated.