How to Non-Disruptively Change the VLAN/IP for vSAN VMk in a Production Environment
Article ID: 326852
Updated On:
Products
VMware vSAN
Issue/Introduction
The steps outlined in this KB Article provide a step-by-step tutorial on how to "move" the data traffic in a production vSAN cluster to a new VLAN without risking production interruption or requiring an outage to complete.
Symptoms:
The expectation is to have vSAN traffic flowing on a new VLAN without disruption to production VMs.
Symptoms:
The expectation is to have vSAN traffic flowing on a new VLAN without disruption to production VMs.
Environment
VMware vSAN 7.0.x
VMware vSAN 8.0.x
VMware vSAN 6.x
VMware vSAN 8.0.x
VMware vSAN 6.x
Cause
Occasionally, customers find it necessary to change the IP scheme in use for parts or all of their environments.
A "vmk" HA functionality feature was introduced in vSAN which allows 2 VMK interfaces to to have vSAN enabled on them at the same time.
This KB takes you through the process step-by-step and assumes the original (old VLAN) VMK interface will be deleted.
A "vmk" HA functionality feature was introduced in vSAN which allows 2 VMK interfaces to to have vSAN enabled on them at the same time.
This KB takes you through the process step-by-step and assumes the original (old VLAN) VMK interface will be deleted.
Resolution
The steps outlined in this KB Article assume the following,
1. The vSAN Cluster is currently serving production data
2. The expectation is to have vSAN traffic flowing on a new VLAN at the conclusion of this KB without disruption to production VMs.
3. The vSAN hosts are already connected to 10G switch ports configured with the old/existing VLAN
4. The vSAN hosts are already connected to 10G switch ports configured with the new VLAN.
5. A New Standard vSwitch has been Added with the NEW VLAN configured, OR
6. A new port group has been added to an existing vDS with the new VLAN configured on it.
7. /VSAN/IgnoreClusterMemberListupdates is set to the default value of 0. If this setting is not 0 vCenter will not be able to push out any updates to the unicastagent list on the ESXi hosts resulting in a production outage.
Run the command esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates on all hosts in the cluster to check the current setting.
If the value is 1 please set it back to the default value of 0 via the following command:
esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
+-------------------------------------------------------+
Steps required
First, we need to list the VMks currently configured.
In this sample configuration, we have 4 hosts with IPs ending in -- .81 .82 .83 and .84
**SSH to each host in your vSAN Cluster, login as root and type the following command
# esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ ----------- --------
vmk0 192.168.0.81 255.255.255.0 192.168.0.255 STATIC 192.168.0.1 false
vmk1 192.168.10.81 255.255.255.0 192.168.10.255 STATIC 0.0.0.0 false <<----VLAN10
vmk2 192.168.20.81 255.255.255.0 192.168.20.255 STATIC 0.0.0.0 false <<----VLAN20
Next, we need to know the VMK currently being used by vSAN with this command.
# esxcli vsan network list
You'll see,
# esxcli vsan network list
Interface
VmkNic Name: vmk2 <<----This is the VMK interface used for VSAN
IP Protocol: IP
Interface UUID: 2fe2be5c-d466-97c4-6482-005056016fbb
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan
To move the vSAN Data to the new VLAN, follow these steps:
1. In the vSphere Web Client, create a new VMkernel Interface on each vSAN host - for this KB the new interface is: vmk3
2. Configure the IP Addresses from your new VLAN on these new VMKs for each host, being sure to select the correct vSwitch and/or Portgroup.
3. Enable vSAN on the new VMKs for each host
4. You should now see the new VMK from the CLI with the following command:
# esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ ----------- --------
vmk0 192.168.0.81 255.255.255.0 192.168.0.255 STATIC 192.168.0.1 false
vmk1 192.168.10.81 255.255.255.0 192.168.10.255 STATIC 0.0.0.0 false <<----VLAN10
vmk2 192.168.20.81 255.255.255.0 192.168.20.255 STATIC 0.0.0.0 false <<----VLAN20
vmk3 192.168.30.81 255.255.255.0 192.168.30.255 STATIC 0.0.0.0 false <<----New VMK on VLAN30
5. Now check to make sure vSAN is aware that both VMKs are tagged for vSAN traffic.
# esxcli vsan network list
You should now see output listing two VMKs similar to below.
# esxcli vsan network list
Interface
VmkNic Name: vmk2 <<----Original vSAN VMK on Old VLAN
IP Protocol: IP
Interface UUID: 2fe2be5c-d466-97c4-6482-005056016fbb
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan <<----Tagged for vSAN traffic
Interface
VmkNic Name: vmk3 <<----New vSAN VMK configured on the new VLAN
IP Protocol: IP
Interface UUID: ea12aa5d-84cf-9207-9895-005056016fbb
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan <<----Tagged for vSAN traffic
+---------------------------------------------------+
6. Next, we will test connectivity between all hosts on the new VMK interface (in this example it's vmk3)
This command will use the new VMK interface (vmk3) to ping the IP addresses of vmk3 on the other host.
If using MTU 1500
# vmkping -I vmk3 -s 1472 -d 192.168.30.82
# vmkping -I vmk3 -s 1472 -d 192.168.30.83
# vmkping -I vmk3 -s 1472 -d 192.168.30.84
If using MTU 9000(Jumbo frames)
# vmkping -I vmk3 -s 8972 -d 192.168.30.82
# vmkping -I vmk3 -s 8972 -d 192.168.30.83
# vmkping -I vmk3 -s 8972 -d 192.168.30.84
Output will look similar to this on each ping.
64 bytes from 192.168.30.82: icmp_seq=0 ttl=64 time=0.933 ms
64 bytes from 192.168.30.83: icmp_seq=1 ttl=64 time=0.908 ms
64 bytes from 192.168.30.84: icmp_seq=2 ttl=64 time=0.735 ms
--- ping statistics ---
3 packets transmitted, 3 packets received, +1 duplicates, 0% packet loss
round-trip min/avg/max = 0.735/1.204/1.035 ms
7. When all hosts can ping each other on the new VMK interface, starting with one host.
**Go to the vSphere Web Client
**Highlight the host in the Inventory Pane
**Click the Configure Tab
**Under "Networking" highlight VMkernel Adapters
**Choose the old vmk (vmk2 in this sample) and click Edit (pencil icon)
**Remove the check mark next to vSAN - this disables vSAN traffic on the original VMK
8. Verify all VMs are in an accessible state - this is precautionary, the expectation is that no VMs will be affected.
**If VMs start becoming non-compliant, shutting down, inaccessible, etc.
**Edit the old VMK and check the vSAN box
**Everything should recover immediately after turning vSAN back on for the original VMK interface.
9. When the original VMK on all hosts has had vSAN disabled (check mark removed in edit mode) and all VMs/hosts are still compliant/operating as expected....
**Delete the old VMK Interface**
1. The vSAN Cluster is currently serving production data
2. The expectation is to have vSAN traffic flowing on a new VLAN at the conclusion of this KB without disruption to production VMs.
3. The vSAN hosts are already connected to 10G switch ports configured with the old/existing VLAN
4. The vSAN hosts are already connected to 10G switch ports configured with the new VLAN.
5. A New Standard vSwitch has been Added with the NEW VLAN configured, OR
6. A new port group has been added to an existing vDS with the new VLAN configured on it.
7. /VSAN/IgnoreClusterMemberListupdates is set to the default value of 0. If this setting is not 0 vCenter will not be able to push out any updates to the unicastagent list on the ESXi hosts resulting in a production outage.
Run the command esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListupdates on all hosts in the cluster to check the current setting.
If the value is 1 please set it back to the default value of 0 via the following command:
esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
+-------------------------------------------------------+
Steps required
First, we need to list the VMks currently configured.
In this sample configuration, we have 4 hosts with IPs ending in -- .81 .82 .83 and .84
**SSH to each host in your vSAN Cluster, login as root and type the following command
# esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ ----------- --------
vmk0 192.168.0.81 255.255.255.0 192.168.0.255 STATIC 192.168.0.1 false
vmk1 192.168.10.81 255.255.255.0 192.168.10.255 STATIC 0.0.0.0 false <<----VLAN10
vmk2 192.168.20.81 255.255.255.0 192.168.20.255 STATIC 0.0.0.0 false <<----VLAN20
Next, we need to know the VMK currently being used by vSAN with this command.
# esxcli vsan network list
You'll see,
# esxcli vsan network list
Interface
VmkNic Name: vmk2 <<----This is the VMK interface used for VSAN
IP Protocol: IP
Interface UUID: 2fe2be5c-d466-97c4-6482-005056016fbb
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan
To move the vSAN Data to the new VLAN, follow these steps:
1. In the vSphere Web Client, create a new VMkernel Interface on each vSAN host - for this KB the new interface is: vmk3
2. Configure the IP Addresses from your new VLAN on these new VMKs for each host, being sure to select the correct vSwitch and/or Portgroup.
3. Enable vSAN on the new VMKs for each host
4. You should now see the new VMK from the CLI with the following command:
# esxcli network ip interface ipv4 get
You'll see output similar to below.
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ ----------- --------
vmk0 192.168.0.81 255.255.255.0 192.168.0.255 STATIC 192.168.0.1 false
vmk1 192.168.10.81 255.255.255.0 192.168.10.255 STATIC 0.0.0.0 false <<----VLAN10
vmk2 192.168.20.81 255.255.255.0 192.168.20.255 STATIC 0.0.0.0 false <<----VLAN20
vmk3 192.168.30.81 255.255.255.0 192.168.30.255 STATIC 0.0.0.0 false <<----New VMK on VLAN30
5. Now check to make sure vSAN is aware that both VMKs are tagged for vSAN traffic.
# esxcli vsan network list
You should now see output listing two VMKs similar to below.
# esxcli vsan network list
Interface
VmkNic Name: vmk2 <<----Original vSAN VMK on Old VLAN
IP Protocol: IP
Interface UUID: 2fe2be5c-d466-97c4-6482-005056016fbb
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan <<----Tagged for vSAN traffic
Interface
VmkNic Name: vmk3 <<----New vSAN VMK configured on the new VLAN
IP Protocol: IP
Interface UUID: ea12aa5d-84cf-9207-9895-005056016fbb
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan <<----Tagged for vSAN traffic
+---------------------------------------------------+
6. Next, we will test connectivity between all hosts on the new VMK interface (in this example it's vmk3)
This command will use the new VMK interface (vmk3) to ping the IP addresses of vmk3 on the other host.
If using MTU 1500
# vmkping -I vmk3 -s 1472 -d 192.168.30.82
# vmkping -I vmk3 -s 1472 -d 192.168.30.83
# vmkping -I vmk3 -s 1472 -d 192.168.30.84
If using MTU 9000(Jumbo frames)
# vmkping -I vmk3 -s 8972 -d 192.168.30.82
# vmkping -I vmk3 -s 8972 -d 192.168.30.83
# vmkping -I vmk3 -s 8972 -d 192.168.30.84
Output will look similar to this on each ping.
64 bytes from 192.168.30.82: icmp_seq=0 ttl=64 time=0.933 ms
64 bytes from 192.168.30.83: icmp_seq=1 ttl=64 time=0.908 ms
64 bytes from 192.168.30.84: icmp_seq=2 ttl=64 time=0.735 ms
--- ping statistics ---
3 packets transmitted, 3 packets received, +1 duplicates, 0% packet loss
round-trip min/avg/max = 0.735/1.204/1.035 ms
7. When all hosts can ping each other on the new VMK interface, starting with one host.
**Go to the vSphere Web Client
**Highlight the host in the Inventory Pane
**Click the Configure Tab
**Under "Networking" highlight VMkernel Adapters
**Choose the old vmk (vmk2 in this sample) and click Edit (pencil icon)
**Remove the check mark next to vSAN - this disables vSAN traffic on the original VMK
8. Verify all VMs are in an accessible state - this is precautionary, the expectation is that no VMs will be affected.
**If VMs start becoming non-compliant, shutting down, inaccessible, etc.
**Edit the old VMK and check the vSAN box
**Everything should recover immediately after turning vSAN back on for the original VMK interface.
9. When the original VMK on all hosts has had vSAN disabled (check mark removed in edit mode) and all VMs/hosts are still compliant/operating as expected....
**Delete the old VMK Interface**
Additional Information
Impact/Risks:
**Note: Having 2 vSAN VMK interfaces "up" and connected to 2 different VLANs is NOT a long-term supported solution. Please refer to the following VMWare documentation for vSAN networking best practices:
Best Practices for vSAN Networking
If the steps in this KB Article are followed precisely, there is no risk to network disruption or data availability.
The change is designed to be seamless, without impact to running Virtual Machines.
As a precaution however, this KB advises disabling vSAN on the original VMK interface one host at a time, then briefly monitoring for evidence of VMs becoming non-compliant, shutting down, or becoming Inaccessible, etc. - again this should not happen.
For a more cautious approach, when it's time to disable vSAN on the original VMK interfaces, you can place the host in maintenance mode, disable the original VMK, re-test ping connectivity to all hosts on the new VLAN, then exit maintenance mode and repeat for the next host.
**Note: Having 2 vSAN VMK interfaces "up" and connected to 2 different VLANs is NOT a long-term supported solution. Please refer to the following VMWare documentation for vSAN networking best practices:
Best Practices for vSAN Networking
If the steps in this KB Article are followed precisely, there is no risk to network disruption or data availability.
The change is designed to be seamless, without impact to running Virtual Machines.
As a precaution however, this KB advises disabling vSAN on the original VMK interface one host at a time, then briefly monitoring for evidence of VMs becoming non-compliant, shutting down, or becoming Inaccessible, etc. - again this should not happen.
For a more cautious approach, when it's time to disable vSAN on the original VMK interfaces, you can place the host in maintenance mode, disable the original VMK, re-test ping connectivity to all hosts on the new VLAN, then exit maintenance mode and repeat for the next host.
Feedback
Yes
No
Powered by
