vMotion between VDS/VSS and NSX-T virtual switch

Products

VMware NSX Networking

Issue/Introduction

vMotion is achievable between VDS/VSS and an NSX-T virtual switch (N-VDS or VDS running NSX).

Environment

VMware NSX for vSphere 6.4.x

Resolution

Preamble

vMotion from NSX-T to VDS/VSS is not covered in this document.

Requirements

As of NSX-T 3.0, the NSX-T virtual switch can be either an N-VDS or a VDS prepared for NSX. The following matrix is applicable to both NSX-T virtual switch models.

Here is the matrix listing the supported versions for the vMotion features.

vMotion		vSphere (vCenter Server and ESXi)	NSX-T Version
Source	Destination
VDS DVPG (no NSX)	NSX-T VLAN segment	6.7U1 or higher 6.5P03 or higher		2.2 or higher
VDS DVPG (no NSX)	NSX-T overlay segment	6.7U2 or later		2.2 or higher
VSS PG	NSX-T VLAN segment	6.7U2 or later		2.2 or higher
VSS PG	NSX-T overlay segment	6.7U2 or later		2.2 or higher
VXLAN DVPG (NSX for vSphere overlay)	NSX-T overlay segment	6.7U2 or later		2.5.2 or higher
VDS (on NSX prepared host)	NSX-T VLAN segment	6.7U2 or later		2.5.2 or higher
VDS (on NSX prepared host)	NSX-T overlay segment	6.7U2 or later		2.5.2 or higher

Prior to NSX-T 2.5.2, vMotion between NSX-V (on VDS) and NSX-T is not supported especially with DFW enabled. Only cold migration is supported. For additional information, see #6 under Limitations below.
Starting with NSX-T 2.5.2, it is possible to vMotion between an NSX for vSphere VXLAN-backed logical switch to an NSX-T overlay segment. Concerning DFW: See #6 under limitations.
During vMotion, source switch configurations and runtime state will not be carried over or applied at the destination as a part of vMotion, in both directions. It is expected that the destination network be prepared to be in parity with the source switch configurations, before a vMotion is initiated. This is applicable for features including (but not limited to), NIOC, Spoofguard and Switch Security.
With NSX-V > 6.4.4, the firewall state is carried over.

Note when using the HTML5 vSphere Client, vmotion between vDS and NSX-T (VDS or N-VDS) precheck may fail with the error:

Currently connected network interface 'Network adapter 1' uses network 'DVSwitch: uuid', which is not accessible.

This issue is resolved in VMware vCenter Server 6.7 Update 3g.
As a workaround on impacted versions, the vmotion is possible using the Flash vSphere Client.

Note: Adobe Flash Player is now End of Life (EOL) as of Dec 31, 2020. For more information, see VMware Flash End of Life and Supportability (78589).

Limitations

When migrating from VDS/VSS to NSX-T virtual switch, traffic filter (NIOC, Spoofguard and Switch Security, not including DFW) and QOS marking policy rules created for the vNIC ports on the VM being migrated are not transferred to the destination. Traffic filter and QOS marking policy rules are applied on a portset level and the rules configured on the portset are applied back when the VM is migrated back to the VDS/VSS host. You have to define Segment Profiles in NSX-T to recreate these types of filters when you plan to migrate from VDS/VSS to NSX-T and traffic filtering and QOS marking policy are used.
When migrating VMs from VDS/VSS to NSX-T virtual switch, it is possible for traffic to be blocked based on a certain combination of NSX-T configurations. One combination is having Spoofguard enabled and having only DHCP snooping enabled. Spoofguard derives IP addresses for its whitelist from IP Discovery. After migrating to NSX-T, the VM may or may not perform a DHCP transaction to renew its assigned DHCP IP address with the DHCP server. The IP will only be discovered when a DHCP transaction happens. Until that happens, the Spoofguard allow list will be empty and traffic transmitted from the VM will be dropped.

It is required to have Spoofguard disabled during the migration in NSX-V, and on post-migration, enable Spoofguard in NSX-T once IP Discovery discovered the necessary set of IP addresses for enforcement.
It is required that both the source and destination ESXi hosts are running the supported versions as mentioned in above table, besides the supported vCenter build version.

In the case the vMotion is initiated from a host running an un-supported ESXi version, if in-case of a failure, rollback will be disruptive. All vNIC ports of the VM will lose network connectivity. Once the VM is in this state, it is strongly advised against any attempt to reconfigure the VM. To safely recover, power off the VM and then reconfigure the vNIC ports of the VM to the desired network.
If NIOC/traffic shaping or bandwidth reservation is configured per VM vNIC and the NIOC requirements cannot be met by the destination host, the port will end up in disconnected state. To resolve this, either the vNIC bandwidth reservation needs to be updated to a value that can be satisfied by the destination host and connected back or the bandwidth reservation needs to be removed on all the vNICs before initiating the vMotion.
vMotion of Container VMs between VSS/VDS and NSX-T virtual switch is not supported.
Distributed Firewall and vMotion between VDS on NSX-V and NSX-T:

To allow vMotion without impact you need the export version of the firewall to be at least 1000. This will allow to have firewall state carried over. For more information, see the Configure Export Version of Distributed Firewall Filter on Hosts section of the NSX-T Data Center Migration Coordinator Guide.

In case of inconsistent filters, the VM port can be disabled at destination, requiring the interface to be disabled/enabled or the port to be disconnected and reconnected.

A workaround to updating the filter is to put the VM in the NSX-V exclusion list before doing vMotion.

Notes: Pay attention of DFW Rules when a VM is migrated from NSX-V VDS to NSX-T virtual switch. Your DFW Rules MUST be configured to cover what you are intended to do:

At the Destination
And at the Source

Example:

If you use Dynamic Grouping in NSX-V and your VM is moved to NSX-T on a new vCenter, due to the fact the VM is no more in the inventory of the Source vCenter, then, the VM is no more part of the Dynamic Group. In this case, you might have to update the DFW Rules at the source to keep the VM in the Dynamic Grouping by using IP Based Grouping, for instance.

If the DFW Filter version cannot be changed, then, prior to vMotion, place the VMs in the exclusion list on destination NSX-T prepared host.

DFW Rules Design

You need to take care of the flow FROM this migrated VM and TO this migrated VM in the NSX-T DFW rules.
If you are using Dynamic Group Membership in NSX-T, then the moved VM should be directly included in a Group with the correct firewall rules.

Important: If you are using grouping based on tag, you have to add a tag to this VM once the VM is migrated and seen by NSX-T.

Using Exclusion List at the destination can help. Pay attention, ALL the VM traffic will be ALLOWED. If it's not permitted, then you may have to play with grouping based on IP address.
You need to take care of the flow FROM this migrated VM and TO this migrated VM in the NSX-V DFW rules too. If you are using Dynamic Group Membership at the source in NSX-V, you should use IP Based Grouping to cover the gap created by the move IF it is no more in the vCenter inventory (moving to another VC case as described in this example).

What is this gap?

When moving the VM outside the inventory that NSX-V can see (to another VC), the VM would not be part of any Security Group Membership using Dynamic Grouping.

For example: SG1 based on Dynamic Group based on VM Name containing "WEB". This SG1 is used as a Dest in a firewall rule allowing traffic TO this SG1.

When moving the VM to a Logical Switch not managed by NSX-V and not seen by the VC where NSX-V is, the object is no more in the SG1 group. In zero trust model, this involves the traffic to the VM is denied.

How to cover this gap?

Before moving the VM, an IPSet must be added to the SG1 with the IP Address of the moving VM.
If you are using Static Group Membership, then, even if the VM is moved, it will be part of these Static Grouping.

Additional Information

Cross vCenter Migration and Clone requirements in VMware vSphere 6.0 and later (2106952)