This article provides information about Mellanox Spectrum SN2000 Series Switches Hardware VXLAN Gateway integration and compatibility with NSX.

Disclaimer: Mellanox Onyx referenced in this article is Mellanox’s network operating system that is developed and supported by Mellanox. Use of this product is also governed by the end user license agreement of Mellanox. You must obtain from Mellanox the OS, support, and the licensing for using this product. For more information, see Mellanox Technologies.

VMware NSX for vSphere 6.4.x

Mellanox Onyx version 3.7.1200

This paper is focused on Mellanox Onyx integration with VMware NSX.

The Mellanox Onyx Advanced Ethernet Operating System delivers a new level of flexibility and scalability to next-generation data centers with storage, cloud, financial, or media & entertainment fabrics. Featuring a robust Layer 3 protocols stack, built-in workflow automation, monitoring & visibility tools, and high availability mechanisms, Onyx enables highly scalable leaf-spine deployments and simplifies network processes and workflows, increasing efficiencies and reducing operating expenses and time-to-service.

Onyx is running on Mellanox Spectrum series of switches and provides the VXLAN Tunnel End-Point (VTEP) network endpoint, enabling layer-2 overlay (tunneling) over layer-3 infrastructure between virtual and physical machines.

This common use case contains the following components:

• ESXi servers pre-configured with VXLAN networking using VMware NSX
• Network virtualization platform (NSX) controllers
• Mellanox Spectrum switch running VXLAN HW GW, connected to the ESXi servers and to physical servers
• Out-of-band network for management and a VLAN network to carry VXLAN traffic

This is the first certification release between VMware NSX 6.44 and Mellanox Onyx 3.7.1200. 

Please refer to the steps below describing the setup of the solution.

VMware used the topology below as an example of a common setup where virtual machines running on top of ESXi servers communicate to a database server over VXLAN.



Mellanox Onyx – Switch Configuration

1. Set MTU to Jumbo frames on the relevant switch ports (This should be aligned with the ESXi servers and database server). Run:
   switch (config) # interface ethernet 1/1-1/3 mtu 9216 force
    
2. Create VLAN 3 to carry VXLAN traffic. Run:
   switch (config) # vlan 3
    
3. Set the switch interfaces towards the ESXi servers to be part of VLAN 3 in trunk mode. Run:
   switch (config) # interface ethernet 1 switchport mode trunk
   switch (config) # interface ethernet 2 switchport mode trunk
    
4. Enable IP routing. Run:
   switch (config) # ip routing vrf default
    
5. Create a VLAN interface and assign it an IP address.
   
   The IP address needs to be the default gateway of the "vxlan" netstack created by NSX after enabling VXLAN traffic on the hosts.
   
   Note: The traffic can be routed between the ESXi servers and the Spectrum switch through other routers, in this example we directly connect the Spectrum switch to the ESXi servers.
   switch (config) # interface vlan 3
   switch (config) # interface vlan 3 ip address 33.33.33.254 255.255.255.0
   switch (config) # interface vlan 3 mtu 9216
   
   Note: To check the default gateway in vSphere web client, select an ESXi host and go to Configure -> TCP/IP configuration.
   
   
6. Create a loopback interface. This interface will be the VTEP IP address assigned to this switch, which will communicate with the VTEPs on the ESXi servers by routing through "interface vlan 3". Run:
   switch (config) # interface loopback 1
   switch (config) # interface loopback 1 ip address 1.2.3.4 255.255.255.255
    
7. Enable the Network Virtualization Edge (NVE) protocol, create an NVE interface, and specify that it is the source interface (loopback 1). The following commands enable VTEP on the switch and assign loopback1 as the interface to handle all Tx/Rx VXLAN traffic. Run:
   switch (config) # protocol nve
   switch (config) # interface nve 1
   switch (config) # interface nve 1 vxlan source interface loopback 1
    
8. Start the ovsdb server and connect it to the NSX controllers. Run:
   switch (config) # ovs ovsdb server
   switch (config) # ovs ovsdb manager remote ssl ip address 10.130.200.100
   switch (config) # ovs ovsdb manager remote ssl ip address 10.144.200.101
   switch (config) # ovs ovsdb manager remote ssl ip address 10.144.200.102
    
9. Configure the port facing the database server as an NVE port. Run:
   switch (config) # interface ethernet 1/3 nve mode only force
   Note: The mode "switchport" is controlled via the NSX Controller and cannot be configured manually once the port is in NVE mode.
    
10. Obtain the switch certificate. You will need this certificate when you configure NSX Manager later.
    switch (config) # show crypto certificate name system-self-signed public-pem
    ...
    Copy the certificate starting with the line:
    -----BEGIN CERTIFICATE-----
    Until the line:
    -----END CERTIFICATE-----
    Make sure to include both of those lines.

NSX Controller Configuration

Add the Spectrum Switch

This section describes how to configure the NSX Controller to add the Spectrum switch.

1. Add hosts to the replication cluster. Go to "Service Definitions" and select "Hardware Devices", Under "Replication Cluster", click Edit. Add the two ESXi servers to the replication cluster.
   Note: All hosts you add to the replication cluster can replicate Broadcast, Unknown unicast and Multicast (BUM) traffic to other ESXi servers. When the switch needs to send BUM traffic to a virtual machine, it will select one of the hosts in the replication cluster and send the traffic to it. The host will then replicate it to all other ESXi hosts.
   Note: It is recommended that you add at least two ESXi servers to the replication cluster for redundancy.
   
   
   
2. Add the Mellanox Spectrum switch to NSX.
   Under the Hardware Devices tab click the plus sign to add a new hardware device. Enter a name for the new hardware device. Enter the switch certificate obtained earlier, and click OK.
   
   

1. In NSX Manager select "Logical Switches". Right click the logical switch which you want to map to the physical switch port and select "Manage Hardware Bindings".
   Note: The segment ID is the VNI.
   
   
2. Click the plus sign to add a new mapping. Click Select under the port column and select port "eth3". This corresponds to “interface ethernet 1/3" you configured earlier as an NVE port in the switch. Under the VLAN column, set the VLAN that will map this logical switch to this specific switch port. You can have multiple logical switches mapped to the same port on a different VLAN (for example to connect a firewall appliance to logical switches).
   Note: For an "access" (untagged) configuration use VLAN 0 and click OK. In this case, we have no VLAN configured on the database server, therefore we choose VLAN 0.
   
   

1. Configure the required IP address on the database server in the same subnet as the virtual machines in the logical switch.
   Note: If you used a VXLAN in the previous step also configure the VLAN in the database server OS.
    
2. Ping between a virtual machine connected to the logical switch and the database server.
3. On the switch, run:
   
   switch (config) # show interfaces nve 1 mac-address-table
   NVE Interface  Logical Switch        VNI ID  Mac Address        Address Type        Remote Endpoint IP Address
   -------------  --------------        ------  -----------        ------------        --------------------------
   1              240ac937-1ec2-371a-a  5002    00:50:56:99:33:09  remote configured   33.33.33.1
   1              240ac937-1ec2-371a-a  5002    00:50:56:99:98:14  remote configured   33.33.33.2
   
   You should see the MAC addresses of all the virtual machines in the connected logical switch and the MAC address of the database server.
    
4. Check the NVE Interface for status and counters.
   
   switch (config) # show interfaces nve
   Remote Manager IP Address  Port  Connection Type
   -------------------------  ----  ---------------
   10.130.200.100             6640  ssl
   10.144.200.101             6640  ssl
   10.144.200.102             6640  ssl
   Interface NVE 1 status:
   Admin state: up
   Source interface: loopback 1
   NVE member interfaces: Eth1/1
    0 encapsulated (Tx) NVE packets
    0 decapsulated (Rx) NVE packets
    0 dropped NVE-encapsulated packets
    0 NVE-encapsulated packets with errors
    
5. Get the peer list per VNI.
   
   switch (config) # show interfaces nve 1 peers
   NVE Interface   Logical Switch         VNI ID   Peer IP Address
   -------------   --------------         ------   ----------------
   1               415585bd-389b-3965-9   5002     33.33.33.1
   1               415585bd-389b-3965-9   5002     33.33.33.2
   
   Note: Refer to Mellanox Onyx User Manual for full list of commands.