VMXNET3 resource considerations on a Windows virtual machine that has vSphere DirectPath I/O with vMotion enabled
Article ID: 343839
Updated On:
Products
Issue/Introduction
In a Cisco Unified Computing System (UCS) that has Cisco Virtual Machine Fabric Extender (VM-FEX) deployed, you can configure resources and offload settings for VMXNET3 adapters in an adapter policy and apply the policy to multiple hosts. The policy must have enough queue and interrupt resources to satisfy the VMXNET3 driver configuration in the guest operating system. If you use the vSphere DirectPath I/O with vMotion passthrough technology for high networking performance, the number of transmit (Tx) queues, receive (Rx) queues, completion queues (CQs), and interrupt vectors set in the VMWarePassThru adapter policy in the UCS Manager must match the numbers that are configured for the VMXNET3 driver in the guest operating system.
For information about configuring Cisco VM-FEX on vSphere, see the Cisco UCS B-Series Servers documentation.
For best practices in configuring Cisco VM-FEX and vSphere including vSphere DirectPath I/O with vMotion, see Cisco VM-FEX Best Practices for VMware ESX Environment Deployment Guide. For troubleshooting Cisco VM-FEX switch on vSphere, see Cisco VM-FEX Using VMware ESX Environment Troubleshooting Guide.
For information about resouce considerations for using VMXNET3 with vSphere DirectPath I/O with vMotion on Linux virtual machines, see Managing VMXNET3 resource requirements on a Linux virtual machine that has vSphere DirectPath I/O with vMotion enabled (2058349).
Environment
VMware vSphere ESXi 5.0
VMware vCenter Server 6.0.x
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5
Resolution
Resource allocation for VMXNET3 networking in vSphere DirectPath I/O with vMotion
For a multiprocessor system, such as an ESXi host, increasing the number of queues increases the data that is transmitted and received in parallel. Transmitting and receiving data in parallel increases throughput and reduces latency. The packets in a particular transmit queue or receive queue can be processed by a specific virtual CPU. For improved performance you can use as many queues as the number of virtual CPUs in a virtual machine.
To send or receive packets simultaneously in multiple queues on a host, processing of each queue must be triggered by an individual interrupt line. In a system that requires many interrupts, MSI or MSI-X interrupts are used instead of wired ones. The default interrupt mode for the VMXNET3 adapter in passthrough mode is MSI-X. In passthrough mode, VMXNET3 adapters do not work with INTx interrupts.
One MSI or MSI-X interrupt vector is typically assigned to one queue. When a VMXNET3 adapter that is used for vSphere DirectPath I/O with vMotion sends or receives data, the interrupt vectors that are assigned to the adapter are allocated directly on the physical host. An ESXi host has a finite number of interrupt vectors for I/O device operations. Under certain conditions, the number of vectors might be exhausted when a great number of virtual I/O device requests are generated, for example, in workloads with high packet rates and many queues.
Determining resource requirements
On a Windows guest operating system, the default amount of resources that the VMXNET3 driver consumes depends on whether Receive Side Scaling (RSS) is enabled. By default, RSS is disabled for the VMXNET3 driver.
To check whether RSS is enabled for all virtual machine adapters on the Windows guest operating system, run the NetShell command-line tool in the guest operating system.
-
At the command prompt, run the following console command:
netsh int tcp show global - In the command output, verify that the
Receive-Side Scaling Stateparameter is enabled.
Allocated resources cannot be examined from within the guest operating system unless they have been manually configured. The upper bound of the resources that are used in Windows for an VMXNET3 adapter is as follows:
| RSS State | Resource Type | Resource Amount or Mode |
| Disabled | Number of Tx queues | 1 |
| Number of Rx queues | 1 | |
| Number of CQs | 2 | |
| Number of interrupt vectors | 2 | |
| Interrupt mode | MSI-X | |
| Enabled | Number of Tx queues | The VMXNET3 driver limits the number of Tx queues in the following way:
The following expression represents the calculation: |
| Number of Rx queues | The VMXNET3 driver limits the number of Rx queues in the following way:
The following expression represents the calculation: | |
| Number of CQs | Number of Tx queues + Number of Rx queues | |
| Number of interrupt vectors | Max(Number of Tx queues, Number of Rx queues) | |
| Interrupt Mode | MSI-X |
Configuring resources
To configure the resources and the RSS mode in the Windows guest operating system, from the Device Manager edit the advanced settings of the VMXNET3 adapter.
- In the Windows Device Manager, expand Network adapters.
- Right-click the VMXNET3 adapter whose resource settings you want to configure and select Properties.
- In the adapter properties dialog box, click the Advanced tab.
- Select the resource configuration property and edit its value to meet the requirements of the virtualized workload.
- Click OK.
Certain restrictions apply to the maximum resources that can be configured to a VMXNET3 adapter.
| Resource Type | Configuration Restrictions |
| Maximum number of Tx queue | Cannot exceed the number of vCPUs. |
| Maximum number of Rx queue | Cannot exceed the number of vCPUs. |
| Maximum number of RSS processors | Cannot exceed the number of vCPUs. |
| Interrupt mode and number of interrupt vectors | Cannot be configured on the Windows guest operating system. The mode is always MSI-X. The VMXNET3 driver determines the number of interrupt vectors per adapter. The number of interrupt vectors does not exceed Number of CQs + 2. |
Resource requirements for VMXNET3 driver 1.2.22.0 or later
Version 1.2.22.0 of the VMXNET3 driver or later supports vSphere DirectPath I/O with vMotion for Windows guest operating systems. The queues and interrupt vectors that VMXNET3 driver 1.2.22.0requires when in passthrough mode are as follows:
| Windows Guest Operating System | RSS State | Resource Type | Requirements |
| Disabled | Number of Tx queues | 1 |
| Number of Rx queues | 1 | ||
| Number of interrupt vectors | 2 | ||
| Interrupt mode | MSI-X | ||
| Enabled VMXNET3 adapter is configured with one Tx queue | Number of Tx queues | 1 |
| Number of Rx queues | Determine the required number of Rx queues in the following way:
The following expression represents the calculation: | ||
| Number of interrupt vectors | Number of Rx queues + 2 | ||
| Interrupt mode | MSI-X | ||
| Enabled VMXNET3 adapter is configured with multiple Tx queue | Number of Tx queues | Determine the required number of Tx queues in the following way:
The following expression represents the calculation: |
| Number of Rx queues | Determine the required number of Rx queues in the following way:
The following expression represents the calculation: | ||
| Number of interrupt vectors | Number of Rx queues + 1 | ||
| Interrupt mode | MSI-X |
Additional Information
For translated versions of this article, see:Feedback
