Windows VM with NSX Network Introspection driver lose TCP connectivity
search cancel

Windows VM with NSX Network Introspection driver lose TCP connectivity

book

Article ID: 341277

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:

  • Windows VM with NSX Network Introspection driver (vnetflt.sys) connected to USVM (Guest Introspection SVM) loses temporary TCP network connectivity for new connections.
     
  • Running dmesg command on the USVM console to show the logs, you see entries similar to:
    Out of memory: Kill process <process_id> (java) score <score> or sacrifice child
     
  • In the NSX Manager log, you see entries similar to:
    Code:'260007'
    Event Message: 'Lost communication with ESX module.'

Environment

VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x

Cause

NSX Network Introspection driver (vnetflt.sys) is used to send Network related events to USVM through Multiplexor (MUX). Network events obtained in USVM is used in Activity Monitoring and Identity Firewall.
The driver collects TCP connectivity event and push it to USVM. Since there is memory leak issue in the underlying connection between MUX and USVM, USVM event manager process is restarted due to out of memory. While the event manager process is restarting, TCP connecting event processing stays incomplete for a while and may result connectivity issue in Windows VM.

Resolution

This issue is resolved in:
To work around this issue if you do not want to upgrade, disable the NSX Network Introspection driver.
 
To disable the vnetflt.sys driver:
 
Note: This procedure modifies the Windows registry. Before making any registry modifications, ensure that you have a current and valid backup of the registry and the virtual machine. For more information on backing up and restoring the registry, see the Microsoft Knowledge Base article 136393.
  1. Connect to the affected virtual machine with a console or RDP sessions.
  2. Click Start > run, type regedit and click OK.
  3. Navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vnetflt\
  4. Right-click the Start key and select Modify.
  5. Change the value to 4 and click OK.
  6. Close the Registry Editor Window.
  7. Reboot the virtual machine.
Notes:
  • If you are using Agentless AV only, disabling the NSX Network Introspection Driver does not affect Agentless AV functionality.
  • If you have IDFW, use the Active Directory Event Log Scraper instead of USVM. For more information on the Event Log Scraper, see the Identity Firewall Overview section of the NSX Administration Guide.

Additional Information

You experience these additional symptoms:

ESXi syslog contains entries similar to:

<timestamp>:03Z EPSecMux[23051084]: [WARNING] (EPSEC) [0x15fbb4c] SolutionHandler[0x1f001cd8] failed to connect to solution[100] at [169.254.1.24:48655]: Connection refused (111)
<timestamp>:03Z EPSecMux[23051084]: [WARNING] (EPSEC) [0x15fbb4c] SolutionHandler[0x1f001cd8] scheduling reconnect to solution[100] at 169.254.1.24:48655 in 100 ms


Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

VM network timeouts / packet drops after upgrading to VMware Tools 11.x with Guest Introspection Driver on ESXi 6.5/6.7

Powered by Wolken Software