Search the VMware Knowledge Base (KB)
View by Article ID

Troubleshooting NSX for vSphere 6.x Distributed Firewall (DFW) (2125437)

  • 4 Ratings

Symptoms

  • Publishing Distributed Firewall rules fails
  • Updating Distributed Firewall rules fails

Purpose

This article provides information on understanding and troubleshooting VMware NSX for vSphere 6.x Distributed Firewall (DFW). For more information, see the NSX Distributed Firewall (DFW) section in the VMware NSX for vSphere (NSX-V) Network Virtualization Design Guide.

Resolution

Notes
  1. Verify that these prerequisites are met to run Distributed Firewall (DFW):

    •     VMware vCenter Server 5.5
    •     VMware ESXi 5.1 or ESXi 5.5
    •     VMware NSX for vSphere 6.0 and later

  2. Verify that the DFW VIBs are successfully installed on each of the ESXi hosts in the cluster.

    On each of the ESXi host, run this command:

    esxcli software vib list

    For example:

    # esxcli software vib list | grep esx-vsip

    esx-vsip   5.5.0-0.0.2318233   VMware   VMwareCertified   2015-01-24


    # esxcli software vib list | grep dvfilter

    esx-dvfilter-switch-security   5.5.0-0.0.2318233   VMware  VMwareCertified   2015-01-24


  3. Verify that the vShield-Stateful-Firewall service is in a running state using this command:

    /etc/init.d/vShield-Stateful-Firewall status

    For example:

    # /etc/init.d/vShield-Stateful-Firewall status

    vShield-Stateful-Firewall is running


  4. Verify that the Message Bus is communicating with the NSX Manager.

    Note: The process is automatically launched by the watchdog script and restarts the process if it terminates for an unknown reason.

    Run this command on each of the ESXi hosts on the cluster:

    ps |grep vsfwd

    For example:

    ps |grep vsfwd

    107557 107557 vsfwd /usr/lib/vmware/vsfw/vsfwd
    107574 107557 vsfwd /usr/lib/vmware/vsfw/vsfwd
    107575 107557 vsfwd /usr/lib/vmware/vsfw/vsfwd

  5. Verify that port 5671 is opened for communication in the firewall configuration.
    For more information, see:

To validate that there are active messaging bus connection, run this command on each of the ESXi hosts on the cluster:

esxcli network ip connection list |grep 5671

  1. Verify that the firewall rules are deployed on a host and are applied to virtual machines.

    1. Log in to the NSX Manager with the admin credentials.
    2. To show a summary of DVFilter information, run this command:

      show dfw host host-id summarize-dvfilter

      Note: You can run the show cluster all command to get host-id information.

      For example:

      show dfw host host-28 summarize-dvfilter

      Fastpaths:
      agent: dvfilter-faulter, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfilter
      agent: dvfilter-generic-vmware, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfilter-generic-fastpath
      agent: dvfg-igmp, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfg-igmp
      agent: dvfilter-generic-vmware-swsec, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfilter-switch-security
      agent: bridgelearningfilter, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: vdrb
      agent: vmware-sfw, refCount: 2, rev: 0x1010000, apiRev: 0x1010000, module: vsip <= DFW module is loaded and running
      Slowpaths:
      slowPath: 4, agent serviceinstance-1, refCount: 2, rev: 0x4, apiRev: 0x4, capabilities: csum

      Filters:
      world 3222538 vmm0:win-xp-64-pro-sp2-133 vcUuid:'50 3f e3 37 83 6f 5a fd-60 7a 69 0a 9c 3d d3 1f'
      port 50331655 win-xp-64-pro-sp2-133.eth0
      vNic slot 2
      name: nic-3222538-eth0-vmware-sfw.2
      agentName: vmware-sfw        <================ DFW filter
      state: IOChain Attached
      vmState: Detached
      failurePolicy: failClosed
      slowPathID: none
      filter source: VMX File
      vNic slot 4
      name: nic-3222538-eth0-serviceinstance-1.4
      agentName: serviceinstance-1
      state: IOChain Attached
      vmState: Attached
      failurePolicy: failOpen
      slowPathID: 4
      filter source: Dynamic Filter Creation

    3. To display the rules configured on the filter, run this command:

      show dfw host host-id vnic vnic-id filter filter-name rules

      For example:

      show dfw host host-34  vnic 501f02cc-9078-22ad-0186-4a05d7f18978.000 filter nic-3222538-eth0-vmware-sfw.2 rules

    4. To display the addrsets configured on the filter, run this command show dfw host host-id vnic vnic-id filter filter-name addrsets

      For example:

      show dfw host host-34  vnic 501f02cc-9078-22ad-0186-4a05d7f18978.000 ilter nic-3222538-eth0-vmware-sfw.2 addrsets

      For more information, see the NSX Central Commands section of the NSX Command Line Interface Reference Guide.
 
Note: If you have validated each of these troubleshooting steps and cannot publish firewall rules to the host virtual machines, execute a host-level force synchronization using the NSX Manager UI or use the following REST API: 
URL : [https:]https://<nsx-mgr-ip>/api/4.0/firewall/forceSync/<host-id>
HTTP Method : POST 
Headers , 
Authorization : base64encoded value of username password
Accept : application/xml
Content-Type : application/xml

Additional Information

Log location

VMware NSX for vSphere 6.0.x:

  • /var/log/vmkernel.log file on the ESXi host
  • /var/log/vsfwd.log file on the ESXi host

VMware NSX for vSphere 6.1.x and later:

  • /var/log/vmkernel.log file on the ESXi host
  • /var/log/dfwpktlogs.log file on the ESXi host
  • /var/log/vsfwd.log file on the ESXi host

For more information, see Location of VMware NSX for vSphere 6.1.x and later Firewall Rule logs (2128082).

 
Using the export host-tech-support Central CLI command

Starting with NSX 6.2.4, the export host-tech-support Central CLI command is available. This command allows users to export an ESXi host Support bundle to a specified server. In addition, this command collects NSX related outputs and files (not limited to the following) on specified hosts such as:
  • vmkernel and vsfwd log files
  • list of filters
  • list of dfw rules
  • list of containers
  • spoofguard details
  • host related information
  • ipdiscovery related info
  • rmq command outputs
  • security group and services profile and instance details
  • esxcli related outputs

To collect NSX related outputs and files:

  1. Log in to the NSX Manager using the admin credentials.
  2. Run this command:

    export host-tech-support host-id scp uid@ip:/path

    Notes:
    • This command generates the NSX tech-support bundle and copies it to a specified server.
    • This also removes any temporary files on the NSX Manager.
    • Run the show cluster all command to get host-id information

ESXi Host Command Details

nsx-support

  • Usage: /bin/nsx-support {-h|start|getstatus|cleanup} [<datastore_name>]
  • Command outputs with different command arguments
  • nsx-support start [<datastore_name>]
  • If the command arguments are normal, it returns In progress
  • If nsx-support start <datastore_name> is not correct, for example: nsx-support start abc, you see output similar to:

    Path does not exist: /vmfs/volumes/abc. Please specify output datastore name.
nsx-support getstatus
  • If there is an available log bundle, it returns the absolute directory of the bundle in the datastore.

    For example:

    /vmfs/volumes/”{datastoreName}"/esx-prmh-nsx-dfw-dhcp-78-123.eng.vmware.com-2015-11-17--19.35.tgz.
  • Otherwise, it returns No NSX tech support bundle found.

    nsx-support delete


    Returns Done

Tags

DFW, Troubleshooting Distributed Firewall, Troubleshooting DFW

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 4 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 4 Ratings
Actions
KB: