Search the VMware Knowledge Base (KB)
View by Article ID

Troubleshooting IPSec VPN in NSX for vSphere 6.x (2123580)

  • 1 Ratings

Purpose

This article provides information about troubleshooting IPSec VPN in VMware NSX for vSphere.

Resolution

Basic IPSec tunnel configuration check

NSX Edge supports site-to-site IPSec VPN between an NSX Edge instance and remote sites. The IPSec VPN tunnel is made up of two ends. The tunnel must be consistent on both sides, including:
  • IP subnets
  • Encryption standard

Ports that must be open on all components of the IPSec tunnel

IKE (Internet Key Exchange) uses these ports to generate keys to secure traffic: 
  • Port 500 is used if there is no NAT device between the endpoints
  • Port 4500 is used if there is a NAT device between the endpoints

For more information about basic IPSec configuration and design, see:

Common misconfigurations

  1. MTU configuration on the vDS is set too low. This causes packet fragmentation which results in tunnel creation failure.
  2. Some third party solutions offer an aggressive negotiation mode. NSX for vSphere only supports standard negotiation mode.
  3. Virtual machine configured for IPv6 communication through the IPSec tunnel. NSX-V does not currently support IPv6.

Troubleshooting IPSec VPN tunnel creation and connectivity issues

  1. Gather a complete log bundle from both sites, see:

  2. Gather logs from any related 3rd party solutions.
  3. Check the IPsec configuration on both sides of the Edge by running this command on the NSX Edge CLI:

    # show config ipsec

    See the Additional Information section for an IPsec configuration example.

    Note: If feasible in your environment, you may find it easier to review and capture the output of these commands using SSH. For more information on enabling SSH on the Edge appliance, see the Logging In and Out of the CLI section of the NSX Command Line Interface Reference guide.

  4. On NSX Edge, record the real time status when the issue is occurring. Run these commands and record the results:

    1. Run this command to check the VPN service status:

      #  show service ipsec

    2. Run this command to check Security Policy (SP) status:

      #  show service ipsec sp

    3. Run this command to check Security Associations (SA) status:

      # show service ipsec sa

  5. While the issue is still occurring, run these three commands again on the 3rd party device side and record the results.
  6. Review the captured service output and logging for issues. Verify that the IPSec service is running, security polices are created, security associations between the devices are configured.

    Common errors are:
    • Invalid ID: INVALID_ID_INFORMATION or PAYLOAD_MALFORMED
    • No trusted CA: INVALID_KEY_INFORMATION or a more specific error.
      For example:
      no RSA public key known for 'C=CN, ST=BJ, O=VMWare, OU=CINS, CN=left‘, or PAYLOAD_MALFORMED
    • Proposed proxy-id is not found: INVALID_ID_INFORMATION or PAYLOAD_MALFORMED
    • DPD no response from peer.
      For example:
      DPD: No response from peer - declaring peer dead
    • For configuration issue examples, see the Troubleshooting NSX Edge Configuration Example section in the NSX Administration Guide. The issue examples include:
      • Phase 1 Policy Not Matching
      • Phase 2 Not Matching
      • PFS Mismatch
      • PSK not Matching

  7. Set up a packet capture for IKE packets and/or ESP packets on the NSX Edge side. See the Packet Capture for a Successful Negotiation section in the NSX Administration Guide for a working example.

    • Set up packet capture at point 1 and 2
    • Ping from VM1 to host2
    • Ping from host2 to VM1
    • Check at which point the packet is failed or dropped



  8. Review all collected data and analyze. This will help you distinguish where the problem is. For example:
    • Phase 1 error
    • Phase 2 error
    • Data path error
    • Tunnel is down
    • Which direction is the tunnel connection down?

Troubleshooting tunnel instability issues

When facing a tunnel instability issue, gather and review this information to troubleshoot the issue:

  1. Gather the VMware NSX for vSphere logs as detailed in step 1 of the previous section.
  2. Ensure the NSX Edge VPN is correctly configured to work with any third party hardware VPN firewall solutions (e.g Sonicwall/Watchguard). For more information, see the NSX Edge VPN Configuration Examples section in the NSX Administration Guide, or contact your vendor for more specific configuration information.
  3. Set up a packet capture of IKE/ESP packets between the Edge and third party firewall, refer to the examples in step 5 in the IPSec VPN issues section.
  4. On the NSX Edge, record the real time status when the issue is occurring. Run these commands and record the results:

    1. Run this command to check the VPN service status:

      #  show service ipsec

    2. Run this command to check Security Policy (SP) status:

      #  show service ipsec sp

    3. Run this command to check Security Associations (SA) status:

      # show service ipsec sa

  5. Review and analyze the captured service output for issues. The examples in step 6 of the Troubleshooting IPSec VPN tunnel creation and connectivity issues section are also relevant here.
  6. While the issue is occurring, capture the runtime, traffic state, and the packet capture data of the entire data path. A ping from a private subnet on one side of the IPSec tunnel to another private subnet on the other side of the IPSec tunnel will reveal where the traffic is having issues. 

    1. Set up packet capture at point 1,2,3,4.
    2. Ping from VM1 to host2.
    3. Ping from host2 to VM1.
    4. Check at which point the packet is failed or dropped.
      • Ensure packets can be sent and received on UDP ports 500 and 4500
      • Ensure firewall rules allow ports 500 and 4500
      • Ensure firewall rules allow Encapsulating Security Payload (ESP) packets
      • Ensure local subnet routing over IPSec interface is correctly configured
      • Check MTU configuration for fragmentation issues by sending a small ping payload and then a larger ping payload to the IP at the end of the tunnel. 
        For example: 
        ping -s 500 Host 2 IPping -s 2000 Host 2 IP

Enhanced Operational Manageability in NSX 6.2.4

Prior to NSX 6.2.4, IPSec logging is disabled by default. Customers are then asked to enable this logging when issue has already occurred. There are issues that are not reproducible hence enabling logging after the issue has occurred is no longer useful in troubleshooting.

Starting with NSX for vSphere 6.2.4, the IPSec logging has now been enabled by default and it has been set to WARNING. This ensures that issues are logged from the beginning when IPSec issues are encountered which drastically improves troubleshooting.

Additional Information

For more information about IPSec VPN tunnels in NSX for vSphere, see the IPSec VPN Overview of the NSX Administration Guide.

For more information about the status check commands used, see the NSX 6.1 Command Line Reference guide.

IPsec configuration example


Key information is in blue bold

# show config ipsec
{
            "certificate" : null,
            "encryptionAlgorithm" : "aes",
            "mtu" : 1500,
            "enabled" : true,
            "psk" : "****",
            "peerSubnets" : [
               "192.168.202.0/24",
               "192.168.204.0/24"
            ],
            "peerIp" : "10.117.35.202",
            "name" : null,
            "description" : null,
            "localSubnets" : [
               "192.168.203.0/24"
            ],
            "dhGroup" : "dh2",
            "peerId" : "10.117.35.202",
            "localIp" : "10.117.35.203",
            "authenticationMode" : "psk",
            "enablePfs" : true,
            "localId" : "10.117.35.203"
         } 

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 1 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 1 Ratings
Actions
KB: