Disabling the Reverse Path Forwarding using the REST API resolves this issue.

Symptoms:
In an asymmetric routing environment using vCloud Networking and Security (vCNS) 5.x or NSX for vSphere 6.x, vShield/NSX Edge drops packet causing the network communication to fail.

VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.0.x
VMware vCloud Networking and Security 5.1.x
VMware NSX for vSphere 6.3.x
VMware vCloud Networking and Security 5.5.x
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.2.x

In NSX/vCNS Edge by default Reverse Path Forwarding (RPF) is enabled.

When RPF is enabled, the Edge only forward packets if they are received on the same interface that would be used to forward the traffic to the source of the packet. If the route to the source address of the packet is through a different interface than the one it is received on, the packet is dropped.

To resolve this issue, disable Reverse Path Forwarding using REST API.

Note: RPF can be a desirable security feature filtering traffic that should not originate from certain networks, this should be taken into consideration before disabling it.

For example, the <edge-ID> of the Edge in question is found on the Web User Interface (UI). The number of vNic interfaces to disable it depends on how many interfaces the Edge has. RPF can be re-enabled by running the same API again and replacing =0 with =1.

VMware vCloud Networking and Security 5.x:

PUT https://vShield_manager_IP/api/3.0/edges/<edge-ID>/systemcontrol/config

<systemControl>
<property>sysctl.net.ipv4.conf.all.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_0.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_1.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_2.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_3.rp_filter=0</property>
</systemControl>

HTTP Result Code: 204 NO CONTENT

VMware NSX for vSphere 6.x:

PUT https://NSX_mgr_IP/api/4.0/edges/<edge-ID>/systemcontrol/config

<systemControl>
<property>sysctl.net.ipv4.conf.all.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_0.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_1.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_2.rp_filter=0</property>
<property>sysctl.net.ipv4.conf.vNic_3.rp_filter=0</property>
</systemControl>

NSX 6.x 和 vCNS 5.x Edge 中的反向路径转发