Troubleshooting NFS datastore connectivity issues

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

The NFS share cannot be mounted by the ESXi host.
The NFS share is mounted, but nothing can be written to it.
The NFS datastore is inaccessible.
You see entries similar to:

NFS Error: Unable to connect to NFS server

WARNING: NFS: 983: Connect failed for client 0xb613340 sock 184683088: I/O error

WARNING: NFS: 898: RPC error 12 (RPC failed) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server (xxx.xxx.xxx.xxx)

Network cable is unplugged

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 8.0.x

Resolution

To resolve this issue, validate that these steps are true for your VMware environment:

Caution: Do not skip a step. The steps provide instructions or a link to a document for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution.

Check the MTU size configuration on the port group which is designated as the NFS VMkernel port group. If it is set to anything other than 1500 or 9000, test the connectivity using the vmkping command:

# vmkping -I vmkN -s nnnn xxx.xxx.xxx.xxx

Where:
- vmkN is vmk0, vmk1, etc, depending on which vmknic is assigned to NFS.
  nnnn is the MTU size minus 28 bytes for overhead. For example, for an MTU size of 9000, use 8972.
- xxx.xxx.xxx.xxx is the IP address of the target NFS storage.
To reveal the vmknics, run the command:

esxcfg-vmknic -l

Check the output for the vmk_ interface associated with NFS.
Verify connectivity to the NFS server and ensure that it is accessible through the firewalls. For more information, see Cannot connect to NFS network share (1007352).
Run the netcat (nc) command to see if you can reach the NFS server nfsd TCP/UDP port (default 2049) on the storage array from the host:

# nc -z array-IP 2049

Example output:

Connection to 10.1.10.100 2049 port [tcp/http] succeeded!
Verify that the ESXi host can vmkping the NFS server. For more information, see Testing VMkernel network connectivity with the vmkping command (1003728).
Verify that the NFS host can ping the VMkernel IP of the ESXi host.
Verify that the virtual switch being used for storage is configured correctly.

Note: Ensure that there are enough available ports on the virtual switch.
Verify that the storage array is listed in the VMware Hardware Compatibility Guide. For more information, see the VMware Compatibility Guide. Consult your hardware vendor to ensure that the array is configured properly.

Note: Some array vendors have a minimum microcode/firmware version that is required to work with the ESXi host.
Verify that the physical hardware functions correctly. Consult your hardware vendor for more details.
If this is a Windows server, verify that it is correctly configured for NFS. For more information, see Troubleshooting the failed process of adding a datastore from a Windows Services NFS device (1004490).

To troubleshoot a mount being read-only:

Verify that the permissions of the NFS server have not been set to read-only for this ESXi host.
Verify that the NFS share was not mounted with the read-only box selected.

In addition please see: Remounting a disconnected NFS datastore from the ESXi command line (1005057)

If the above troubleshooting has not resolved the issue and there are still locked files, e.g. attempting to unmount the NAS volume may fail with an error similar to:

WARNING: NFS: 1797: 8564d0cc-58f6-4573-886f-693fa721098c has open files, cannot be unmounted

To troubleshoot the lock:

Identify the ESXi host holding the lock. For more information, see Investigating virtual machine file locks on ESXi host (10051).
Restart the management agents on the host. For more information, see Restarting the Management agents in ESXi (1003490).
If the lock remains, a host reboot is required to break the lock.

Note: If you wish to investigate the cause of the locking issue further, ensure to capture the host logs before rebooting.

Additional Information

If an attempt is made to mount NFS share over UDP, in the /var/log/vmkernel.log file, you see an error similar to:

WARNING: NFS: 918: Server (10.x.x.x) does not support Mount Program (100005) Version (3) Protocol (TCP)