Search the VMware Knowledge Base (KB)
View by Article ID

Troubleshooting connectivity issues to an NFS datastore on ESX and ESXi hosts (1003967)

  • 154 Ratings
Language Editions

Symptoms

  • The NFS share cannot be mounted by the ESX/ESXi host.
  • The NFS share is mounted, but nothing can be written to it.
  • You see entries similar to:

    • NFS Error: Unable to connect to NFS server
    • WARNING: NFS: 983: Connect failed for client 0xb613340 sock 184683088: I/O error
    • WARNING: NFS: 898: RPC error 12 (RPC failed) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server (xxx.xxx.xxx.xxx)
    • Network cable is unplugged

Resolution

To resolve this issue, validate that these steps are true for your environment:

Caution: Do not skip a step. The steps provide instructions or a link to a document for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution.

  1. Check the MTU size configuration on the port group which is designated as the NFS VMkernel port group. If it is set to anything other than 1500 or 9000, test the connectivity using the vmkping command:

    # vmkping -I vmkN -s nnnn xxx.xxx.xxx.xxx

    Where:

    • vmkN is vmk0, vmk1, etc, depending on which vmknic is assigned to NFS.

      Note: The -I option to select the vmkernel interface is available only in ESXi 5.1. Without this option in 4.x/5.0, the host uses the vmkernel associated with the destination network being pinged in the host routing table. The host routing table can be viewed by running the esxcfg-route -l command.

    • nnnn is the MTU size minus 28 bytes for overhead. For example, for an MTU size of 9000, use 8972.
    • xxx.xxx.xxx.xxx is the IP address of the target NFS storage.

    To reveal the vmknics, run the command:

    esxcfg-vmknic -l

    Check the output for the vmk_ interface associated with NFS.

  2. Verify connectivity to the NFS server and ensure that it is accessible through the firewalls. For more information, see Cannot connect to NFS network share (1007352).
  3. Run netcat (nc) command to see if you can reach the NFS server nfsd TCP/UDP port (default 2049) on the storage array from the host:

    # nc -z array-IP 2049

    Example output:

    Connection to 10.1.10.100 2049 port [tcp/http] succeeded!

    Note: The netcat command is available with ESX 4.x and ESXi 4.1 and later.

  4. Verify that the ESX host can vmkping the NFS server. For more information, see Testing VMkernel network connectivity with the vmkping command (1003728).
  5. Verify that the NFS host can ping the VMkernel IP of the ESX host.
  6. Verify that the virtual switch being used for storage is configured correctly. For more information, see the Networking Attached Storage section of the ESX Configuration Guide.

    Note: Ensure that there are enough available ports on the virtual switch. For more information, see Network cable of a virtual machine appears unplugged (1004883) and No network connectivity if all ports are in use (1009103).

  7. Verify that the storage array is listed in the Hardware Compatibility Guide. For more information, see the VMware Compatibility Guide. Consult your hardware vendor to ensure that the array is configured properly.

    Note: Some array vendors have a minimum microcode/firmware version that is required to work with ESX.

  8. Verify that the physical hardware functions correctly. Consult your hardware vendor for more details.
  9. If this is a Windows server, verify that it is correctly configured for NFS. For more information, see Troubleshooting the failed process of adding a datastore from a Windows Services NFS device (1004490).

To troubleshoot a mount being read-only:
  1. Verify that the permissions of the NFS server have not been set to read-only for this ESX host.
  2. Verify that the NFS share was not mounted with the read-only box selected.
  3. For troubleshooting NFS by enabling the /NFS/LogNfsStat3 advanced parameters, see Using nfsstat3 to troubleshoot NFS error: Failed to get object: No connection (2010132).

If the above troubleshooting has not resolved the issue and there are still locked files. Attempting to unmount the NAS volume may fail with an error similar to:

WARNING: NFS: 1797: 8564d0cc-58f6-4573-886f-693fa721098c has open files, cannot be unmounted

To troubleshoot the lock:
  1. Identify the ESX/ESXi host holding the lock. For more information, see Investigating virtual machine file locks on ESXi/ESX (10051).
  2. Restart the management agents on the host. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).
  3. If the lock remains, a host reboot is required to break the lock.

    Note: If you wish to investigate the cause of the locking issue further, ensure to capture the host logs before rebooting.

If the issue persists after completing the steps in this article:

Additional Information

Note: ESXi supports only NFS version 3 over TCP/IP. For more information, see the vSphere Networking guide.

If an attempt is made to mount NFS share over UDP, in the /var/log/vmkernel.log file, you see error similar to:

WARNING: NFS: 918: Server (10.x.x.x) does not support Mount Program (100005) Version (3) Protocol (TCP) 

Tags

nfs-connectivity add-nfs-datastore

See Also

Update History

05/11/2010 - Added additional troubleshooting steps. 06/08/2012 - Added error messages.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 154 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 154 Ratings
Actions
KB: