Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Advanced troubleshooting of Windows shares (CIFS) with VMware Data Recovery (vDR) (2010785)

Purpose

This article provides guidance on advanced troubleshooting techniques for issues with CIFS/SMB shares when used as dedupe stores on a VMware Data Recovery (vDR) appliance.

For basic CIFS troubleshooting steps, see Troubleshooting CIFS Shares issues for VMware Data Recovery (1037855).

Resolution

Note: Ensure that the vDR appliance has 100% exclusive access to the destination dedupe datastore.

Note: Mechanisms that lock files, such as those employed by backups or virus scanning, will impact the vDR appliance and may cause undesired behavior within the appliance.

Advanced troubleshooting steps


  1. On the vDR appliance, log into the console as root and check the /var/log/messages file for any CIFS-related messages by running the command:

    # grep CIFS /var/log/messages

    Nov 30 09:46:27 localhost kernel:  CIFS VFS: No response for cmd 50 mid 34376
    Nov 30 09:46:27 localhost kernel:  CIFS VFS: No response for cmd 50 mid 34375
    Nov 30 09:46:48 localhost kernel:  CIFS VFS: server not responding
    Nov 30 09:46:48 localhost kernel:  CIFS VFS: No response for cmd 114 mid 34377


  2. Check the state of the mounted CIFS shares by running the command:

    # cat /proc/fs/cifs/DebugData

    Display Internal CIFS Data Structures for Debugging
    ---------------------------------------------------
    CIFS Version 1.60RH
    Active VFS Requests: 0
    Servers:
    1) Name: 10.16.224.189  Domain: WORKGROUP Uses: 1 OS: Windows 7 Ultimate 7601 Service Pack 1
          NOS: Windows 7 Ultimate 6.1     Capability: 0x1e3fc
          SMB session status: 1   TCP status: 1
          Local Users To Server: 1 SecMode: 0x3 Req On Wire: 0
          Shares:
          1) \\10.16.224.189\share Mounts: 1 Type: NTFS DevInfo: 0x20 Attributes: 0xc700ff
    PathComponentMax: 255 Status: 0x1 type: DISK
          MIDs:


  3. Verify that the vDR appliance can successfully connect to the Windows server providing the CIFS share by running the command:

    # smbclient -d 7 -L //SMB_HOST_OR_IP -W WorkGroup -U user%password

    Note: The -d parameter specifies the output verbosity. The value is configurable from 0 (the default) to 7 (most verbose).

    Example usage:

    # smbclient -d 7 -L //10.0.0.1 -W MyWorkGroup -U MyUser%MyPassword

    Note: If you see the session setup failed: SUCCESS - 0 error, you must uninstall all Windows Live Essentials applications from the Windows 7 machine hosting the CIFS share. For more information, see Samba bug 7577.

Debug Troubleshooting


Enabling verbose logging to disk

To aid in troubleshooting, we must ensure that all kernel messages are being logged to the system log.

To enable kernel logging:

  1. Log into a console or SSH session as root on the vDR appliance.
  2. Verify whether all kernel messages are being logged to /var/log/kernel.log by running the command:

    # grep kern /etc/syslog.conf

    If all kernel messages are being logged, output is similar to:

    kern.*                      /var/log/kernel.log

  3. If all kernel messages are not being logged, enable them by running:

    # echo "kern.*             /var/log/kernel.log" >> /etc/syslog.conf

    Note: Verify that the content of the /etc/syslog.conf file is similar to this example by running the command:

    # cat /etc/syslog.conf

    Example content:

    kern.*                                       /var/log/kernel.log
    *.info;mail.none;authpriv.none;cron.none     /var/log/messages
    authpriv.*                                   /var/log/secure
    mail.*                                       -/var/log/maillog
    cron.*                                       /var/log/cron
    *.emerg                                      *
    uucp,news.crit                               /var/log/spooler
    local7.*                                     /var/log/boot.log


  4. To apply the changes, restart the syslog service on the vDR appliance by running the command:

    # service syslog restart

VMware recommends increasing the CIFS logging level while troubleshooting CIFS-related issues. The additional logging information is then sent to /var/log/messages on the vDR appliance.

To enable increased CIFS logging:

  1. Log into a console or SSH session as root on the vDR appliance.
  2. Enable increased CIFS logging by running the command:

    # echo 3 > /proc/fs/cifs/cifsFYI

  3. Verify the value in cifsFYI by running:

    # cat /proc/fs/cifs/cifsFYI
    3


    Note: The value is configurable from 0 (the default) to 7 (most verbose).

Note: The increased CIFS logging setting will persist only until the next vDR appliance reboot. It then reverts to the default value of 0 .


Troubleshooting network connectivity between vDR and the CIFS server


When checking for CIFS issues, it is useful to check the network communication between the vDR appliance and the CIFS server. This can be achieved using a packet capture tool such as tcpdump . The vDR appliance does not include a packet capture tool, so you must install tcpdump to use these steps.

Note: This process requires that the vDR appliance has Internet connectivity. For vDR appliances without Internet connectivity, the tcpdump package (in RPM format) must be manually downloaded, uploaded to the vDR appliance, and installed using the rpm tool.

To install tcpdump on vDR appliances connected to the Internet:

  1. Log into the console of the vDR appliance as the root user.
  2. Install tcpdump on the vDR appliance by running the command:

    # yum install tcpdump

  3. The installer prompts you to confirm installing the package. Answer yes to install tcpdump .
  4. When complete, you see a message similar to:

    Installed:
     tcpdump.x86_64 14:3.9.4-15.el5
    Complete!

To capture CIFS packets (on port 445) between the CIFS server and the vDR appliance to a file, run the command:

# tcpdump -i eth0 -s0 -w /tmp/cifs-traffic.pcap host cifs_server.example.com and port 445

The output is similar to:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
<Ctrl+c>
387995 packets captured
407094 packets received by filter
19012 packets dropped by kernel

 
Note: Press Ctrl+c to stop packet capture.


To analyze the communication data in the capture file between the CIFS server and the vDR appliance, run the command:

# tcpdump -r /tmp/cifs-traffic.pcap | less

The output is similar to:

09:45:28.839695 IP 10.16.224.189.microsoft-ds > 10.16.225.8.53293: P 1:187(186) ack 116 win 260 <nop,nop,timestamp 163093413 61721271>
09:45:28.840064 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: . ack 187 win 827 <nop,nop,timestamp 61721273 163093413>
09:45:28.840866 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: P 116:264(148) ack 187 win 829 <nop,nop,timestamp 61721274 163093413>
09:45:28.841874 IP 10.16.224.189.microsoft-ds > 10.16.225.8.53293: P 187:405(218) ack 264 win 259 <nop,nop,timestamp 163093413 61721274>
09:45:28.842644 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: P 264:412(148) ack 405 win 829 <nop,nop,timestamp 61721276 163093413>
09:45:28.843876 IP 10.16.224.189.microsoft-ds > 10.16.225.8.53293: P 405:623(218) ack 412 win 259 <nop,nop,timestamp 163093413 61721276>
09:45:28.845164 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: P 412:528(116) ack 623 win 829 <nop,nop,timestamp 61721278 163093413>


Note: The capture file can be imported and analyzed in a packet capture tool such as Wireshark.




See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 1 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 1 Ratings
Actions
KB: