Knowledge Base

Problem Description

Symptoms and Elements of the Problem

1. The system stops responding for a period of two to five minutes or there is a total lockup of the service console.
   
   
2. During this period, the service console and virtual machines are not responsive.
   
   
3. The ESX Server host usually recovers at the end of this time period, but some of the systems may not recover.
   
   
4. Systems from multiple hardware vendors, with both Intel and AMD processors can be affected.
   
   
5. Affected systems have been found to have the following criteria:
   
   
   
   1. USB modules loaded in the service console
      
      
   2. At least one local VMFS volume
      
      
   3. At least one SAN VMFS volume
      
      
   4. Rescan of all devices and all VMFS volumes at the same time
      
      
   5. There are two or more devices configured that have a shared IRQ assignment. To determine this, perform the steps in the following section, "How to Tell if You Have Shared IRQ Assignments."

How to Tell if You Have Shared IRQ Assignments

To determine whether you may be affected by this problem, start by listing the interrupt line usage. In the service console, type:

cat /proc/vmware/interrupts

Vector    PCPU  0    PCPU  1
  0x21:         163          0 COS irq 1 (ISA edge)
  0x29:           0          0 
  0x31:           1          0 VMK serial
  0x39:           0          0 
  0x41:           0          0 
  0x49:           0          0 
  0x51:           0          0 
  0x59:           1          0 COS irq 12 (ISA edge)
  0x61:        1885          0 COS irq 14 (ISA edge)
  0x69:           1          0 COS irq 15 (ISA edge)
  0x71:          30          0 COS irq 19 (PCI level), VMK aic7xxx
  0x79:           1      52596 <COS irq 17 (PCI level)>, VMK vmnic0
  0x81:       66860          0 COS irq 16 (PCI level)
  0x89:          42         46 <COS irq 18 (PCI level)>, VMK aic7xxx
  0xdf:     3588590    3589262 VMK timer
  0xe1:           0          0 VMK ipi
  0xe9:           4          1 VMK resched
  0xf1:           3          0 VMK tlb
  0xf9:        2871          0 VMK noop
  0xfc:           0          0 VMK thermal
  0xfd:           0          0 VMK lint1
  0xfe:           0          0 VMK error
  0xff:           0          0 VMK spurious   

The example above contains a line at vector 0x71 with both VMK and COS devices, another line at vector 0x79 with both VMK and COS (in angle brackets), and a third line at vector 0x89 with both VMK and COS (in angle brackets). You can ignore the latter two lines with the COS devices in angle brackets, and focus on the line at vector 0x71.

Next, list the PCI device assignments (VMkernel or service console) for the shared interrupt lines.

cat /proc/vmware/pci

Use the interrupt vectors of the shared interrupt lines to index into the PCI devices output, and determine the affected controllers. The interrupt vector is found in the ISA/irq/Vec column of the output.

A mode value of C indicates the device is dedicated to the service console. V indicates the device is dedicated to the VMkernel.

You are affected by the problem if you identify a group of controllers that share the same Vec number and both of these are true:

• One controller has mode C (managed by the service console).
• Another controller has mode V (managed by the VMkernel).

• All controllers in the group have mode C (assigned to the service console).
• All controllers in the group have mode V (assigned to the VMkernel).

To continue the example, the only interrupt vector of concern is 0x71, which is found in two rows of this output. The affected controllers are the Intel USB and the Adaptec SCSI controller, which share interrupt 19 and vector 0x71.

USB has mode C (meaning it is assigned to the service console) while vmhba1 has mode V (meaning it is assigned to the VMkernel). In this example, the controllers that share the interrupt line are managed by different entities. This is likely a system that will encounter the rescan problem.

Root Cause

The rescan system not responding is due to a dead lock between two requests coming from the service console. These requests are serviced by VMkernel on behalf of the service console.