Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

VMware VirtualCenter Server service fails after upgrading to vCenter Server 4.1 in an environment configured with Nexus 1000v (1026009)

Symptoms

  • VMware VirtualCenter Server service fails if:

    • Cisco Nexus 1000v Virtual Supervisor Module (VSM) version 4.0(4)SV1(1) or earlier is used to create a vNetwork Distributed Switch (vDS) in vCenter Server 4.0.x.
    • vCenter Server is upgraded from version 4.0.x to 4.1.
    • Optionally, Cisco Nexus 1000v VSM is upgraded to a version newer than 4.0(4)SV1(1).

  • When ESX 4.0.x and/or ESX 4.1 hosts are added to the existing vDS in the vCenter Server that was upgraded to version 4.1, the VirtualCenter Server service crashes.
  • In the vpxd.log file, you see entries similar to:
[2010-08-02 09:50:32.921 02788 info 'App' opID=32AEE5DF-0000051B] [VpxLRO] -- BEGIN task-27851 -- dvs-329 -- vim.DistributedVirtualSwitch.reconfigure -- B0AC3929-F890-4C3B-A1D4-5FAFC783E8DC(64582860-3236-4E67-A4B2-C00CB200406B)
[2010-08-02 09:51:23.827 02788 info 'App' opID=32AEE5DF-0000051B] CoreDump: Writing minidump
[2010-08-02 09:51:46.655 02788 panic 'App' opID=32AEE5DF-0000051B]
Panic: Win32 exception: Access Violation (0xc0000005)
Read (0) at address 0000000000000008
rip: 0000000140b6d0e4 rsp: 000000001131d780 rbp: 0000000000000064
rax: 0000000000000001 rbx: 00000000055ac810 rcx: 00000000000058b0
rdx: 0000000000000000 rdi: 0000000000000000 rsi: 0000000000000000
r8: 0000000007552c00 r9: 000000000754ff00 r10: 0000000100000001
r11: 000000001131d770 r12: 000000001131db18 r13: 00000000064730d0
r14: 0000000000000000 r15: 000000000b70ebc8

Backtrace:
backtrace[00] rip 00000001800e8018 Vmacore::System::SystemFactoryImpl::CreateFileWriter
backtrace[01] rip 00000001800e850e Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace
backtrace[02] rip 0000000180129205 Vmacore::PanicExit
backtrace[03] rip 0000000180129377 Vmacore::PanicExit
backtrace[04] rip 000000018011b746 Vmacore::System::IsEnlisted
backtrace[05] rip 0000000078664e25 _CxxFrameHandler3
backtrace[06] rip 0000000078663fd2 _CxxExceptionFilter
backtrace[07] rip 00000000786646da _CxxExceptionFilter
backtrace[08] rip 0000000078664919 _CxxExceptionFilter
backtrace[09] rip 0000000078664d27 _CxxFrameHandler3
backtrace[10] rip 0000000077ee451d RtlSizeHeap
backtrace[11] rip 0000000077ee6027 RtlLookupFunctionEntry
backtrace[12] rip 0000000077ef31dd KiUserExceptionDispatcher
backtrace[13] rip 0000000140b6d0e4 (no symbol)
backtrace[14] rip 0000000140c21e8b (no symbol)
backtrace[15] rip 0000000140bcb8ec (no symbol) 
backtrace[16] rip 0000000140bce64d (no symbol)
backtrace[17] rip 0000000140bce6f6 (no symbol)
backtrace[18] rip 00000000006cc926 Vim::DistributedVirtualSwitch::OverlayClassConfigInfo::_GetDataType
backtrace[19] rip 00000000003688e0 Vmomi::RuntimeFault::_GetDataType
backtrace[20] rip 0000000140a6d00f (no symbol)
backtrace[21] rip 0000000140a4a772 (no symbol)
backtrace[22] rip 0000000140a50b08 (no symbol)
backtrace[23] rip 000000018011f23d Vmacore::System::IsEnlisted
backtrace[24] rip 000000018012061f Vmacore::System::IsEnlisted
backtrace[25] rip 0000000180119915 Vmacore::System::GetThreadId
backtrace[26] rip 0000000078622fdf endthreadex
backtrace[27] rip 0000000078623080 endthreadex
backtrace[28] rip 0000000077d6b71a BaseThreadStart
backtrace[29] rip 0000000000000000 (no symbol)
[2010-08-02 09:51:46.655 02788 panic 'App' opID=32AEE5DF-0000051B] Win32 exception: Access Violation (0xc0000005)
Read (0) at address 0000000000000008
rip: 0000000140b6d0e4 rsp: 000000001131d780 rbp: 0000000000000064
rax: 0000000000000001 rbx: 00000000055ac810 rcx: 00000000000058b0
rdx: 0000000000000000 rdi: 0000000000000000 rsi: 0000000000000000
r8: 0000000007552c00 r9: 000000000754ff00 r10: 0000000100000001
r11: 000000001131d770 r12: 000000001131db18 r13: 00000000064730d0
r14: 0000000000000000 r15: 000000000b70ebc8

Resolution

This issue occurs if an unexpected vDS version that cannot be handled by the vCenter Server exists in the vCenter Server database.

This issue is resolved by vCenter 4.1 Update 1, released on February 10, 2011. For download and installation information, see the release notes. The update can be downloaded from the VMware Download Center.

For vCenter Server 4.1, to work around this issue, you must check the vCenter Server database for such a vDS version and manually update its value:

  1. Run this SQL query on the vCenter Server 4.1 database:

    Note: You need Microsoft SQL Server Management Studio to run the query.

    SELECT PRODUCT_NAME, PRODUCT_VENDOR, PRODUCT_VERSION
    FROM VPX_DVS
    WHERE PRODUCT_VERSION = 'Cisco Nexus 1000V'
    ;

  2. If this query returns a row(s), perform these steps to manually update the PRODUCT_VERSION value for the vDS:

    1. Stop the VMware VirtualCenter Server service. The VMware Management Webservices also stops.
    2. Create a backup of the vCenter Server database.
    3. Run the SQL script in the Attachments section of this article on the vCenter Server 4.1 database. The vDS version information is updated.
    4. Run the SQL query in Step 1 to ensure that the vDS version in the VPX_DVS table is updated.
    5. Start the VMware VirtualCenter Server service and the VMware Management Webservices service.

Hosts can now be added to the existing Nexus vDS in the upgraded vCenter Server 4.1.

Additional Information

The error Win32 exception: Access Violation can occur for many issues. For other issues that experience this error, see:

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

See Also

Update History

09/07/2010 - Added links to 1016837 and 1014778 06/28/2011 - Added resolution information, clarified workaround

Attachments

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 7 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 7 Ratings
Actions
KB: