Search the VMware Knowledge Base (KB)
View by Article ID

Virtual machine Power on requests from vRealize Operations manager fail (2149064)

  • 0 Ratings

Symptoms

Note: The log entries referenced below were found when logging was set to debug or verbose. For information on changing the logging level see the Additional Information section.
  • Powering on a virtual machine from vRealize Operation Manager fails.
  • In the workflow-manager.log file, there are entries similar to:

    2016-12-29T10:23:51.451-05:00 [pool-9-thread-2 DEBUG opID=] [WorkflowBodyHandlerImpl] -- Creating proxy object for moId vm-11688
    2016-12-29T10:23:51.454-05:00 [pool-5-thread-2 DEBUG opID=] [TaskInvoker] -- Issuing PowerOnVM_Task on ManagedObjectReference: type = VirtualMachine, value = vm-11688, serverGuid = null
    2016-12-29T10:23:51.454-05:00 [pool-5-thread-2 DEBUG opID=1354c09e-AC] [WorkflowImpl] -- Updating pre-specified key parentTaskMoId
    2016-12-29T10:23:51.454-05:00 [pool-5-thread-2 INFO opID=1354c09e-AC] [WorkflowImpl] -- BEGIN executing WSDL method PowerOnVM_Task on vm-11688


    2016-12-29T10:23:51.581-05:00 [workerThreadpool-14 DEBUG opID=1354c09e-AC] [WorkflowImpl] -- Processing results for step VmLockingHandler-QueryLockRequirements in PowerOnVM_Task
    2016-12-29T10:23:51.581-05:00 [workerThreadpool-14 DEBUG opID=1354c09e-AC] [WorkflowImpl] -- Finished step: VmLockingHandler-QueryLockRequirements
    2016-12-29T10:23:51.581-05:00 [workerThreadpool-14 DEBUG opID=1354c09e-AC] [WorkflowImpl] -- Running step: VmLockingHandler-AcquireLock
    2016-12-29T10:23:51.581-05:00 [workerThreadpool-14 DEBUG opID=1354c09e-AC] [LockingStepImpl] -- Making lock acquire request for requestId 3812e0af-be4d-46fe-b762-4605b266f800
    2016-12-29T10:23:51.581-05:00 [workerThreadpool-14 DEBUG opID=1354c09e-AC] [SchedulerImpl] -- Acquiring locks for 3812e0af-be4d-46fe-b762-4605b266f800
    2016-12-29T10:23:51.585-05:00 [VlsiClientPool-14 DEBUG opID=1354c09e-AC] [wire] -- >> "<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Header><stepContribution xsi:type="string">5</stepContribution><parentTask xsi:type="string">task-399060</parentTask><isWorkflowRedirect xsi:type="string">true</isWorkflowRedirect><workflow_user_session xsi:type="string">699b4b089182814823b1144368e00f9d57dc3af7</workflow_user_session><operationID xsi:type="string">1354c09e-AC</operationID></soapenv:Header><soapenv:Body><AcquireLockSLM xmlns="urn:internalvim25" xmlns:internalvim25="urn:internalvim25"><_this type="StateLockManager">StatelockManager</_this><requestId>3812e0af-be4d-46fe-b762-4605b266f800</requestId><spec><object type="VirtualMachine" serverGuid="2B579EE3-E0D5-41B5-A4C5-C60C8DEA3C49">vm-11688</object><lockType>exclusive</lockType></spec></AcquireLockSLM></soapenv:Body></soapenv:Envelope>"


    2016-12-29T10:23:51.589-05:00 [VlsiClientPool-14 DEBUG opID=1354c09e-AC] [SchedulerImpl] -- Lockacquire returned for 3812e0af-be4d-46fe-b762-4605b266f800
    2016-12-29T10:23:51.589-05:00 [VlsiClientPool-14 ERROR opID=1354c09e-AC] [SchedulerImpl] -- Locks were not granted for 3812e0af-be4d-46fe-b762-4605b266f800. Will retry later

    The workflow-manager.log file is located at:
    • Windows installed vCenter Server - C:\ProgramData\VMware\vCenterServer\logs\workflow\
    • vCenter Server Appliance - /var/log/vmware/workflow/

  • After the above entries in the workflow-manager.log file there are no further retry events listed.
  • In the vpxd.log file, there are entries similar to:

    2016-12-29T10:23:51.586-05:00 verbose vpxd[29952] [Originator@6876 sub=Default opID=1354c09e-AC-7f] [VpxVmomi] Invoking [acquire] on [vim.modularity.StateLockManager:StatelockManager] session [5260c05e-0985-4202-e8df-b4d28acc59e9(52e9d502-b259-6dbd-217f-18948af257be)]
    2016-12-29T10:23:51.586-05:00 info vpxd[07500] [Originator@6876 sub=vpxLro opID=1354c09e-AC-7f] [VpxLRO] -- BEGIN task-internal-23869691 -- StatelockManager -- vim.modularity.StateLockManager.acquire -- 5260c05e-0985-4202-e8df-b4d28acc59e9(52e9d502-b259-6dbd-217f-18948af257be)
    2016-12-29T10:23:51.586-05:00 verbose vpxd[07500] [Originator@6876 sub=vpxdStateLockManager opID=1354c09e-AC-7f] Acquire: 3812e0af-be4d-46fe-b762-4605b266f800 vim.VirtualMachine:vm-11688
    2016-12-29T10:23:51.586-05:00 verbose vpxd[07500] [Originator@6876 sub=vpxdStateLockManager opID=1354c09e-AC-7f] Failed to acquire lock for requestID: 3812e0af-be4d-46fe-b762-4605b266f800, moID vm-11688. Registering callback

    And

    2016-12-29T10:23:53.722-05:00 error vpxd[10220] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:000000002bac7660, TCP:localhost:8088>; cnx: (null), error: class Vmacore::SystemException(No connection could be made because the target machine actively refused it)
    2016-12-29T10:23:53.738-05:00 warning vpxd[31304] [Originator@6876 sub=vpxdStateLockManager opID=2cf5c62f] Ignoring exception while invoking LocksReleased: No connection could be made because the target machine actively refused it

    For more information on the logging locations in vCenter Server, see  Location of VMware vCenter Server 6.0 log files (2110014).

Purpose

This article resolve port miss-configuration when using a custom port for the VMware vCenter Workflow Manager.

Cause

This issue occurs when the VMware vCenter Workflow Manager service is configured to use custom ports (Default ports used by the service are 8088 and 19999).

Resolution

To resolve this issue, update the vpxd.cfg file on vCenter Server to use custom ports when communicating with the VMware Workflow Manager:
  1. Open the vpxd.cfg file in a plain text editor.
    • Windows installed vCenter Server - C:\ProgramData\VMware\VMware VirtualCenter\
    • vCenter Server Appliance - /etc/vmware-vpx/

  2. Add these entries to the vpxd.cfg file:

    <vpxd>
    ....
    ....
    <stateLockManager>
    <callbackUrl>localhost:Custom_Port</callbackUrl>
    </stateLockManager>
    ....
    ....
    </vpxd>


  3. Restart the vCenter Server Service. For more information, see How to stop, start, or restart vCenter Server 6.x services (2109881).

Additional Information


To increase logging on the Workflow Manager:
  1. Open the workflow.log4j.properties file in a plain text editor.
    • Windows installed vCenter Server - C:\Program Files\VMware\vCenter Server\workflow\conf\
    • vCenter Server Appliance - /etc/vmware/vmware-vpx-workflow/conf/

  2. Edit these lines setting them to Debug:

    log4j.rootCategory
    log4j.appender.console.Threshold
    log4j.logger.com.vmware.vim.workflow
    log4j.logger.com.vmware.vim.workflow.startup


    For example:

    log4j.rootCategory=DEBUG,Rolling
    log4j.appender.console.Threshold=DEBUG
    log4j.logger.com.vmware.vim.workflow=DEBUG
    log4j.logger.com.vmware.vim.workflow.startup=DEBUG,Startup

To increase logging for the VMware VirtualCenter Server service, see Increasing VMware vCenter Server and VMware ESX/ESXi logging levels (1004795).

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 0 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 0 Ratings
Actions
KB: