Controller deployment fails after upgrading from NSX-v 6.4.0/6.4.1 to 6.4.2/6.4.3
search cancel

Controller deployment fails after upgrading from NSX-v 6.4.0/6.4.1 to 6.4.2/6.4.3

book

Article ID: 321112

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • Controller deployment fails.
  • The Controller VM that has been deployed is deleted.
  • NSX User Interface (UI) displays an error similar to:

    Error on controller <controller-id>. Controller error or invalid ntp servers. Please check if NTP server(s) are reachable from <controller-id> and try again. Refer to API documentation for more details.
     
  • The vsm.log file contains NTP errors similar to:

    2018-09-10 20:47:10.220 GMT ERROR taskScheduler-26 ControllerServiceUtils:890 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] API call to Controller controller-4 failed, Error: 500 Internal Server Error
    2018-09-10 20:47:10.220 GMT ERROR taskScheduler-26 ControllerPoweronAdvisor:325 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Failed to update NTP Servers on controller controller-4
    2018-09-10 20:47:10.220 GMT ERROR taskScheduler-26 ControllerPoweronAdvisor:329 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Controller controller-4 node error statusCode = 500, errorPayload = {"message": "The server has either erred or is incapable of performing the requested operation.<br /><br />\nError occurred while trying to process the request\n\n", "code": "500 Internal Server Error", "title": "Internal Server Error"}
    2018-09-10 20:47:10.220 GMT WARN taskScheduler-26 ControllerPoweronAdvisor:382 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Failed to update NTP Servers on controller
    2018-09-10 20:47:10.223 GMT INFO taskScheduler-26 RelationshipManagerImpl:840 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Recursively removing domain object controller-4 updateParent false
    2018-09-10 20:47:10.265 GMT INFO taskScheduler-26 AuditingServiceImpl:174 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] [AuditLog] UserName:'nsx.local\administrator', ModuleName:'VdnNvpController', Operation:'DELETE', Resource:'controller-1', Time:'Mon Sep 10 20:47:10.228 GMT 2018', Status:'SUCCESS', Universal Object:'false'
    2018-09-10 20:47:10.266 GMT INFO taskScheduler-26 ControllerPoweronAdvisor:385 - - [nsxv@6876 comp="nsx-manager" subcomp="manager"] Removed controller controller-4 from DB

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware NSX for vSphere 6.4.x

Cause

This issue occurs due to the upgrade workflow copying invalid NTP data into the cluster level scope table.

Resolution

This is a known issue affecting upgrades of VMware NSX for vSphere 6.4.0 and 6.4.1 to VMware NSX Data Center for vSphere 6.4.2 and 6.4.3.

Currently, there is no resolution.

Workaround:
To work around this issue, follow either one of these three options:
  1. Clear the incorrect NTP Server cluster setting from the User Interface (UI).

    a. Navigate to the vCenter Server > Manager Plugin > Controllers page > Common Controller Attributes. 
    b. Click Edit. 
    c. Specify empty entry for NTP Servers.
     
  2. Make the following API calls to clear out the invalid data in the NTP:

    For example:

    PUT https://<NSXMGR_IP>/api/2.0/vdn/controller/cluster/ntp

    Body:
    {
        "ntpServers": []
    }


    PUT https://<NSXMGR_IP>/api/2.0/vdn/controller/cluster/syslog

    Body:
    {
        "syslogServers": []
    }
     
  3. Alternatively, create a legitimate NTP Server entry using the User Interface (UI) or API as per above.


Powered by Wolken Software