Search the VMware Knowledge Base (KB)
View by Article ID

NSX Edge Load Balancer temporary returns 503 after pool member change (2151231)

  • 0 Ratings

Symptoms

  • Connecting to NSX Edge Loadbalancer fails with HTTP Error 503 after a number of pool member change.
  • NSX Edge Loadbalancer has been configured to use Monitor Service for health monitoring in the past and currently uses BUILT-IN.
  • To identify which health monitor is used, use "show service loadbalancer virtual" in NSX Edge CLI.


    Example for Monitor Service:
    +->POOL MEMBER: poolX/memberX, STATUS: DOWN
    | | HEALTH MONITOR = MONITOR SERVICE, monitorX:CRITICAL
    | | | LAST STATE CHANGE: <DATE> <TIME>
    | | | LAST CHECK: <DATE> <TIME>
    | | | FAILURE DETAIL: PING CRITICAL - Packet loss = 100%
    | | SESSION (cur, cps, total) = (0, 0, 0)
    | | BYTES in = (0), out = (0
)

Example of Built-IN:

     +->POOL MEMBER: poolX/memberX, STATUS: UP
    | | HEALTH MONITOR = BUILT-IN, monitorX:L7OK
    | | | LAST STATE CHANGE: <DATE> <TIME>
    | | SESSION (cur, max, total) = (0, 0, 0)
    | | BYTES in = (0), out = (0)
  • You see entries similar to NSX Edge logs when you cannot access backend servers through Loadbalancer:

         loadbalancer[<PID>]: [LB]: [local0.info] XXX.XXX.XXX.XXX - - [<DATE>:<TIME>] "GET / HTTP/1.1" 503 XXX "" "" XXXXX XXX "X" "X" "<NOSRV>" 0 -1 -1 -1 0 SC-- 0 0 0 0 0 0 0 ""

  • You see entries similar to NSX Edge logs when you configure loadbalancer pool members:
    config: [daemon.warning] WARN :: C_UTILS :: File /var/db/networkmonitor/monitor_retention.dat not exist
    loadbalancer[<PID>]: [LB]: [local0.alert] Server poolXX/member1 is DOWN, changed from CLI. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    loadbalancer[<PID>]: [LB]: [local0.alert] Server poolXX/member2 is DOWN, changed from CLI. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    loadbalancer[<PID>]: [LB]: [local0.alert] Server poolXX/member3 is DOWN, changed from CLI. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    loadbalancer[<PID>]: [LB]: [local0.emerg] backend poolXX has no server available!
    config: [daemon.info] INFO :: CONFIG_MGR :: update ipvs...
    config: [daemon.info] INFO :: C_IPVS :: IPVS: stop connection sync-up daemon
    config: [daemon.info] INFO :: CONFIG_MGR :: update nagios...
    config: [daemon.info] INFO :: C_ServiceControl :: update nagios to down
    config: [daemon.info] INFO :: CONFIG_MGR :: --------------- Collecting the configurator output ---------------
    config: [daemon.info] INFO :: Utils :: saved data to /var/db/vmware/vshield/vse_one/resource_save.psf
    config: [daemon.info] INFO :: Utils :: saved data to /var/db/vmware/vshield/vse_one/config_save.psf
    config: [daemon.info] INFO :: vse_configure :: update success
    config: [daemon.info] INFO :: Utils :: ha: UpdateHaResourceFlags:

Cause

This issue occurs due to miss-loading old health status reported by Monitor Service.

Resolution

This issue occurs due to miss-loading old health status reported by Monitor Service.

This is a known issue affecting NSX for vSphere 6.3.x.

To work around this issue, Configure BUILT-IN with all different id & name pairs for pool / member / monitor from what were used in Monitor Service.

Using Web Client, to create NSX Edge Loadbalancer with new id & name pairs, navigate to Networking & Security > NSX Edges > Manage > Load Balancer.
  • For monitor, navigate to Service Monitoring.
  • For pool and its members, navigate to Pools.
Using REST API, change (poolId, name), (memberId, name), (monitorId, name)

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 0 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 0 Ratings
Actions
KB: