Search the VMware Knowledge Base (KB)
View by Article ID

Troubleshooting NSX for vSphere Edge Load Balancer (2122708)

  • 4 Ratings

Symptoms

When using an NSX for vSphere Edge Load Balancer (LB), you experience one or more of these symptoms:
  • Load balancing on TCP port 443 does not work
  • A member of the load balancing pool is not utilized
  • Edge traffic is not load balanced
  • The L7 Load Balancer engine is stopped
  • Health monitor engine is stopped
  • Load Balancer pool member monitor status is WARNING/CRITICAL
  • Load Balancer pool member is in INACTIVE status
  • The L7 sticky table is not synced up to the standby Edge

Purpose

This article provides general and troubleshooting information on NSX for vSphere Edge Load Balancer and identifying common configuration problems.

Edge traffic not balanced

The NSX Edge Load Balancer (LB) enables network traffic to follow multiple paths to a specific destination. It distributes incoming service requests evenly among multiple servers in such a way that the load distribution is transparent to users. Load balancing thus helps in achieving optimal resource utilization, maximizing throughput, minimizing response time, and avoiding overload. NSX Edge provides load balancing up to Layer 7.

You map an external, or public, IP address to a set of internal servers for load balancing. The load balancer accepts TCP, HTTP, or HTTPS requests on the external IP address and decides which internal server to use. Port 8090 is the default listening port for TCP, port 80 is the default port for HTTP, and port 443 is the default port for HTTPs. You must have a working NSX Edge instance before you can configure the NSX load balancer.

Tools used in this troubleshooting article:

REST client: Restful API client uses include fetching Load Balancer configurations, gathering statistics, modifying NSX Manager variables and logging levels
For instructions on using the NSX REST API, see the Using the NSX REST API section of the NSX API Guide.

Note: The preceding links were correct as of September 10, 2015. If you find a link is broken, provide feedback and a VMware employee will update the link.

Resolution

Issue classification

Validate and identify the type of issue you are experiencing.
  • Configuration issue
    When the LB configuration operation is rejected by the NSX User Interface or Rest API call, this is classified as a configuration issue.
  • Data plane issue
    The LB configuration is accepted by NSX manager but there are connectivity or performance issues among the client-edgeLB-server, this is classified as a data plane issue. Data plane issues also include: LB runtime CLI issues, LB SystemEvent issues.

Basic Troubleshooting

Important: Validate that each of the following troubleshooting step is true for your environment. Each step provides instructions or a link to a document, in order to eliminate possible causes and take corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Do not skip any of the steps.

  1. Check the LB configuration status in the vSphere Web Client:

    1. Click Networking & Security > NSX Edges.
    2. Double-click an NSX Edge.
    3. Click Manage.
    4. Click the Load Balancer tab. Check the Load Balancer Status and logging level configured.

      For more information about the available options, see the Set Up Load Balancing section of the NSX Administration Guide.

  2. Before troubleshooting the Load Balancer service, run this command on the Edge to ensure that the service is up an running:

    show service loadbalancer

    See the Additional Information section for output examples.

    Note : If feasible in your environment, you may find it easier to review and capture the output of these commands using SSH. For more information on enabling SSH on the Edge appliance, see the Logging In and Out of the CLI  section of the NSX Command Line Interface Reference guide .

    List of  show service loadbalancer commands

    These commands provide more information about the state of Load Balancer components and services.

    • Show the latest error information:
      show service loadbalancer error

    • Show network health monitor information:
      show service loadbalancer monitor

    • Show pool information:
      show service loadbalancer pool

    • Show current session information:
      show service loadbalancer session

    • Show Sticky-Table information:
      show service loadbalancer table

    • Show virtual server information:
      show service loadbalancer virtual

  3. Check the Release Notes for current releases to see a list of resolved issues. For NSX for vSphere Release Notes related documentation, see the VMware NSX for vSphere Documentation. For more specific information, see the Logical Load Balancer section of the VMware NSX Administration Guide.

Troubleshooting configuration issues

  1. Prior to the LB configuration, change the edge logging level in NSX Manager from  INFO to  TRACE using this REST API call:

    URL: https://NSX_Manager_IP/api/1.0/services/debug/loglevel/com.vmware.vshield.edge?level=TRACE
    Method: POST

  2. Configure the LB in the vSphere Client or using the Restful API. For more information, see the Set Up Load Balancing section of the NSX Administration Guide.
  3. Copy or take a screenshot of any errors that occur during LB configuration.
  4. After the LB configuration, change the edge logging level in NSX Manager, back to  INFO using this REST API call:

    URL: https://NSX_Manager_IP/api/1.0/services/debug/loglevel/com.vmware.vshield.edge?level=INFO
    Method: POST

  5. Record the LB configuration with a screenshot of the LB configuration properties in the vSphere Web Client.

    Note: If you configure the LB with Restful APIs, you must capture the request payload (XML content) also.

  6. Download the NSX Manager log file from your Syslog server and analyze any returned errors.

    For more information about viewing NSX logs, see the About NSX Logs section in the NSX Administration Guide.

Troubleshooting data plane issues


  1. Change the Edge logging level in NSX Manager from  INFO to  TRACE using this REST API call:

    URL: https://NSX_Manager_IP/api/1.0/services/debug/loglevel/com.vmware.vshield.edge?level= TRACE 
    Method:  POST

    Note: You can also change the logging level to  DEBUG if required.

  2. Check the pool member status in the vSphere Web Client:

    1. Click Networking & Security > NSX Edges.
    2. Double-click an NSX Edge.
    3. Click Manage.
    4. Click the Load Balancer tab.
    5. Click Pools to see a summary of the configured LB pools.
    6. Select your LB pool, click Show Pool Statistics and verify that the pool state is UP.

  3. You can get more detailed LB pool configuration statistics from NSX Manager using this REST API call:

    URL: https://NSX_Manager_IP/api/4.0/edges/{edgeId}/loadbalancer/statistics
    Method: GET

    Note: An output example is available in the Additional Information section.

  4. To check LB statistics from the command line, run these commands on the NSX Edge:

    To show LB statistics for a particular pool, run this command:

    show statistics loadbalancer pool

    To show LB statistics for a particular virtual machine, run this command:
    show statistics loadbalancer virtual

    To show statistics for a particular TCP pool, run this command:

    show statistics loadbalancer pool tcp_pool

  5. Review the LB statistics for signs of failure.

VMware NSX for vSphere 6.2.4 Enhancements

The show service loadbalancer command is  improved to show all active sessions that are used globally per NSX Edge.

The L4 load balancer connection statistics (active/inactive/total/max connection number) is also added when running the show service loadbalancer command.

For example:

show service loadbalancer
        
Loadbalancer Services Status:

L7 Loadbalancer : running
L7 Loadbalancer Status Information:

STATUS  PID  MAX_MEM_MB MAX_SOCK MAX_CONN MAX_PIPE CUR_CONN CONN_RATE CONN_RATE_LIMIT MAX_CONN_RATE
running 2604 0          2079     1024     0        0        50        43              0

L4 Loadbalancer Statistics:
MAX_CONN ACT_CONN INACT_CONN TOTAL_CONN
65536    245      537        782

Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP  10.117.6.30:80   wrr
->  192.168.10.2:80   Masq    1      24         171
->  192.168.10.3:80   Masq    1      25         169
TCP 10.117.6.30:443   wrr
->  192.168.10.2:80   Masq    1      196        1
->  192.168.10.3:80   Masq    1      0          196

                                                                                                                                Additional Information

                                                                                                                                Load Balancer service status example

                                                                                                                                The output of the Load Balancer service status command:

                                                                                                                                show service loadbalancer 
                                                                                                                                -----------------------------------------------------------------------
                                                                                                                                Loadbalancer Services Status: 

                                                                                                                                L7 Loadbalancer     : running             
                                                                                                                                Health Monitor      : running             
                                                                                                                                -----------------------------------------------------------------------
                                                                                                                                L7 Loadbalancer Status Information: 
                                                                                                                                STATUS     PID        MAX_MEM_MB MAX_SOCK   MAX_CONN   MAX_PIPE   CUR_CONN   CONN_RATE  CONN_RATE_LIMIT MAX_CONN_RATE  
                                                                                                                                running    6718       0          2091       1024       0          0          0          0               0              
                                                                                                                                -----------------------------------------------------------------------
                                                                                                                                L4 Loadbalancer Statistics: 
                                                                                                                                Prot LocalAddress:Port Scheduler Flags
                                                                                                                                  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
                                                                                                                                TCP  10.117.35.186:1234 wrr
                                                                                                                                  -> 192.168.1.40:1234            Masq    1      0          0         
                                                                                                                                  -> 192.168.1.50:1234            Masq    0      0          0         
                                                                                                                                  -> 192.168.1.100:1234           Masq    1      0          0 


                                                                                                                                Load Balancer health monitor statistics example


                                                                                                                                The output of the health monitor statistics command:

                                                                                                                                show service loadbalancer monitor 
                                                                                                                                -----------------------------------------------------------------------
                                                                                                                                Loadbalancer HealthMonitor Statistics: 

                                                                                                                                POOL                           MEMBER                         HEALTH STATUS                 
                                                                                                                                tcp-pool                       m3                             default_tcp_monitor:OK        
                                                                                                                                tcp-pool                       m1                             default_tcp_monitor:OK        
                                                                                                                                tcp-pool                       m2                             default_tcp_monitor:OK        
                                                                                                                                http-pool                      m1                             default_http_monitor:OK       
                                                                                                                                http-pool                      m2                             default_http_monitor:OK


                                                                                                                                Load Balancer Virtual Machine information example


                                                                                                                                The output of the load balanced virtual server information command:

                                                                                                                                show service loadbalancer virtual http-vip
                                                                                                                                -----------------------------------------------------------------------
                                                                                                                                Loadbalancer VirtualServer Statistics: 

                                                                                                                                VIRTUAL http-vip
                                                                                                                                |  ADDRESS [10.117.35.186]:80  
                                                                                                                                |  SESSION (cur, max, limit, total) = (0, 0, 1024, 0)
                                                                                                                                |  RATE (cur, max, limit) = (0, 0, 100)
                                                                                                                                |  BYTES in = (0), out = (0)
                                                                                                                                   +->POOL http-pool
                                                                                                                                   |  LB METHOD leastconn
                                                                                                                                   |  LB PROTOCOL L7
                                                                                                                                   |  Transparent disabled 
                                                                                                                                   |  SESSION (cur, max, limit, total) = (0, 0, 103, 0)
                                                                                                                                   |  BYTES in = (0), out = (0)
                                                                                                                                      +->POOL MEMBER: http-pool/m1, STATUS: UP
                                                                                                                                      |  |  STATUS = UP, MONITOR STATUS = default_http_monitor:OK 
                                                                                                                                      |  |  SESSION (cur, max, limit, total) = (0, 0, , 0)
                                                                                                                                      |  |  BYTES in = (0), out = (0)
                                                                                                                                      +->POOL MEMBER: http-pool/m2, STATUS: UP
                                                                                                                                      |  |  STATUS = UP, MONITOR STATUS = default_http_monitor:OK 
                                                                                                                                      |  |  SESSION (cur, max, limit, total) = (0, 0, , 0)
                                                                                                                                      |  |  BYTES in = (0), out = (0)
                                                                                                                                      +->POOL MEMBER: http-pool/m3, STATUS: UP
                                                                                                                                      |  |  STATUS = UP, MONITOR STATUS = default_http_monitor:OK 
                                                                                                                                      |  |  SESSION (cur, max, limit, total) = (0, 0, , 0)
                                                                                                                                      |  |  BYTES in = (0), out = (0)


                                                                                                                                Load Balancer pool configuration statistics example

                                                                                                                                The output of this REST API call:

                                                                                                                                URL: https://NSX_Manager_IP/api/4.0/edges/{edgeId}/loadbalancer/statistics 
                                                                                                                                Method: GET

                                                                                                                                Output example:

                                                                                                                                <?xml version="1.0" encoding="UTF8"?>
                                                                                                                                <loadBalancerStatusAndStats>
                                                                                                                                <timeStamp>1359722922</timeStamp>
                                                                                                                                <pool>
                                                                                                                                <poolId>pool1</
                                                                                                                                poolId>
                                                                                                                                <name>poolhttp</
                                                                                                                                name>
                                                                                                                                <member>
                                                                                                                                <memberId>member1</
                                                                                                                                memberId>
                                                                                                                                <name>m1</name>
                                                                                                                                <ipAddress>192.168.101.201</ipAddress>
                                                                                                                                <status> UP  </status>
                                                                                                                                <bytesIn>70771</bytesIn>
                                                                                                                                <bytesOut>74619</bytesOut>
                                                                                                                                <curSessions>0</curSessions>
                                                                                                                                <maxSessions>1</maxSessions>
                                                                                                                                <rate>0</rate>
                                                                                                                                <rateMax>17</rateMax>
                                                                                                                                <totalSessions>142</totalSessions>
                                                                                                                                </member>
                                                                                                                                <member>
                                                                                                                                <memberId>member2</
                                                                                                                                memberId>
                                                                                                                                <name>m2</name>
                                                                                                                                <ipAddress>192.168.101.202</ipAddress>
                                                                                                                                <status> UP  </status>
                                                                                                                                <bytesIn>70823</bytesIn>
                                                                                                                                <bytesOut>70605</bytesOut>
                                                                                                                                <curSessions>0</curSessions>
                                                                                                                                <maxSessions>1</maxSessions>
                                                                                                                                <rate>0</rate>
                                                                                                                                <rateMax>17</rateMax>
                                                                                                                                <totalSessions>141</totalSessions>
                                                                                                                                </member>
                                                                                                                                <status> UP  </status>
                                                                                                                                <bytesIn>141594</bytesIn>
                                                                                                                                <bytesOut>145224</bytesOut>
                                                                                                                                <curSessions>0</curSessions>
                                                                                                                                <maxSessions>2</maxSessions>
                                                                                                                                <rate>0</rate>
                                                                                                                                <rateMax>34</rateMax>
                                                                                                                                <totalSessions>283</totalSessions>
                                                                                                                                </pool>
                                                                                                                                <virtualServer>
                                                                                                                                <virtualServerId>virtualServer9</
                                                                                                                                virtualServerId>
                                                                                                                                <name>http_vip</name>
                                                                                                                                <ipAddress>10.117.35.172</ipAddress>
                                                                                                                                <status>OPEN</status>
                                                                                                                                <bytesIn>141594</bytesIn>
                                                                                                                                <bytesOut>145224</bytesOut>
                                                                                                                                <curSessions>1</curSessions>
                                                                                                                                <httpReqTotal>283</httpReqTotal>
                                                                                                                                <httpReqRate>0</httpReqRate>
                                                                                                                                <httpReqRateMax>34</httpReqRateMax>
                                                                                                                                <maxSessions>2</maxSessions>
                                                                                                                                <rate>0</rate>
                                                                                                                                <rateLimit>0</rateLimit><rateMax>2</rateMax>
                                                                                                                                <totalSessions>13</totalSessions>
                                                                                                                                </virtualServer>
                                                                                                                                </loadBalancerStatusAndStats>

                                                                                                                                HTTP result codes
                                                                                                                                200 Ok

                                                                                                                                Tags

                                                                                                                                Edge traffic not balanced

                                                                                                                                See Also

                                                                                                                                Request a Product Feature

                                                                                                                                To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

                                                                                                                                Feedback

                                                                                                                                • 4 Ratings

                                                                                                                                Did this article help you?
                                                                                                                                This article resolved my issue.
                                                                                                                                This article did not resolve my issue.
                                                                                                                                This article helped but additional information was required to resolve my issue.

                                                                                                                                What can we do to improve this information? (4000 or fewer characters)




                                                                                                                                Please enter the Captcha code before clicking Submit.
                                                                                                                                • 4 Ratings
                                                                                                                                Actions
                                                                                                                                KB: