NSX-T NCP troubleshooting and debug logging
search cancel

NSX-T NCP troubleshooting and debug logging

book

Article ID: 327341

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

This KB article describes NCP troubleshooting steps and how to enable debug logging

Symptoms:
NCP troubleshooting steps and enable debug logging


Environment

VMware NSX-T Data Center

Resolution

For PKS setup , NCP runs as a service, hence login PKS master node 
steps can be found in https://kb.vmware.com/s/article/67648

If NCP runs as a POD such as openshift setup, SSH into the NCP pod

#oc project nsx-system

# oc get pods

NAME                   READY     STATUS    RESTARTS   AGE

nsx-ncp-snn56          1/1       Running   1          3d

nsx-node-agent-fk9xl   2/2       Running   331        3d

nsx-node-agent-lzxmn   2/2       Running   8          3d

nsx-node-agent-smjlw   2/2       Running   8          3d

nsx-node-agent-tgj8m   2/2       Running   18         3d

nsx-node-agent-w9dsr   2/2       Running   1444       3d

nsx-node-agent-zb4h6   2/2       Running   20         3d

#kubectl exec -it nsx-ncp-snn56 -- /bin/bash
 

Collect NCP logs from Bosh / OC / Kubectl

#bosh logs -d [service-instance_uuid]

#oc logs [ncp-pod-name]

#kubectl logs [ncp-pod-name]

NCP configuration file (ncp.ini) in PKS 

ncp.ini located under /var/vcap/jobs/ncp/config
settings can be adjusted . 


content looks like this:

[DEFAULT]
use_stderr = False

[coe]
enable_snat = true
cluster = pks-60c7bef1-a796-4276-8001-cfa0aca85707

[ha]
enable = True
[k8s]

apiserver_host_ip = master.cfcr.internal
apiserver_host_port = 8443
ca_file = /etc/nsx-ujo/certs/k8s/ca.crt
client_cert_file = /etc/nsx-ujo/certs/k8s/client.crt
client_private_key_file = /etc/nsx-ujo/certs/k8s/client.key
ingress_mode = nat
...

In PKS master node root mode or NCP pod , execute nsxcli 
nsxcli command path may different in various setup 

I.E. in PKS 
# /var/vcap/data/jobs/ncp/45626f1c68983490975053434143ed2eff8b138b/bin/nsxcli

81b91b91-3001-45cb-8ec3-c035e70b118f> get version
2.3.1

81b91b91-3001-45cb-8ec3-c035e70b118f> get ncp-nsx status
NSX Manager status: Healthy

81b91b91-3001-45cb-8ec3-c035e70b118f> get ncp-master status
This instance is the NCP master
Current NCP Master id is c9a754fc-c3f4-4957-b140-3ad238b7160b
Current NCP Instance id is c9a754fc-c3f4-4957-b140-3ad238b7160b
Last master update at Mon Apr  1 20:11:46 2019

81b91b91-3001-45cb-8ec3-c035e70b118f> get ncp-k8s-api-server status
Kubernetes ApiServer status: Healthy

To Debug NCP , change logging level from Info to Debug

81b91b91-3001-45cb-8ec3-c035e70b118f> set ncp-log-level DEBUG
NCP log level is changed to DEBUG

To Stop /Start / restart NCP in PKS master node

# monit restart ncp
# monit stop ncp
# monit start ncp

To disable NCP HA
change ncp.ini and restart ncp
[ha]
enable = True to  False

To change NCP HA HB and timeout timer
By default below values are hardcoded with default values, If we need change these timers then add below parameters and restart ncp.
 
ncp.ha.master_timeout = 9
ncp.ha.heartbeat_period = 3
 
[ha]
master_timeout = 18
heartbeat_period = 6
update_timeout = 6

verify if the ncp is running on the master node
#monit summary | grep ncp

Also check Hyperbus status on ESXi host

vESXi-1.nsx-lab.local> get hyperbus connection info
                VIFID                            Connection                         Status
acbffb45-c762-4dd6-8c6e-6fa8584f74af         169.254.1.10:2345                     HEALTHY