Symptoms:

• NSX 4.1.0.2 and above
• NSX Alarms indicate certificates are expired or about to expire

VMware NSX 4.1.0.2

There are two main factors that can contribute to this behaviour:

NSX Managers have many certificates for internal services.
In version NSX 3.2.1, Cluster Boot Manager (CBM) service certificates were incorrectly given a validity period of 825 days instead of 100 years.
This was corrected to 100 years in NSX 3.2.3.
However any environment originally installed on NSX 3.2.1 will have the internal CBM Corfu certs expire after 825 regardless of upgrade to the fixed version or not.

On NSX-T 3.2.x internal server certificates could expire and no alarm would trigger. There was no functional impact.
Starting from NSX 4.1.0.2, NSX alarms now monitor validity of internal certificates and will trigger for expired or soon to expire certificates. Note on NSX 4.1.x, there is currently no functional impact when an internal certificate expires however alarms will continue to trigger.

This is expected behaviour.

Workaround:
Scripted Resolution
VMware have developed a script that will replace all internal self signed certs with new certs of validity period 100 years.
The script is compatible with NSX version 4.1.0 and above.
The script does not replace API and cluster certificates.
An NSX backup must be taken before running the script. Also ensure the passphrase is known.

This is a python version 3 script which should be run from a client machine which has paramiko and cryptography python packages installed.
Depending on the system this may be installed with a command such as #sudo pip3 install cryptography 
The script cannot be run directly on the NSX Manager as it does not have the required python module. It is not supported to install it on the NSX Manager.

1) Download the attached script replace_certs.py.
2) To execute the script run the following command and follow the prompts 3) You will need to input the NSX Manager cluster IP and admin credentials at the relevant prompts.
4) In some environments it may be necessary to increase the timeout value used by the script to allow the script to complete successfully.
     long_wait_time defaults to a value of 60 but can be increased to 180 (or higher) and then re-run the script.

If the script does not work in your environment, there is the option to follow the manual procedure.
Alternatively open an SR to report the failure scenario.



Manual Resolution
It is recommended to upgrade to NSX 4.1.2 or higher before running this procedure manually.
This is because of a known issue impacting NSX 4.1.1 that can result in services failing to start after certificate replacement, see https://kb.vmware.com/s/article/94664 .
 
Internal expired certificates should be replaced to stop alarms triggering.

1.    Generate the self signed certificates for the expired components. Note the maximum validity period if using NSX will be 825 days, use another tool such as openssl to generate the certificate if a longer validity is required.
2.    When using NSX UI, Generate Self signed certificate disable to Service Certificate radio button.
3.    In the Self signed certificates Common Name and the Name of the Certificate should be the same as the expired certificate.
4.    Postman or curl client required for the below API to run.
5.    If there are 3 NSX Manager Nodes, the certificates on each Manager will need to be replaced individually by running the API directly against that Manager.


Local manager
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate>?action=apply_certificate&service_type=LOCAL_MANAGER

MP-cluster
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=MGMT_CLUSTER

Tomcat/API
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=API&node_id=<nsx-mgr-Node-UUID>

API-Corfu Client certificate (CBM_API)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_API&node_id=<nsx-mgr-Node-UUID>

AR-Corfu Client certificate  (CBM_AR)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_AR&node_id=<nsx-mgr-Node-UUID>

CCP-Corfu Client certificate (CBM_CCP)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_CCP&node_id=<nsx-mgr-Node-UUID>

Cluster Manager-Corfu Client certificate (CBM_CLUSTER_MANAGER)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_CLUSTER_MANAGER&node_id=<nsx-mgr-Node-UUID>

CSM-Corfu Client certificate (CBM_CSM)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_CSM&node_id=<nsx-mgr-Node-UUID>

GM-Corfu Client certificate for node (CBM_GM)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_GM&node_id=<nsx-mgr-Node-UUID>

 IDPS reporting-Corfu Client certificate (CBM_IDPS_REPORTING)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_IDPS_REPORTING&node_id=<nsx-mgr-Node-UUID>

Monitoring-Corfu Client certificate (CBM_MONITORING)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>7?action=apply_certificate&service_type=CBM_MONITORING&node_id=<nsx-mgr-Node-UUID>

MP-Corfu Client certificate (CBM_MP)
POST https://<nsx-mgr-IP/FQDN>/api/v1/trust-management/certificates/<New self signed certificate ID>?action=apply_certificate&service_type=CBM_MP&node_id=<nsx-mgr-Node-UUID>

Note:
The Manager service will restart upon replacing this certificate[MP-Corfu Client certificate] but it should automatically come up
In the event the service does not come back up, start it manually from the admin shell of the Manager 
start service manager