CVE-2021-44228 & CVE-2021-45046 has been determined to potentially impact VMware NSX-T Data Center via the Apache Log4js open-source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

• CVE-2021-44228 & CVE-2021-45046 – VMSA-2021-0028

December 22nd, 2021 Note: The previous version of the workaround distributed on December 17th, 2021, omitted some JndiLookup class files embedded in internal services within NSX-T. Due to an abundance of caution, we have updated the Debian package to remove additional occurrences of java class files. This updated package is comprehensive and includes the previous workarounds. VMware recommends applying the new Debian package 'unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all.deb' if you have not already completed the previous workaround. There are no known exploits of the aforementioned internal services for customers who have already used the earlier Debian package, 'unified-appliance-log4j2-patch_3.1.3.6.0.19078151_all.deb

• NSX-T 2.5.x should upgrade to 2.5.3.4   
• NSX-T 3.0.x should upgrade to 3.0.3.1 
• NSX-T 3.1.x should upgrade to 3.1.3.5 

1. Take an SFTP Backup of your NSX-T Manager.  

Reference the following document for guidance, if needed: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-E6181BF1-2CB7-4870-B508-BFAF5B47D702.html

2. Download the Debian package 'unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all.deb' from the attachment section on the right side of this KB article.

st en

Enter the root password when prompted.

6. Install the Debian package with the following command, run in the directory in which you've copied the Debian package to on the NSX-T Unified Appliance.

dpkg -i unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all.deb

7. Reboot the system:

/sbin/reboot

Warning: Do not reboot several NSX-T Manager appliances at the same time. Ensure the cluster services stabilize, prior to proceeding to the next NSX-T Manager appliance.

To monitor NSX-T Manager cluster services, at the 'admin' level of SSH, use the following command:

get cluster status  
 

Verification Steps
To verify the workaround for CVE-2021-44228 & CVE-2021-45046 has been correctly applied to each NSX-T Manager Appliance, perform the following steps with root access via SSH on the appliance: 

Step 1: Verify the configuration files have been updated with the “formatMsgNoLookup=true” flag.
grep "wrapper.java.additional.100=" /usr/tanuki/conf/*-wrapper.conf
Example of expected output:

Note:  The number of files below will vary for NSX-T Manager and NSX-T Cloud Service
Manager. The presence of any number of files confirms that the workaround was
successfully applied.

/usr/tanuki/conf/async-replicator-service-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/cluster-boot-manager-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/cm-inventory-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/corfu-log-replication-server-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/corfu-nonconfig-server-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/corfu-server-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/cross-cloud-upgrade-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/idps-reporting-service-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/migration-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/nsx-ccp-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/phonehome-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/policy-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/proton-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/proxy-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/search-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/upgrade-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true

Step 2: Verify the JndiLookup class was removed from all jar files on the appliance.
find /usr /opt -xdev -iname "*.jar" | while read l; do  unzip -l $l|grep "JndiLookup.class"; if [[ ${PIPESTATUS[1]} -eq 0 ]]; then echo $l; fi; done 
Expected Return: None

Note: If any files are found with the above command, please contact VMware Support.
 

To revert the workaround for CVE-2021-44228 & CVE-2021-45046 to VMware NSX-T Data Center perform the following steps on each NSX-T Manager or NSX-T Cloud Service Manager.

• December 13th 2021 - 11:00 PST: Removed validation test steps as it was only applicable to one attack vector, specifically for NSX-T 3.0.1 and earlier. 
• December 13th 2021 - 11:30 PST: Clarified that NSX-T Edge Nodes, VM or Bare Metal, are not affected by this issue. 
• December 13th 2021 - 12:30 PST: Removed erroneous word 'exit' from workaround. 
• December 13th 2021 - 13:00 PST:  Added note regarding re-applying the workaround if an environment is restored to backup. 
• December 13th 2021 - 16:00 PST: Minor edits to include NSX-T Cloud Service Manager to this KB as well.
• December 14th 2021 - 05:00 PST: Minor edit to add -p on cp commands.
• December 14th 2021 - 12:30 PST: Added clarity regarding upgrading to a newer, vulnerable version of NSX-T after applying the workaround.
• December 15th 2021 - 11:30 PST: Added notice acknowledging CVE-2021-45046 and an impending release containing log4j version 2.16. 
• December 17th 2021 - 15:00 PST: Added detail regarding NSX T 3.2.0 release pertinent to CVE-2021-44228 & CVE-2021-45046.
• December 17th 2021 - 17:00 PST: Changed the workaround provided to the new VMware recommended NSX-T workaround.  
• December 21th 2021 - 02:30 PST: Added a note related to workaround which need to be carried by all those customers who followed the previous version of workaround.
• December 22nd 2021 - 10:00 PST: Changed the provided Debian package to an updated version, which provides further mitigation against CVE-2021-44228 & CVE-2021-45046. 
• December 22nd 2021 - 12:00 PST: Minor changes to formatting and verbiage within the article to highlight important of prior update.  No procedural changes. 
• December 22nd 2021 - 14:00 PST: Clarified recommendations around updated Debian package. 
• December 23rd 2021 - 11:00 PST: Added additional versions that provide resolution to CVE-2021-44228 & CVE-2021-45046. Clarified upgrade path guidance. Provided more accurate command in verification step 2.
• December 24th 2021 - 02:10 PST Improve validation Step 2 command to avoid false positives detected from backup files.

A malicious actor with network access to an impacted VMware product may exploit this issue to invoke remote code execution.  However, there are multiple layers of protection in NSX-T that will make exploiting CVE-2021-44228 & CVE-2021-45046 difficult:

• Internal NSX-T processes/JVM’s are prevented from making external connections via the iptables configuration in the NSX-T Unified Appliance.
• The version of Java JDK in NSX-T Datacenter version 3.0.2 and above prevents the most common remote code execution publicized (ie. LDAP attack vector).

Note: NSX-T Edge VM's and Bare Metal Edge Nodes are not affected by this issue. 

Note: If the below workaround is applied to an NSX-T Manager, and that NSX-T Manager is subsequently upgraded to a newer vulnerable version of NSX-T, the workaround must be re-applied post upgrade. 

Note: This workaround will have to be re-applied on the post-restore NSX-T Managers or NSX-T Cloud Service Managers if an environment is restored from backup.