CVE-2021-44228 & CVE-2021-45046 have been determined to impact VMware Cloud Foundation via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review the document before continuing:

• CVE-2021-44228 & CVE-2021-45046 – VMSA-2021-0028

• On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient.
• VMware believes the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.
• VMware expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of VMware Cloud Foundation, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we will be updating this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance. 

• This KB introduces updated scripts that address both CVE-2021-44228 and CVE-2021-45046 and is recommended to be run even if an earlier workaround was applied for CVE-2021-44228
• Subscribe to this article for latest updates and Keep a track on the "Change Log" information updated in the Related Section below

• The workarounds described in this document are meant to be a temporary solution only.
• Upgrades documented in the VMSA-2021-0028 should be applied to remediate CVE-2021-44228 & CVE-2021-45046 as and when upgrades become available.

To Apply the workaround for CVE-2021-44228 and CVE-2021-45046 perform the following steps

1. Click here for the workaround steps for Cloud Builder
   • Click here to run workaround on the Cloud Builder using a automation script
2. Click here for the workaround steps for SDDC Manager
   • Click here to run workaround on the SDDC Manager using a automation script
3. Click here for the workaround steps for Other VMware products within VMware Cloud Foundation BOM
4. Click here for further information on VxRail Manager workaround to remediate Log4Shell

Steps for Cloud Builder
To apply the workaround for CVE-2021-44228 follow below steps
Note: These steps are applicable for all VCF 3.x and 4.x Cloud Builders.

1. Please make sure all imaging runs are completed  and none is in-progress before proceeding with this procedure.
2. Login to the Cloud Builder VM
   • Change the user to 'root' account using command
     • su
3. Run the below command, to stop Host Imaging Service
   • systemctl stop imaging
4. Run the below command to check and confirm the service is stopped 
   • systemctl status imaging
5. Run the below command, to take the backup of the original file before editing it
   • cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh.orig
6. Modify  /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh to add -Dlog4j2.formatMsgNoLookups=true as an extra argument for running the parent imaging service in "nohup /etc/alternatives/jre/bin/java -jar". Make sure this argument is added after -Dserver.port=$VIA_SERVICE_PORT
   • The following is given as an example. Please make sure the equivalent command in the file is modified accordingly: 
   • BEFORE:
   • nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$VIA_SERVICE_PORT -Dspring.config.additional-location=$VIA_EXTERNAL_PROPERTIES_PATH,$VIA_DB_PROPERTIES_FILE -Dserver.servlet.context-path=$VIA_CONTEXT_PATH $VIA_SERVICE_PATH < /dev/null >>$LOGFILE 2>&1 &
   • AFTER:                                                                                                                                      
   •  nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$VIA_SERVICE_PORT -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=$VIA_EXTERNAL_PROPERTIES_PATH,$VIA_DB_PROPERTIES_FILE -Dserver.servlet.context-path=$VIA_CONTEXT_PATH $VIA_SERVICE_PATH < /dev/null >>$LOGFILE 2>&1 &
7. Run the below command, to take the backup of the original file before editing it
   • cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh.orig
8. Modify  /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh to add -Dlog4j2.formatMsgNoLookups=true as an extra argument for running the imaging service in "nohup /etc/alternatives/jre/bin/java -jar". Make sure this argument is added after -Dserver.port=$SECOND. The following is given as an example. Please make sure the equivalent command in the file is modified accordingly:  
   • BEFORE:
   • nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$SECOND -Dspring.config.additional-location=$VIA_DB_PROPERTIES_FILE $name < /dev/null >>$LOGFILE 2>&1 &
   • AFTER:
   • nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$SECOND -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=$VIA_DB_PROPERTIES_FILE $name < /dev/null >>$LOGFILE 2>&1 &
9. Run the below command, to start the Host Imaging Service 
   • systemctl start imaging
10. Run the below command, to confirm Host Imaging service is up and running
    • systemctl status imaging
    • Note: In the event of service startup failure, please revert the config files by following below steps to bring up the service and you may contact VMware Global Support to apply the workaround steps to mitigate the vulnerability :
    • Run the below command, to stop Host Imaging Service
      • systemctl stop imaging
    • Run the below command to confirm the service is stopped 
      • systemctl status imaging
      • Copy the Original files back :
        • cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh.orig /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh
        • cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh.orig /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh
      • Change the file permissions
        • chmod 755 /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh
        • chmod 755 /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh
      • Restart the imaging service
        • systemctl start imaging
11. Run the below command to confirm Host Imaging Service is run with -Dlog4j2.formatMsgNoLookups=true option. The output should look something like the following ones
    • ps -ef|grep jar

To apply the workaround for CVE-2021-45046 follow below steps
Note: These steps are applicable for all VCF 3.x and 4.x Cloud Builders.

1. Log on to Cloud Builder using SSH 
2. Change the user to 'root' account using command
   • su
3. Stop the imaging service
   • systemctl stop imaging
4. Handling the vulnerability in evorack-imaging-esxi-service jar
   1. Create a temporary directory with the command below
      • mktemp -d -t log4j-XXXXXXXXXX
   2. Run the command below to list if log4j-core is present in the jar file evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar
      • zipinfo -1 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar" | grep log4j-core-*
   3. This lists the presence of log4j-core-*.jar in the above jar which confirms the vulnerability
   4. Copy the temp directory path created as part of Step #4.1 and execute below commands
   5. In this instance the temp folder gets created with name log4j-KANsTCxgjM
   6. Example command:                                                                                                                      unzip "/opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar" "BOOT-INF/lib/log4j-core-2.13.1.jar" -d "/tmp/log4j-KANsTCxgjM"
   7. Similarly, replace the temporary folder names in below commands where applicable with created temp folder name from Step #5
      • zipinfo -1 /tmp/log4j-KANsTCxgjM/BOOT-INF/lib/log4j-core-2.13.1.jar | grep -q "org/apache/logging/log4j/core/lookup/JndiLookup.class"
   8. Next run command  echo $?
      • If the returned value is 1 - No further action needed as the class is not present then move to step below "Handling the vulnerability in via.jar" Step#5
      • If returned value is 0 - Means the JndiLookup.class is present and continue running commands below - 
      • zip -q -d "/tmp/log4j-KANsTCxgjM/BOOT-INF/lib/log4j-core-2.13.1.jar" "org/apache/logging/log4j/core/lookup/JndiLookup.class"
   9. This deletes the JndiLookup.class file from the log4j-core*.jar
   10. Update the changed log4j-core to the evorack-imaging jar using command below
       • cd "/tmp/log4j-KANsTCxgjM/" && zip -ur -0 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar"

1. Create a temporary directory with the command - mktemp -d -t log4j-XXXXXXXXXX
2. Run the command below to list if log4j-core is present in the jar via.jar
   • zipinfo -1 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar" | grep log4j-core-*
   • This lists the presence of log4j-core-*.jar in the above jar which confirms the vulnerability
3. Copy the temp directory path created as part of Step #5.1 and execute below commands
   • unzip "/opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar" "BOOT-INF/lib/log4j-core-2.13.1.jar" -d "/tmp/log4j-mO2dWNlgef"
   • zipinfo -1 /tmp/log4j-mO2dWNlgef/BOOT-INF/lib/log4j-core-2.13.1.jar | grep -q "org/apache/logging/log4j/core/lookup/JndiLookup.class"
4. Run the below command 
   • echo $?
   • If the returned value is 1 - No further action needed as the class is not present then move to last step below of "Start the imaging service" Step#5
   • If returned value is 0 - Means the JndiLookup.class is present and continue running commands below
     • zip -q -d "/tmp/log4j-mO2dWNlgef/BOOT-INF/lib/log4j-core-2.13.1.jar" "org/apache/logging/log4j/core/lookup/JndiLookup.class"
     • This deletes the JndiLookup.class file from the log4j-core*.jar
     • Update the changed log4j-core to the via.jar using command below
       • cd "/tmp/log4j-mO2dWNlgef/" && zip -ur -0 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar"
5. Start the imaging service

• systemctl start imaging

• This automated script addresses both CVE-2021-44228 and CVE-2021-45046
• It is recommended to run the script even in case any earlier workaround was applied for CVE-2021-44228

1. Copy the script file named "log4j_via_v3.sh" attached in this KB article to your Cloud Builder via WinSCP, to the /home/admin directory 
2. Log on to Cloud Builder using SSH
3. Change the user to ‘root’ account using the command below
   • su -
4. Change the directory to the location to /home/admin
5. Execute the command below:
   • $bash log4j_via_v3.sh

To apply the workaround for CVE-2021-45046 follow below steps

There is one jar that require mitigation in SDDC Manager - Follow the steps below to fix the same

1. Log on to SDDC Manager VM and change the user to 'root' account using command shown below and provide the root user password to get super user privileges
   • su
2. Stop the vip manager i18n service
   • systemctl stop vip-manager-i18n.service
3. Handling the vulnerability in vip-manager-i18n-common.jar
   1. Create a temporary directory with the command below
      • mktemp -d -t log4j-XXXXXXXXXX
   2. Run the command below to list if log4j-core is present in the jar vip-manager-i18n-common.jar
      • zipinfo -1 "/usr/local/vip/vip-manager-i18n-common.jar" | grep log4j-core-*
      • This lists the presence of log4j-core-*.jar in the above jar which confirms the vulnerability
   3. Copy the temp directory path created as part of Step #3.1 and execute below commands
      • unzip "/usr/local/vip/vip-manager-i18n-common.jar" "BOOT-INF/lib/log4j-core-2.13.3.jar" -d "/tmp/<temp-directory-name>"
      • In this run - the temp folder gets created with name log4j-hUuV38E2Jc
      • Example command:
        • unzip "/usr/local/vip/vip-manager-i18n-common.jar" "BOOT-INF/lib/log4j-core-2.13.3.jar" -d "/tmp/log4j-hUuV38E2Jc"
   4. Similarly, replace the temporary folder names in below commands where applicable with created temp folder name from Step #3.1
      • zipinfo -1 /tmp/log4j-hUuV38E2Jc/BOOT-INF/lib/log4j-core-2.13.3.jar | grep -q "org/apache/logging/log4j/core/lookup/JndiLookup.class"
   5. Run the command below 
      • echo $?
      • If the returned value is 1 - No further action needed as the class is not present the move to Step#4 below to start the vip manager service 
      • If returned value is 0 - Means the JndiLookup.class is present and continue running commands below
   6. When the JndiLookup.class is present - run the command below
      • zip -q -d "/tmp/log4j-hUuV38E2Jc/BOOT-INF/lib/log4j-core-2.13.3.jar" "org/apache/logging/log4j/core/lookup/JndiLookup.class"
      • This deletes the JndiLookup.class file from the log4j-core*.jar
   7. Update the changed log4j-core to the vip-manager-i18n-common jar using command below
      • cd "/tmp/log4j-hUuV38E2Jc/" && zip -ur -0 "/usr/local/vip/vip-manager-i18n-common.jar"
4. Start the vip manager i18n service
   • systemctl start vip-manager-i18n.service

1. Copy the script file named "log4j_vip_v3.sh" attached in this KB article to your SDDC Manager via WinSCP under /home/vcf directory 
2. Log on to SDDC Manager using SSH
3. Change the user to ‘root’ account using the command below
   • su -
4. Change the directory to location to /home/vcf
5.  Execute command below
   • $bash log4j_vip_v3.sh

NOTE:-

• The Knowledge Base Articles listed below has specific workarounds for specific versions for each of the BOM specific components.
• Make sure the Knowledge Base article is followed based on the version of Product running in the VCF environment
• Keep a tab on the below Knowledge Base articles in the related information section on "Change.log" for any new updates

Change Log:

• December 13th 2021 -  1:28AM PST : Updated the BOM components and workaround updates to this article 
• December 13th 2021 -  12:00PM PST : Updated automation script availability column.
• December 13th 2021 -  10:24PM PST : Cloud builder Automation script update 
• December 14th 2021 -   04:46AM PST : SDDC Manager Automation script update
• December 14th 2021 - 05:37AM PST : Horizon, vIDM and Workspace One Access Automation script update
• December 14th 2021 -  10:32PM PST : Added VxRail Manager workaround to remediate Log4Shell
• December 15th 2021 -  09:40am PST : Updated the remediation steps for Cloud Builder and SDDC Manager.
• December 15th 2021 -  02:12PM PST : Updated that the automated script is only available for CVE-2021-44228.
• December 15th 2021 - 03:22PM PST - Removed KB link from resolution section and updated BOM hyperlinks
• December  15th 2021 - 8:58PM PST - Updated the KB for both CVE-2021-44228 and CVE-2021-45046 and Automation Script updated for both Cloud Builder and SDDC Manager 
• December 16th 2021 - 11:05am PST - Changed the formatting of steps for Cloud Build  and SDDC manager.
• December 16th 2021 - 01:25PM PST - Changed the formatting and indentation for manual steps, commands and step wise exceution of commands
• December 18th 2021 - 12:58AM - PST - Updated with new v3 versions of scripts with additional log messaging. NOTE:- If previous v2 versions of the scripts were run in your environment, then there is no need to run again with latest v3 versions.
• December 28th 2021 - 09:51AM PST - Updated BOM Table under section "Steps For Other VMware products within VMware Cloud Foundation BOM" with KB87200 for vRSLCM 2.x (for VCF 3.x environments)
• January 19th 2021 - Updated the Note section for the SDDC Manager workaround steps (Manual and automated) to include upgrade scenarios.

• Products that are part of the VMware Cloud Foundation BOM (Bill of Materials) are impacted.
• Follow VMSA-2021-0028 to identify the impacted product areas.
• VCF Cloud Builder and SDDC Manager, which are part of VMware Cloud Foundation are also impacted.