CVE-2021-44228 & CVE-2021-45046 has been determined to impact Windows-based vCenter 6.7.x & vCenter 6.5.x via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

• CVE-2021-44228 & CVE-2021-45046 - VMSA-2021-0028

IMPORTANT: This article pertains only to Windows-based vCenter Server 6.5 and 6.7. For information about the vCenter Server Appliance, see Workaround instructions to address CVE-2021-44228 in vCenter Server and vCenter Cloud Gateway (87081)

See the Change log at the end of this article for all changes and subscribe to the article for updates.

VMware vCenter Server 6.5.x
VMware vCenter Server 6.0.x
VMware vCenter Server 6.x
VMware vCenter Server 6.7.x

This issue is resolved in:

• vCenter Server 7.0 Update 3c, build 19234570
• vCenter Server 6.7 Update 3q, build 19300125
• vCenter Server 6.5 Update 3s, build 19261680

Please note that it is not necessary to revert the workaround steps in this article before upgrading to a fixed release of vCenter Server.
Do not use the vc_log4j_mitigator.py script on vCenter Servers that have already been upgraded to a fixed version.
 

Workaround:

The workarounds described in this document are meant to be a temporary solution only.

IMPORTANT: vc_log4j_mitigator.py will now mitigate CVE-2021-44228 & CVE-2021-45046 on vCenter Server end-to-end without extra steps. This script replaces the need to run any of the manual steps or use remove_log4j_class.py. However, it is not necessary to run if you've already used those in your environment.

Please use the below 2 scenarios to verify if you're previous steps were sufficient:

Completed remediation scenarios:
1. Used vc_log4j_mitigator.py
or
2. Used the manual workaround steps in this KB and remove_log4j_class.py.

If you are unsure if the previous steps were sufficient, run the vc_log4j_mitigator.py script with the "--dryrun" option to verify if the environment still has vulnerable files. If any are found, run it without the dryrun flag to correct them.

Automated Workaround (Recommended)

To apply the workaround for CVE-2021-44228  & CVE-2021-45046with the unified script:

1. Download the script attached to this KB (vc_log4j_mitigator.py)

2. Open an administrator cmd prompt and navigate to the location of the script.

3. Execute the script copied in step 1:

"%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py

This will stop all vCenter services, updates all necessary files with the formatMsgNoLookups flag, removes the JndiLookup.class from all jar/war files on the appliance, and finally starts all vCenter services. The files that the script modifies will be reported as the script runs.

5. To verify that no more vulnerable files exist, execute the script again with the "dry run" flag:

"%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py -r

The list of vulnerable files found should be zero. The vCenter Server is now fully mitigated.

Note: Certain 3rd party vSphere Client plugins may be detected as vulnerable even after remediation is done. This is because the jar files are recreated each time the client is started. These must be addressed by either patching the plugin to an unaffected version or removing the plugin via the vCenter MOB.

This has been found with older versions of the Pure Storage HTML client plugin 4.3.1 and below (Pure login required) . The file continually marked as vulnerable will be: ProgramData\VMware\vCenterServer\runtime\vsphere-ui\server\work\Catalina\localhost\ROOT\eclipse\configuration\org.eclipse.osgi\330\0\.cp\log4j-core-2.11.2.jar


Optional Arguments
  -h, --help            show this help message and exit
  -d dirnames [dirnames ...], --directories dirnames [dirnames ...]
                        space separated list of directories to check
                        recursively for CVE-2021-44228 vulnerable java archive
                        files.
  -a, --accept-services-restart
                        accept the restart of the services without having
                        manual prompt confirmation for the same
  -r, --dryrun          Run the script and log vulnerable files without
                        mitigating them. The vCenter services are not
                        restarted with this option.
  -b BACKUP_DIR, --backup-dir BACKUP_DIR
                        Specify a backup directory to store original files.
  -l LOG_DIR, --log-dir LOG_DIR
                        Specify a directory to store log files.

Manual Workaround

VMware highly recommends using the vc_log4j_mitigator.py script instead of these manual steps to avoid errors in this process.
 

vMON Services
(Applicable to all vCenter 6.5/6.7 embedded, vCenter, and PSC nodes)

• Step 1:  Stop the services within the Windows vCenter.

• Step 2: Make a backup of the below two .json files in a different location than the originals and then manually edit them to remove every line starting with "//". This must be completed before running the script in the next step.

 

• C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-client.json

• Step 3: Execute the python script named vMON.py attached to this KB article 

C:\%VMWARE_PYTHON_BIN% vMON.py

Note: If you have issues with running the script in Windows, run the following:

Use the resulting value for the path to python.exe. For example: 
"C:\Program Files\VMware\vCenter Server\python\python.exe" vMON.py

• Step 4: Run the following commands to restart all services via vMON:

 

Secure Token Services (STS)
(Applicable only to vCenter 6.5/6.7 embedded and PSC nodes)

• Step 1: Backup the file manually from the below location
  • C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\conf\wrapper.conf
  • Copy this file to C:\old-svcCfgfiles\sts-wrapper.conf
• Step 2: Edit and add the following line at the end of  "# Java Additional Parameters" section

PSC Client
(Applicable only to vCenter 6.5 embedded and PSC nodes)

• Step 1: Manually backup the file c:\ProgramData\VMware\vCenterServer\runtime\vmware-psc-client\conf\wrapper.conf to c:\old-svcCfgfiles\psc-client-wrapper.conf
• Step 2: Edit c:\ProgramData\VMware\vCenterServer\runtime\vmware-psc-client\conf\wrapper.conf
  and add the following line wrapper.java.additional.23="-Dlog4j2.formatMsgNoLookups=true" at the end of the "# Java Additional Parameters" section.

Identity Management Service
(Applicable only to vCenter Server 6.5/6.7 embedded and PSC nodes)

• Step 1: Make sure you have a full a backup of the Registry before moving forward
• Step 2: Update registry by launching regedit and change the values : 

• Key to modify:
  ------------------
  key - Options
  value - in edit box go to end and after --XX:ThreadStackSize=228k Shift+Enter then append
  -Dlog4j2.formatMsgNoLookups=true

• Reference: complete value of the registry looks as below 

Alternatively 

• Use the folliwng Powershell script to update the registry:

Component Manager

This is now remediated with the remove_log4j_class.py script at the end of this section.
 

Run the remove_log4j_class.py script

vCenter Server 6.0 U3j on Windows

Note: It might be necessary to enable "file name extensions" in the Windows explorer view to navigate into the renamed zip file

5. Double click on the log4j-core-2.11.2.jar.zip

6. Navigate to the folder log4j-core-2.11.2.jar.zip\org\apache\logging\log4j\core\lookup

7. Delete the file JndiLookup.class

Rename the log4j-core-2.11.2.jar.zip file back to log4j-core-2.11.2.jar

8. Start the vCenter services

• C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-cs\lib\log4j-api-2.1.jar
• C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-cs\lib\log4j-core-2.1.jar
• C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-p2p\lib\log4j-api-2.1.jar
• C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-p2p\lib\log4j-core-2.1.jar

Verification Steps 

• Verify the script successfully removed JndiLookup.class from all java files with the following command:

 

• Look for java services started by vMON for new Log4j option, Go to Windows task manager.

 

• For Tanuki wrapper services, check the wrapper service command, check in process explorer (if you have it).

STS

 

PSC client

• (OR) Use the following command in power shell to verify all java services

• Sample output:

 

To revert the workarounds, replace the modified files with the backups created in each step.

Change log :

• December 12th 2021 - 19:45 PST: Added information pertaining to vCenter 6.0. Updated various sections to improve clarity.
• December 12th 2021 - 20:45 PST: Updated versions with vulnerable jar files in VCSA 6.0.
• December 13th 2021 - 12:45 PST: Added Impacts/Risks section. Updated wording in vmon steps. Changed highlighted examples to red text.
• December 15th 2021 - 11:15 PST: Added a notice at the top concerning the recent updates on CVE-2021-44228 from Apache. Updated steps under Component Manager to address "file in use" problem.
• December 16th 2021 - 16:15 PST: Added new remove_log4j_class.py script which replaces the Component Manager step and addresses new findings in CVE-2021-44228 and CVE-2021-45046.
• December 17th 2021 - 11:45 PST: Added step to vCenter Server 6.0 U3j instructions to manually remove the JndiLookup.class class.
• December 17th 2021 - 17:00 PST: Updated remove_log4j_class.py to output a change summary and better handle "File is not a zip file" issues. Added section with new cmd line options for remove_log4j_class.py. Added "--dryrun" verification step.
• December 20th 2021 - 18:00 PST: Added vc_log4j_mitigator.py which unifies the previous steps and scripts into a single workaround.
• December 21st 2021 - 1:45 PST: Added note about 3rd party plugins reverting to a vulnerable state after the vsphere client is started. Added clarification that CVE-45046 is also addressed with all the changes since the "December 16th 2021 - 16:15 PST" update
• December 22nd 2021 - 12:00 PST: Added extra information about known older versions of the Pure Storage plugin which need extra steps to remediate
• January 21st 2022 - 09:00 PST: Changed the Automated Workaround (Recommended) step 3 from %VMWARE_PYTHON_BIN% vc_log4j_mitigator.py  to "%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py 
• January 21st 2022 - 09:00 PST:  Changed the Automated Workaround (Recommended) step 5 from %VMWARE_PYTHON_BIN% vc_log4j_mitigator.py -r  to "%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py -r
• January 27th 2022 - 19:30 PST: Updated resolution section with the vCenter 7.0 U3c release.
• February 8th 2022 - 09:00 PST: Added 6.x versions to resolution section.

Impact/Risks:

• This article contains windows registry edit steps. Ensure that a working offline snapshot or backup of the vCenter (or all VCs if ELM mode) exists before starting.
• Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.
• After completing the steps, do not attempt to upgrade the Windows vCenter Server. Attempting to upgrade the vCenter Server will revert the remediations taken or fail outright. VMware will provide steps to address this in a future release.