Workaround Instructions to address CVE-2021-44228 in VMware Identity Manager 3.3.X
search cancel

Workaround Instructions to address CVE-2021-44228 in VMware Identity Manager 3.3.X

book

Article ID: 318270

calendar_today

Updated On:

Products

VMware

Issue/Introduction

CVE-2021-44228 has been determined to impact VMware Identity Manager via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing: 

Environment

VMware Identity Manager 3.3.x

Resolution

The workarounds described in this document are meant to be a temporary solution only. 
Upgrades documented in the aforementioned advisory  should be applied to remediate CVE-2021-44228 when available.

Workaround:

NOTE:

  • It is recommended to upgrade instances of unsupported versions to newer, supported versions first before applying the workaround. This procedure may not work for older unsupported versions. Please refer to the VMware Lifecycle Matrix for a list supported versions of the product.
  • It is strongly recommended to take a snapshot of the appliance before applying the procedure


STEPS:
Use the attached script
log4j_20Dec2021.sh  to make changes. This script removes JNDILookup.class from log4j and embedded jars

  1. Steps to ensure there is vulnerable files present. Run the following command
grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" /opt/vmware/ > /tmp/scanResult.txt
grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" /usr/local/ >> /tmp/scanResult.txt
  1. Run the following command, and view output to see one or more files listed, indicating vulnerable files present
cat /tmp/scanResult.txt
  1. Download the attached log4j_20Dec2021.sh file and scp to the /var/tmp directory of the appliance
  2. Login into appliance as sshuser, sudo to root level access
  3. Change to the /var/tmp directory

                 cd /var/tmp

  1. Run the following command to make the log4j_20Dec2021.sh script executable:

                 chmod +x log4j_20Dec2021.sh

  1. Run the following command to execute the script:

                   ./log4j_20Dec2021.sh

  1. Steps to ensure there is no vulnerability present. Run the following command

grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" /opt/vmware/ > /tmp/validation.txt
grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" /usr/local/ >> /tmp/validation.txt
  1. Run the following command, and view output to see no files listed, indicating vulnerable files not present

    cat /tmp/validation.txt



    The previously published steps were the following. If they were implemented, no need to revert them.
  1. Login as sshuser, sudo to root level access

  2. Use the following commands to back up and modify every logging pattern layout to say %m{nolookups} instead of %m in all of the log4j config/properties files (located at /usr/local/horizon/conf).
cp -r /usr/local/horizon/conf /tmp/conf
sed -i 's/%m/%m{nolookups}/g' /usr/local/horizon/conf/*.*


Note: Do not run the sed command more than once. Restore the files from /tmp/conf if necessary.
  1. Edit the /opt/vmware/horizon/workspace/bin/setenv.sh file. 
  2. Find the "JVM_OPTS=" section, and find the following configuration line:

                -Dset.rmi.server.hostname=true \
            Under that line insert the following new line, and save the file:
                -Dlog4j2.formatMsgNoLookups=true \

  1.  Restart the horizon-workspace service using the command. This will apply the changes made in steps 2 through 4.

   service horizon-workspace restart 


         NOTE: Steps 6 through 7 are needed only if certproxy for android SSO is configured

  1. Modify every logging pattern layout to say %m{nolookups} instead of %m in /opt/vmware/certproxy/conf/cert-proxy-log4j.properties
For example, in VMware Identity Manager 3.3.5, the file /opt/vmware/certproxy/conf/cert-proxy-log4j.properties, the following change would be made:

Original
appender.rollingfile.layout.pattern=%d{ISO8601} %-5p (%t) [%X{orgId};%X{userId};%X{ip};%X{executionId}] %c - %m%n

Updated
appender.rollingfile.layout.pattern=%d{ISO8601} %-5p (%t) [%X{orgId};%X{userId};%X{ip};%X{executionId}] %c - %m{nolookups}%n
          
  1. Restart the certproxy service using the command 

               /etc/init.d/vmware-certproxy restart 

  1. Edit the /opt/vmware/elasticsearch/config/jvm.options file if it is present on the system. If not, you can skip steps 8-10.
 
  1. Find the "# log4j 2" section, and find the following configuration line:

                 -Dlog4j2.disable.jmx=true
            Under that line insert the following configuration, and save the file:
                 -Dlog4j2.formatMsgNoLookups=true

  1. Restart the elasticsearch service using the command 

                 service elasticsearch restart 

Note: If you are running a cluster deployment, repeat the steps above on all additional nodes of the cluster.

Validation Steps for if Workaround has been successfully applied

  1. Run the following command
grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" /opt  /usr > /tmp/scanResult.txt
  1. The file /tmp/scanResult.txt will be empty indicating workaround has been applied successfully

Procedure for Windows Connectors

  1. Download the attached script checkConnectorJndiWindows.bat and execute as administrator as follows

checkConnectorJndiWindows.bat

  1. If you see “VULNERABILITIES FILES FOUND!” proceed with next steps, else abort
  2. This procedure removes the "JndiLookup.class" from the impacted files. Download and install 7-zip from https://www.7-zip.org/ if your Windows server does not already have 7-zip. 
  3. Login into the Windows Connector machine as a user with administrator privileges.
  4. Copy the attached applyPatchJndiWindows21Dec2021.bat to the Windows Connector machine.
  5. Navigate to the folder via the command prompt (Run as Administrator option is needed).
  6. Run the patch script using the command below from the command line:
            applyPatchJndiWindows21Dec2021.bat
  7. Repeat these steps for all Windows Connector machines in your deployment.
  8. If the scanner reports any more vulnerable files, use a command like this to fix them:
"C:\Program Files\7-Zip\7z" d -tzip  "<detected vulnerable file path>"  org/apache/logging/log4j/core/lookup/JndiLookup.class

where "<detected vulnerable file path>" is a file reported by the scanner. 

Additional Information

To revert the CVE-2021-44228 workaround, you can revert to the snapshot taken before applying these steps. Alternatively, revert the changes made to the individual configuration files and restart the services.

Change Log:
December 11th 2021 12:00PM PST: First version published of Workaround
December 13th 2021 3:05PM PST: Added script to automate changes
December 14th 2021 9:40AM PST: Added link to Lifecycle Matrix
December 17th 2021 2:48PM PST: Added updated script to remove JndiLookup.class
December 17th 2021 5:42PM PST: Added a Windows script to remove JndiLookup.class
December 20th 2021 8:15AM PST: Replaced log4j_17Dec2021.sh with log4j_20Dec2021.sh to remove .zip files and extracted files to save space
December 20th 2021 4:00PM PST: Added script checkConnectorJndiWindows.bat to determine vulnerable files were found
December 21st 2021 11:30AM PST: Script applyPatchJndiWindows17Dec2021.bat updated to fix a typo and renamed as applyPatchJndiWindows21Dec2021.bat

Impact/Risks:
Possible compromise due to crafted API calls 

List of affected versions 

3.3.5  - VMware Identity Manager
3.3.4  - VMware Identity Manager
3.3.3  - VMware Identity Manager


Attachments

applyPatchJndiWindows21Dec2021 get_app
applyPatchJndiWindows17Dec2021 get_app
checkConnectorJndiWindows get_app
log4j_17Dec2021 get_app
log4j_20Dec2021 get_app