Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228 and CVE-2021-45046, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.

VMware vRealize Orchestrator 7.6.x

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available.

Workaround:

Note:  If you have applied the workaround from this KB article prior to December 17, 2021 you will need to re-run through to ensure you have the latest fixes as outlined by Apache.

Note: The following procedure requires downtime of the system due to service restarts.

Prerequisites

• Backup the vRO appliance nodes

Procedure

Note: The following procedure is for external instances only.  For embedded instances, see Workaround instructions to address CVE-2021-44228 in vRealize Automation 7.6

SSH login to each vRO appliance node and run the following steps:

1. Stop the vco-configurator service on each vRO Node:

   service vco-configurator stop

2. Run the command below to update the vRO configuration on each vRO Node:

base64 -d <<< "bG9nX21lc3NhZ2VfbG9nNGooKSB7CiAgZWNobyAiWyQoZGF0ZSAtLXV0YyAiKyVGVCVULiUzTloiKV0gJDEiIHwgdGVlIC1hIC92YXIvbG9nL3Ztd2FyZS92Y28vYXBwLXNlcnZlci92Y29fbG9nNGpfY3ZlLmxvZwp9Cgpsb2dfZXJyb3JfbG9nNGooKSB7CiAgbG9nX21lc3NhZ2VfbG9nNGogIkVSUk9SOiAkMSIKICBleGl0IDEKfQoKc2V0X2phdmFfb3B0KCkgewogIGxvY2FsIGZpbGU9IiQxIgogIGxvY2FsIGJha3VwX3N1ZmZpeD0iJChkYXRlIC0tdXRjICsiJVklbSVkJUglTSIpIgogIAoKICBpZiBncmVwIC1xICdEbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlJyAkZmlsZQogIHRoZW4gCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiVGhlIGphdmEgcHJvcGVydHkgbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIGlzIGFscmVhZHkgc2V0IGluICRmaWxlLiIKICBlbHNlCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiQ3JlYXRpbmcgYmFjayB1cCBmb3Igc2V0ZW52IGZpbGUgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCIKICAgIGNwIC1mICIkZmlsZSIgIiRmaWxlLiRiYWt1cF9zdWZmaXgiCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiQWRkaW5nIC1EbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIHRvIEpWTV9PUFRTIGluICRmaWxlIgogICAgcmVzPSQoYXdrICdGTlI9PU5SeyBpZiAoL15KVk1fT1BUUz0vKSBwPU5SOyBuZXh0fSAxOyBGTlI9PXB7IHByaW50ICJKVk1fT1BUUz1cIiRKVk1fT1BUUyAtRGxvZzRqMi5mb3JtYXRNc2dOb0xvb2t1cHM9dHJ1ZVwiIiB9JyAkZmlsZSAkZmlsZSkgfHwgbG9nX2Vycm9yX2xvZzRqICJGYWlsZWQgdG8gZWRpdCAkZmlsZSIKICAgIGVjaG8gIiRyZXMiID4gJGZpbGUKICBmaQp9Cgp1cGRhdGVfdnJvX3RvbWNhdF9zdGFydCgpIHsKICBsb2NhbCBmaWxlPSIkMSIKCiAgaWYgZ3JlcCAtcSAncGF0Y2hfYWxsX3BsdWdpbnNfdjIgPicgJGZpbGUKICB0aGVuIAogICAgbG9nX21lc3NhZ2VfbG9nNGogIk5vdGhpbmcgdG8gZG8uIEtCIGFscmVhZHkgYXBwbGllZCEiCiAgZWxzZQogICAgbG9jYWwgcmVzCiAgICBsb2NhbCBiYWt1cF9zdWZmaXg9IiQoZGF0ZSAtLXV0YyArIiVZJW0lZCVIJU0iKSIKICAgIGxvZ19tZXNzYWdlX2xvZzRqICJDcmVhdGluZyBiYWNrIHVwIGZvciB0b21jYXQgc3RhcnR1cCBjb25maWcgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCIKICAgIGNwIC1mICIkZmlsZSIgIiRmaWxlLiRiYWt1cF9zdWZmaXgiCgogICAgbG9nX21lc3NhZ2VfbG9nNGogIk1vZGlmeWluZyB2Uk8gdG9tY2F0IHN0YXJ0dXAgY29uZmlnIC0gJGZpbGUiCgogICAgaWYgZ3JlcCAtcSAnI2xvZzRqX2N2ZV93b3JrYXJvdW5kJyAkZmlsZQogICAgdGhlbgogICAgICBzZWQgLWkgJ3MvI2xvZzRqX2N2ZV93b3JrYXJvdW5kL1xuI2xvZzRqX2N2ZV93b3JrYXJvdW5kL2cnICRmaWxlCiAgICAgIHNlZCAtaSAnL2xvZ19tZXNzYWdlX2xvZzRqMl9jdmUgIlBhdGNoaW5nIGRvbmUuIiQve247cy8uKi99XG4jbG9nNGpfY3ZlX3dvcmthcm91bmRfZW5kL30nICRmaWxlCiAgICAgIHNlZCAtaSAnL3ZSTyBzZXJ2ZXIgc2VydmljZSBkaWQgbm90IHN0YXJ0IHdpdGhpbiB0aGUgZXhwZWN0ZWQgcGVyaW9kLiokL3tuO247bjtzLy4qL31cbiNsb2c0al9jdmVfd29ya2Fyb3VuZF9lbmQvfScgJGZpbGUKICAgICAgc2VkIC1pICcvI2xvZzRqX2N2ZV93b3JrYXJvdW5kLywvI2xvZzRqX2N2ZV93b3JrYXJvdW5kX2VuZC9jXGRlbGV0ZV9tYXJrZXJcJyAkZmlsZQogICAgICBwZXJsIC1pIC0wcGUgJ3MvKC4qKVxuLipkZWxldGVfbWFya2VyXG4vXDEvZzsnICRmaWxlCgogICAgICBzZWQgLWkgJy9sb2c0al9jdmVfd29ya2Fyb3VuZCA+L2QnICRmaWxlIAogICAgICBzZWQgLWkgJy9wYXRjaF9hbGxfcGx1Z2lucyA+L2QnICRmaWxlCiAgICBmaQogICAgc2VkIC1pICcvYmFzZTY0IC1kIDw8PC9kJyAkZmlsZSAKCiAgICBpZiAhIGdyZXAgLXFFICdhY3Rpb249InN0YXJ0InxTdGFydGluZyB0Y1NlcnZlcicgIiRmaWxlIgogICAgdGhlbgogICAgICBsb2dfZXJyb3JfbG9nNGogIlVuYWJsZSB0byBhcHBseSBwYXRjaDogVW5leHBlY3RlZCBmb3JtYXQgb2YgdlJPIHRvbWNhdCBzdGFydHVwIGNvbmZpZyAtICRmaWxlLiIKICAgICAgZXhpdCAxCiAgICBmaQoKICAgIGxvY2FsIHV0aWw9IiQoZGlybmFtZSAiJGZpbGUiKS9jdmVfdXRpbC5zaCIKICAgIGJhc2U2NCAtZCA8PDwgIkkyeHZaelJxWDJOMlpWOTNiM0pyWVhKdmRXNWtDbXh2WjE5dFpYTnpZV2RsWDJ4dlp6UnFNbDlqZG1WZmRqSW9LU0I3Q2dsbFkyaHZJQ0piSkNoa1lYUmxJQzB0ZFhSaklDSXJKVVpVSlZRdUpUTk9XaUlwWFNBa01TSUtmUW9LQ25CaGRHTm9YM0JzZFdkcGJsOTJNaWdwSUhzS0NXeHZZMkZzSUhCc2RXZHBibDl3WVhSb1BTSWtNU0lLQ1d4dlkyRnNJSEJzZFdkcGJsOXVZVzFsUFNJa0tHSmhjMlZ1WVcxbElDSWtjR3gxWjJsdVgzQmhkR2dpS1NJS0NXeHZZMkZzSUhSbGJYQmZjR0YwYUQwaUwzUnRjQzhrY0d4MVoybHVYMjVoYldVaUNnbHNiMk5oYkNCaVlXTnJkWEJ6WDNCaGRHZzlJaTkxYzNJdmJHbGlMM1pqYnk5aVlXTnJkWEJ6SWdvS0NXMXJaR2x5SUMxd0lDSWtZbUZqYTNWd2MxOXdZWFJvSWdvS0NYSnRJQzF5WmlBaUpIUmxiWEJmY0dGMGFDSUtDWFZ1ZW1sd0lDMXhJQ0lrY0d4MVoybHVYM0JoZEdnaUlDMWtJQ0lrZEdWdGNGOXdZWFJvSWlCOGZDQmxlR2wwSURFS0Nna2pJRU5vWldOcklHbG1JSFJvWlhKbElHbHpJR0VnYm1WbFpDQjBieUIxY0dSaGRHVWdkR2hsSUhCc2RXZHBiZ29KYVdZZ1ptbHVaQ0FpSkhSbGJYQmZjR0YwYUNJZ0xYaGtaWFlnTFhSNWNHVWdaaUF0Ym1GdFpTQW5iRzluTkdvdFkyOXlaUzB5S21waGNpY2dMV1Y0WldNZ0wzVnpjaTlpYVc0dmVtbHdJQzF6WmlBaWUzMGlJRnc3SUh3Z1ozSmxjQ0J2Y21jdllYQmhZMmhsTDJ4dloyZHBibWN2Ykc5bk5Hb3ZZMjl5WlM5c2IyOXJkWEF2U201a2FVeHZiMnQxY0M1amJHRnpjenNLQ1hSb1pXNEtDUWxzYjJkZmJXVnpjMkZuWlY5c2IyYzBhakpmWTNabFgzWXlJQ0pOYjJScFpubHBibWNnY0d4MVoybHVPaUFrY0d4MVoybHVYMjVoYldVaUNna0piWFlnSWlSd2JIVm5hVzVmY0dGMGFDSWdJaVJpWVdOcmRYQnpYM0JoZEdnaUNna0pabWx1WkNBaUpIUmxiWEJmY0dGMGFDSWdMWGhrWlhZZ0xYUjVjR1VnWmlBdGJtRnRaU0FuYkc5bk5Hb3RZMjl5WlMweUttcGhjaWNnTFdWNFpXTWdjMmdnTFdNZ0p5OTFjM0l2WW1sdUwzcHBjQ0F0Y1NBdFpDQWllMzBpSUc5eVp5OWhjR0ZqYUdVdmJHOW5aMmx1Wnk5c2IyYzBhaTlqYjNKbEwyeHZiMnQxY0M5S2JtUnBURzl2YTNWd0xtTnNZWE56T3lCMGIzVmphQ0F0ZENBeU1ESXhNREV3TVRBd01EQWdJbnQ5SWljZ1hEc0tDUWtvWTJRZ0lpUjBaVzF3WDNCaGRHZ2lJRHNnZW1sd0lDMXhJQzF5SUMxWUlDMUVJQ0lrY0d4MVoybHVYM0JoZEdnaUlDb3BJSHg4SUdWNGFYUWdNUW9LQ1FraklFWnBlQ0J5YVdkb2RITUtDUWxqYUc5M2JpQjJZMjg2ZG1OdklDSWtjR3gxWjJsdVgzQmhkR2dpQ2drSlkyaHRiMlFnTURZME5DQWlKSEJzZFdkcGJsOXdZWFJvSWdvSkNXeHZaMTl0WlhOellXZGxYMnh2WnpScU1sOWpkbVZmZGpJZ0lsTjFZMk5sYzNObWRXeHNlU0J3WVhSamFHVmtJSEJzZFdkcGJqb2dKSEJzZFdkcGJsOXVZVzFsSWdvSlpXeHpaUW9KQ1d4dloxOXRaWE56WVdkbFgyeHZaelJxTWw5amRtVmZkaklnSWs1dmRHaHBibWNnZEc4Z1pHOGdabTl5SUhCc2RXZHBiam9nSkhCc2RXZHBibDl1WVcxbElnb0pabWtLQ1hKdElDMXlaaUFpSkhSbGJYQmZjR0YwYUNJS2ZRb0tjR0YwWTJoZllXeHNYM0JzZFdkcGJuTmZkaklvS1NCN0NnbG1iM0lnWm1sc1pTQnBiaUF2ZFhOeUwyeHBZaTkyWTI4dllYQndMWE5sY25abGNpOXdiSFZuYVc1ekx5b3VaR0Z5Q2dsa2J3b0pDWEJoZEdOb1gzQnNkV2RwYmw5Mk1pQWlKR1pwYkdVaUlIeDhJR3h2WjE5dFpYTnpZV2RsWDJ4dlp6UnFNbDlqZG1WZmRqSWdJa1ZTVWs5U09pQkdZV2xzWldRZ2RHOGdjR0YwWTJnZ2NHeDFaMmx1T2lBa0tHSmhjMlZ1WVcxbElDUm1hV3hsS1NJS0NXUnZibVVLQ2dsc2IyZGZiV1Z6YzJGblpWOXNiMmMwYWpKZlkzWmxYM1l5SUNKUVlYUmphR2x1WnlCa2IyNWxMaUlLZlFvPSIgPiAiJHV0aWwiCgogICAgc2VkIC1pICdzLGV2YWwgZXhlYywnInNvdXJjZSAkdXRpbCInXG4mLCcgIiRmaWxlIgogICAgZ3JlcCAtcSAic291cmNlICR1dGlsIiAiJGZpbGUiIHx8IHsgbG9nX2Vycm9yX2xvZzRqICJGYWlsZWQgdG8gZWRpdCAkZmlsZS4gQmFja3VwIGNhbiBiZSBmb3VuZCBpbiAkZmlsZS4kYmFrdXBfc3VmZml4IjsgZXhpdCAxOyB9CgogICAgc2VkIC1yIC1pICcvYWN0aW9uPSJzdGFydCJ8U3RhcnRpbmcgdGNTZXJ2ZXIvYSBcXHRcdFx0cGF0Y2hfYWxsX3BsdWdpbnNfdjIgPj4gL3Zhci9sb2cvdm13YXJlL3Zjby9hcHAtc2VydmVyL3Zjb19sb2c0al9jdmUubG9nJyAkZmlsZQogICAgZ3JlcCAtcSAncGF0Y2hfYWxsX3BsdWdpbnNfdjIgPicgIiRmaWxlIiB8fCB7IGxvZ19lcnJvcl9sb2c0aiAiRmFpbGVkIHRvIGVkaXQgJGZpbGUuIEJhY2t1cCBjYW4gYmUgZm91bmQgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCI7IGV4aXQgMTsgfQoKICAgIGxvZ19tZXNzYWdlX2xvZzRqICJTdWNjZXNzZnVsbHkgbW9kaWZpZWQgdGhlIHZSTyB0b21jYXQgc3RhcnR1cCBjb25maWcgLSAkZmlsZSIKICBmaQp9CgoKKHNldF9qYXZhX29wdCAiL3Vzci9saWIvdmNvL2NvbmZpZ3VyYXRpb24vYmluL3NldGVudi5zaCIgJiYgc2V0X2phdmFfb3B0ICIvdXNyL2xpYi92Y28vYXBwLXNlcnZlci9iaW4vc2V0ZW52LnNoIiAmJiB1cGRhdGVfdnJvX3RvbWNhdF9zdGFydCAiL3Zhci9saWIvdmNvL2FwcC1zZXJ2ZXIvYmluL2luaXQuZC5zaCIpIHx8IGxvZ19lcnJvcl9sb2c0aiAiRmFpbGVkIHRvIGFwcGx5IHRoZSBsb2c0aiBDVkUgd29ya2Fyb3VuZCBmb3IgdlJPLiBGb3IgbW9yZSBkZXRhaWxzIHNlZSAvdmFyL2xvZy92bXdhcmUvdmNvL2FwcC1zZXJ2ZXIvdmNvX2xvZzRqX2N2ZS5sb2cuIg==" | sh -

3. Run the command below (on any single vRO Node) to update the Control Center (not applicable to versions 7.2 and 7.3):

   /usr/lib/vco/tools/configuration-cli/bin/vro-configure-inner.sh controlcenter-update
   

4. Restart the services on each vRO Node:

   service vco-server restart && service vco-configurator start
   

Validation

To validate that the workaround has succeeded, take the following steps on all nodes:

1. Verify that all vco processes are running with the java property log4j2.formatMsgNoLookups=true:

ps aux | grep -i java | grep Dlog4j2.formatMsgNoLookups=true

2. Monitor the log /var/log/vco/app-server/vco_log4j_cve.log file, until you see 'Patching done.'
3. Run the command below to verify that the JndiLookup.class is not present in any log4j jar file for 2.x versions:

find / -xdev -type f -name 'log4j-core-2*jar' -exec sh -c '/usr/bin/unzip -l "{}" | grep -q org/apache/logging/log4j/core/lookup/JndiLookup.class && echo Found in file: {}' \;

https://www.vmware.com/security/advisories/VMSA-2021-0028.html

Change log

Date	Change
December 13th 2021 - 13:11 MST	Drafted initial document with initial workaround.
December 13th 2021 - 14:30 MST	Modified horizon-service restart due to a backslash incorrectly placed.
December 15th 2021 - 11:17 MST	Modified scripting to address the new guidance that the JVM_OPTS workaround is not enough and address the new CVE.
December 17th 2021 - 07:29 MST	Setting the "Dlog4j2.formatMsgNoLookups=true", as per latest recommendations from security team.  Making this article more robust so that it can: Be applied over random combination of the previous KB versions. Handle non-standard plugin names (e.g containing spaces). Back-up modified vco files. Provide more detailed logging.
January 11th 2022 - 04:42 MST	Revised validation command.
January 24th 2022 - 14:40 MST	Formatting and SEO