VMware Workaround Instructions To Address CVE-2021-44228 In vCenter Server Windows

James Dean

2 years ago

Customers should already be aware of the latest Apache Log4j vulnerability which is affecting servers worldwide. Any system that currently runs Log4j must take action to address this critical vulnerability. VMware have issued workaround instructions to address CVE-2021-44228 In vCenter Server Windows 6.5 & 6.7. This vulnerability has a CVE score of 10 (critical), customers are advised to issue these workarounds immediately.

If customers require assistance from Sebae, please get in touch with us.

Updated On: 13/12/2021

CVE(s): CVE-2021-44228

Impacted Products

VMware vCenter Server Windows 6.5 & 6.7 (vCenter Server)

2. Introduction

CVE-2021-44228 has been determined to impact Windows-based vCenter 6.7.x & vCenter 6.5.x via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 – VMSA-2021-0028

IMPORTANT: This article pertains only to Windows-based vCenter Server 6.5 and 6.7. For information about the vCenter Server Appliance, see Workaround instructions to address CVE-2021-44228 in vCenter Server and vCenter Cloud Gateway.

Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes.

Impact / Risks

This article contains windows registry edit steps. Ensure that a working offline snapshot or backup of the vCenter (or all VCs if ELM mode) exists before starting.
Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.

Resolution

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available

Workaround

vMON Services
(Applicable to all vCenter 6.5/6.7 embedded, vCenter, and PSC nodes)

Step 1: Stop the services within the Windows vCenter.

C:\Program Files\VMware\vCenter Server\bin> .\service-control –stop –all

Step 2: Make a backup of the below two .json files in a different location than the originals and then manually edit them to remove every line starting with “//”. This must be completed before running the script in the next step.

C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-ui.json

example of types of lines to remove
// Enable remote debugging
// NOTE: Use this option only when you really need it. Don’t keep it on by default.
//       It has the potential to cause memory leaks. For further details, see
//       https://bugs.openjdk.java.net/browse/JDK-8164921 as well as our own
//       observations at PR 1878411, comments 21, 33, 34, and 35
//”-Xdebug”,
//”-Xnoagent”,
//”-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8002″,
// Enable JMX
//”-Dcom.sun.management.jmxremote”,
//”-Dcom.sun.management.jmxremote.port=9876″,
//”-Dcom.sun.management.jmxremote.local.only=false”,
//”-Dcom.sun.management.jmxremote.authenticate=false”,
//”-Dcom.sun.management.jmxremote.ssl=false”,5-bri

C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-client.json

example of types of lines to remove
// This option will be removed soon. See JIRA VSUIP-180
// Enable remote debugging
// NOTE: Use this option only when you really need it. Don’t keep it on by default.
//       It has the potential to cause memory leaks. For further details, see
//       https://bugs.openjdk.java.net/browse/JDK-8164921 as well as our own
//       observations at PR 1878411, comments 21, 33, 34, and 35
//”-Xdebug”,
//”-Xnoagent”,
//”-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8001″,
// Enable JMX
//”-Dcom.sun.management.jmxremote”,
//”-Dcom.sun.management.jmxremote.port=9875″,
//”-Dcom.sun.management.jmxremote.local.only=false”,
//”-Dcom.sun.management.jmxremote.authenticate=false”,
//”-Dcom.sun.management.jmxremote.ssl=false”,

Note: Do not simply uncomment these lines. Remove them completely.

Step 3: Execute the python script named vMON.py attached to this KB article

The script can be executed from any path using the following command:

C:\%VMWARE_PYTHON_BIN% vMON.py

Note: If you have issues with running the script in Windows, run the following: echo “%VMWARE_PYTHON_BIN%”

Use the resulting value for the path to python.exe. For example:
“C:\Program Files\VMware\vCenter Server\python\python.exe” vMON.py

Step 4: Run the following commands to restart all services via vMON:

C:\Program Files\VMware\vCenter Server\bin> .\service-control –stop –all
C:\Program Files\VMware\vCenter Server\bin> .\service-control –start –all
C:\Program Files\VMware\vCenter Server\bin> .\service-control –status

Secure Token Services (STS)
(Applicable only to vCenter 6.5/6.7 embedded and PSC nodes)

Step 1: Backup the file manually from the below location

C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\conf\wrapper.conf
Copy this file to C:\old-svcCfgfiles\sts-wrapper.conf

Step 2: Edit and add the following line at the end of “# Java Additional Parameters” section

wrapper.java.additional.27=”-Dlog4j2.formatMsgNoLookups=true”

Output of the file wrapper.conf sample, add the RED line below

PSC Client
(Applicable only to vCenter 6.5 embedded and PSC nodes)

Step 1: Manually backup the file c:\ProgramData\VMware\vCenterServer\runtime\vmware-psc-client\conf\wrapper.conf to c:\old-svcCfgfiles\psc-client-wrapper.conf
Step 2: Edit c:\ProgramData\VMware\vCenterServer\runtime\vmware-psc-client\conf\wrapper.conf
and add the following line wrapper.java.additional.23=”-Dlog4j2.formatMsgNoLookups=true” at the end of the “# Java Additional Parameters” section.

The below is the output of the file psc-client-wrapper.conf, add the RED line below

#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
#encoding=UTF-8
wrapper.license.type=DEV
wrapper.license.id=201702150000053
wrapper.license.licensee=VMware Global, Inc.
wrapper.license.group=Ops
wrapper.license.dev_application=VMware Platform Services Controller Client
wrapper.license.features=pro, 64bit
wrapper.license.upgrade_term.begin_date=2009-10-27
wrapper.license.upgrade_term.end_date=2018-01-27
wrapper.license.key.1=075b-bf64-8f5d-84af
wrapper.license.key.2=a8b6-b28d-62e3-072c
wrapper.license.key.3=962d-5f1d-f01d-d6ff
wrapper.license.key.4=b460-2f95-3124-0ebb
#********************************************************************
# Wrapper Properties
#********************************************************************
set.CATALINA_HOME=%VMWARE_CIS_HOME%\apachetomcat
set.CATALINA_BASE=%VMWARE_RUNTIME_DATA_DIR%\vmware-psc-client
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
# Java Home
set.JAVA_HOME=%VMWARE_JAVA_HOME%
# Java Additional Parameters
wrapper.java.additional.1=”-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed”
wrapper.java.additional.2=”-Dcatalina.base=%CATALINA_BASE%”
wrapper.java.additional.3=”-Dcatalina.home=%CATALINA_HOME%”
wrapper.java.additional.4=”-Djava.io.tmpdir=%CATALINA_BASE%\temp”
wrapper.java.additional.5=”-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager”
wrapper.java.additional.6=”-Djava.util.logging.config.file=%CATALINA_BASE%\conf\logging.properties”
wrapper.java.additional.7=”-Dwrapper.dump.port=-1″
wrapper.java.additional.8=”-Xmx512M”
wrapper.java.additional.9=”-Xss256K”
wrapper.java.additional.10=”-Djdk.map.althashing.threshold=512″
wrapper.java.additional.11=”-XX:MaxPermSize=256m”
wrapper.java.additional.12=”-XX:+HeapDumpOnOutOfMemoryError”
wrapper.java.additional.13=”-XX:HeapDumpPath=%CATALINA_BASE%\logs”
wrapper.java.additional.14=”-XX:+PrintGCDetails”
wrapper.java.additional.15=”-XX:+PrintGCDateStamps”
wrapper.java.additional.16=”-XX:+PrintTenuringDistribution”
wrapper.java.additional.17=”-Xloggc:%CATALINA_BASE%\logs\gc.log”
#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
#encoding=UTF-8
wrapper.license.type=DEV
wrapper.license.id=201702150000053
wrapper.license.licensee=VMware Global, Inc.
wrapper.license.group=Ops
wrapper.license.dev_application=VMware Platform Services Controller Client
wrapper.license.features=pro, 64bit
wrapper.license.upgrade_term.begin_date=2009-10-27
wrapper.license.upgrade_term.end_date=2018-01-27
wrapper.license.key.1=075b-bf64-8f5d-84af
wrapper.license.key.2=a8b6-b28d-62e3-072c
wrapper.license.key.3=962d-5f1d-f01d-d6ff
wrapper.license.key.4=b460-2f95-3124-0ebb
#********************************************************************
# Wrapper Properties
#********************************************************************
set.CATALINA_HOME=%VMWARE_CIS_HOME%\apachetomcat
set.CATALINA_BASE=%VMWARE_RUNTIME_DATA_DIR%\vmware-psc-client
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
# Java Home
set.JAVA_HOME=%VMWARE_JAVA_HOME%
# Java Additional Parameters
wrapper.java.additional.1=”-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed”
wrapper.java.additional.2=”-Dcatalina.base=%CATALINA_BASE%”
wrapper.java.additional.3=”-Dcatalina.home=%CATALINA_HOME%”
wrapper.java.additional.4=”-Djava.io.tmpdir=%CATALINA_BASE%\temp”
wrapper.java.additional.5=”-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager”
wrapper.java.additional.6=”-Djava.util.logging.config.file=%CATALINA_BASE%\conf\logging.properties”
wrapper.java.additional.7=”-Dwrapper.dump.port=-1″
wrapper.java.additional.8=”-Xmx512M”
wrapper.java.additional.9=”-Xss256K”
wrapper.java.additional.10=”-Djdk.map.althashing.threshold=512″
wrapper.java.additional.11=”-XX:MaxPermSize=256m”
wrapper.java.additional.12=”-XX:+HeapDumpOnOutOfMemoryError”
wrapper.java.additional.13=”-XX:HeapDumpPath=%CATALINA_BASE%\logs”
wrapper.java.additional.14=”-XX:+PrintGCDetails”
wrapper.java.additional.15=”-XX:+PrintGCDateStamps”
wrapper.java.additional.16=”-XX:+PrintTenuringDistribution”
wrapper.java.additional.17=”-Xloggc:%CATALINA_BASE%\logs\gc.log”
wrapper.java.additional.18=”-XX:+UseGCLogFileRotation”
wrapper.java.additional.19=”-XX:NumberOfGCLogFiles=2″
wrapper.java.additional.20=”-XX:GCLogFileSize=5M”
wrapper.java.additional.21=”-XX:ErrorFile=%CATALINA_BASE%\logs\hs_err_lotus_pid%p.log”
wrapper.java.additional.22=”-Djava.security.properties=%VMWARE_CFG_DIR%\java\vmware-override-java.security”
wrapper.java.additional.23=”-Dlog4j2.formatMsgNoLookups=true”
….

Identity Management Service
(Applicable only to vCenter Server 6.5/6.7 embedded and PSC nodes)

Step 1: Make sure you have a full a backup of the Registry before moving forward
Step 2: Update registry by launching regedit and change the values :

Regedit HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\VMwareIdentityMgmtService\Parameters\Java

Key to modify:
——————
key – Options
value – in edit box go to end and after –XX:ThreadStackSize=228k Shift+Enter then append
-Dlog4j2.formatMsgNoLookups=true
Reference: complete value of the registry looks as below

-Djava.security.policy=C:\Program Files\VMware\vCenter Server\VMware Identity Services\\server_policy.txt
-Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true
-Dorg.apache.xml.security.ignoreLineBreaks=true
-XX:ErrorFile=C:\ProgramData\VMware\vCenterServer\logs\sso\hs_err_idm_pid%p.log
-XX:HeapDumpPath=C:\ProgramData\VMware\vCenterServer\logs\sso\
-Dvmware.log.dir=C:\ProgramData\VMware\vCenterServer\logs\sso\
-Xmx168m
-XX:CompressedClassSpaceSize=160m
-Xss256k
-XX:ParallelGCThreads=1
-Dlog4j2.formatMsgNoLookups=true

Alternatively

Use the folliwng Powershell script to update the registry:

$opts = $((Get-ItemProperty ‘HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\VMwareIdentityMgmtService\Parameters\Java’).Options)
$opts1 = $opts + “-Dlog4j2.formatMsgNoLookups=true”
Set-ItemProperty -Path ‘HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\VMwareIdentityMgmtService\Parameters\Java’ -Name ‘Options’ -Value $opts1

Component Manager
(Applicable to all vCenter 6.5/6.7 embedded, vCenter, and PSC nodes)

Step 1: Make a backup of the C:\Program Files\VMware\vCenter Server\cm\lib\log4j-core.jar file in a new directory like C:\cm_lib.
Step 2: Rename the C:\Program Files\VMware\vCenter Server\cm\lib\log4j-core.jar file to log4j-core.jar.zip
Double click on the log4j-core.jar.zip
Navigate to folder log4j-core.jar.zip\org\apache\logging\log4j\core\lookup
Delete the file JndiLookup.class
Step 3: Now Rename the log4j-core.jar.zip to log4j-core.jar

Restart all the vCenter Services

C:\Program Files\VMware\vCenter Server\bin> .\service-control –stop –all
Operation not cancellable. Please wait for it to finish…
Performing stop operation on profile: ALL…
Successfully stopped service vmon
Successfully stopped profile: ALL.
Performing stop operation on service VMwareDNSService…
Successfully stopped service VMwareDNSService
Performing stop operation on service VMwareSTS…
Successfully stopped service VMwareSTS
Performing stop operation on service VMwareIdentityMgmtService…
Successfully stopped service VMwareIdentityMgmtService
Performing stop operation on service VMWareCertificateService…
Successfully stopped service VMWareCertificateService
Performing stop operation on service VMWareDirectoryService…
Successfully stopped service VMWareDirectoryService
Performing stop operation on service VMWareAfdService…
Successfully stopped service VMWareAfdService
Performing stop operation on service vmware-cis-config…
Successfully stopped service vmware-cis-config

C:\Program Files\VMware\vCenter Server\bin> .\service-control –start –all
Operation not cancellable. Please wait for it to finish…
Performing start operation on service vmware-cis-config…
Successfully started service vmware-cis-config
Performing start operation on service VMWareAfdService…
Successfully started service VMWareAfdService
Performing start operation on service VMWareDirectoryService…
Successfully started service VMWareDirectoryService
Performing start operation on service VMWareCertificateService…
Successfully started service VMWareCertificateService
Performing start operation on service VMwareIdentityMgmtService…
Successfully started service VMwareIdentityMgmtService
Performing start operation on service VMwareSTS…
Successfully started service VMwareSTS
Performing start operation on service VMwareDNSService…
Successfully started service VMwareDNSService
Performing start operation on profile: ALL…
Successfully started service vmon
Successfully started profile: ALL.

To check the status of the services:

C:\Program Files\VMware\vCenter Server\bin> .\service-control –status
Running:
EsxAgentManager VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS VServiceManager content-library rhttpproxy vPostgres vapiEndpoint vimPBSM vmon vmonapi vmsyslogcollector vmware-cis-config vmware-license vmware-perfcharts vmwareServiceControlAgent vpxd vpxd-svcs vsan-health vsphere-ui vspherewebclientsvc
Stopped:
VMWareCAMService mbcs vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump

vCenter Server 6.0 on Windows

vCenter Server 6.0 is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. Mitigation steps have been identified as follows:

Backup and edit C:\Program Files\VMware\vCenter Server\perfcharts\wrapper\conf\wrapper.conf on the appliance and add a new line below “wrapper.java.additional.13=-Dlog4j.configurationFile=file:” with the following content:

wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true

Save the file and restart the VMware Performance Charts service

Note: vCenter Server Appliance versions 6.0GA – 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.

C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-cs\lib\log4j-api-2.1.jar
C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-cs\lib\log4j-core-2.1.jar
C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-p2p\lib\log4j-api-2.1.jar
C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-p2p\lib\log4j-core-2.1.jar

Verification Steps

Look for java services started by vMON for new Log4j option, Go to Windows task manager.

For Tanuki wrapper services, check the wrapper service command, check in process explorer (if you have it).

STS

PSC client

(OR) Use the following command in power shell to verify all java services

Get-WmiObject Win32_Process -Filter “name = ‘java.exe'” | Select-Object CommandLine | fl

Sample output:

Related Information To revert the workarounds, replace the modified files with the backups created in each step.

Change log:

December 12th 2021 – 19:45 PST: Added information pertaining to vCenter 6.0. Updated various sections to improve clarity.
December 12th 2021 – 20:45 PST: Updated versions with vulnerable jar files in VCSA 6.0.
December 13th 2021 – 12:45 PST: Added Impacts/Risks section. Updated wording in vmon steps. Changed highlighted examples to red text.