VMware Workaround Instructions To Address CVE-2021-44228 In vCenter Server and vCenter Cloud Gateway

Customers should already be aware of the latest Apache Log4j vulnerability which is affecting servers worldwide. Any system that currently runs Log4j must take action to address this critical vulnerability. VMware have issued workaround instructions to address CVE-2021-44228 In vCenter Server and vCenter Cloud Gateway. This vulnerability has a CVE score of 10 (critical), customers are advised to issue these workarounds immediately.

If customers require assistance from Sebae, please get in touch with us.

Updated On: 13/12/2021

CVE(s): CVE-2021-44228

Impacted Products

• VMware vCenter Server and vCenter Cloud Gateway

2. Introduction

CCVE-2021-44228 has been determined to impact vCenter Server 7.0.x, vCenter 6.7.x & vCenter 6.5.x via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 – VMSA-2021-0028

Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes and subscribe the article for updates.

Impact / Risks

• VCHA needs to be removed before executing the steps in this KB article.
• Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.

Resolution

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available

Workaround

To apply the workaround for CVE-2021-44228 with an automated script, please use the following link:

Python script to automate the workaround steps of VMSA-2021-0028 vulnerability on vCenter Server Appliance (Recommended)

To manually apply the workaround for CVE-2021-44228 to vCenter Server Appliance 7.x and 6.x, skip to the relevant sections below:

To manually apply the workaround for CVE-2021-44228 to vCenter Server Appliance 7.x and 6.x, skip to the relevant sections below:

Click here for vCenter Server Appliance 7.0.x workaround

Click here for vCenter Server Appliance 6.7.x workaround

Click here for vCenter Server Appliance 6.5.x workaround

Click here for vCenter Server Appliance 6.0.x workaround

Note: For vCenter Cloud Gateway, only the steps for the vMon Service and Analytics Service are necessary.

vCenter Server Appliance 7.0.x Workaround

vMON Service

1. Backup the existing java-wrapper-vmon file

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

2. Update the java-wrapper-vmon file with a text editor such as vi

vi /usr/lib/vmware-vmon/java-wrapper-vmon

3. At the very bottom of the file, replace the very last line with 2 new lines

Execute this step number ‘3’ based on the vCenter version running in your environment.
NOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-

• vCenter 7.0 Update 3, 3a, 3b
• vCenter 7.0 Update 2, 2a, 2b, 2c, 2d

Original
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
  Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args

NOTE :- The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below :-

• vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
• vCenter 7.0 Update 1, U1a, U1c, U1d

Original
exec $java_start_bin $jvm_dynargs “$@”

Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@”

4. Ensure the file permissions are set correctly with the below commands:

chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

4. Restart vCenter Services

service-control –stop –all
service-control –start –all

Update Manager Service

1. Back up the existing start.ini file

cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak

2. Update the start.ini file

vi /usr/lib/vmware-updatemgr/bin/jetty/start.ini

3. Add the following line to the end of the file

-Dlog4j2.formatMsgNoLookups=true

4. Restart the Update Manager Service

service-control –restart vmware-updatemgr

Analytics Service

1. Back up the log4j-core-2.8.2.jar file

cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak

2. Run the zip command to disable the class

zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

3. Restart the Analytics service

service-control –restart vmware-analytics

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

1. Verify if the vMon services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

2. Verify the Update Manager changes are shown under “System Properties” in the output of the following two commands:

cd /usr/lib/vmware-updatemgr/bin/jetty/
java -jar start.jar –list-config
 
System Properties:
——————
 log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)

3. Verify the Analytics Service changes:

grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
  This should return 0 lines 

vCenter Server Appliance 6.7.x Workaround

vMON Service

1. Backup the existing java-wrapper-vmon file

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

2. Update the java-wrapper-vmon file with a text editor such as vi

vi /usr/lib/vmware-vmon/java-wrapper-vmon

3. At the very bottom of the file, replace the very last line with 2 new lines

Original exec $java_start_bin $jvm_dynargs “$@”

Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@” 

3. Restart vCenter Services

service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

• chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
• chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

Analytics Service

NOTE:- The below workaround (Analytics service) is applicable for vCenter Server Appliance 6.7 Update 3o and Older versions only. vCenter Server Appliance 6.7 Update 3p is by default covered by vMON Service workaround and is verified with the ‘ps auxww’ command. 

1. Back up the log4j-core-2.8.2.jar file

cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak

2. Run the zip command to disable the class

zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

3. Restart the Analytics service

service-control –restart vmware-analytics  

CM Service

1. Back up the log4j-core.jar file

cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak

2. Run the zip command to disable the class

zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

3. Restart the CM service

service-control –stop vmware-cm
service-control –start vmware-cm

Secure Token Service

1. Back up and edit the the vmware-stsd file

cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak vi /etc/rc.d/init.d/vmware-stsd

2. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

start_service()
{
  perform_pre_startup_actions

  local retval
  JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
  $JSVC_BIN -procname $SERVICE_NAME \
            -home $JAVA_HOME \
            -server \
            <snip>
            -Dauditlog.dir=/var/log/audit/sso-events  \
            -Dlog4j2.formatMsgNoLookups=true \
            $DAEMON_CLASS start

3. Restart the vmware-stsd service

service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

1. Back up and edit the the vmware-sts-idmd file

cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak vi /etc/rc.d/init.d/vmware-sts-idmd

2. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
          -wait 120 \
          -server \
          <snip>
          -Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
          -Dlog4j2.formatMsgNoLookups=true \
          $DEBUG_OPTS \
          $DAEMON_CLASS

3. Restart the vmware-sts-idmd service

service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

1. Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

2. Verify the Analytics Service changes:

grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
  This should return 0 lines

3. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

vCenter Server Appliance 6.5.x Workaround 

vMON Service

1. Backup the existing java-wrapper-vmon file

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

2. Update the java-wrapper-vmon file with a text editor such as vi

vi /usr/lib/vmware-vmon/java-wrapper-vmon

3. At the very bottom of the file, replace the very last line with 2 new lines

Original exec $java_start_bin $jvm_dynargs “$@”

Updated
log4j_arg=”-Dlog4j2.formatMsgNoLookups=true”
exec $java_start_bin $jvm_dynargs $log4j_arg “$@” 

3. Restart vCenter Services

service-control –stop –all
service-control –start –all

Note: If the services do not start, ensure the file permissions are set correctly with these commands:

• chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
• chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

CM Service

1. Back up the log4j-core.jar file

cp -rfp /usr/lib/vmware-cm/lib/log4j-core.jar /usr/lib/vmware-cm/lib/log4j-core.jar.bak

2. Run the zip command to disable the class

zip -q -d /usr/lib/vmware-cm/lib/log4j-core.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

3. Restart the CM service

service-control –stop vmware-cm
service-control –start vmware-cm

Secure Token Service

1. Back up and edit the the vmware-stsd file

cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak vi /etc/rc.d/init.d/vmware-stsd

2. Find the section labeled start_service(). Insert a new line near line 266, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

start_service()
{
  perform_pre_startup_actions

  local retval
  $JSVC_BIN -procname $SERVICE_NAME \
            -home $JAVA_HOME \
            -server \
            <snip>
            -Dauditlog.dir=/var/log/audit/sso-events  \
            -Dlog4j2.formatMsgNoLookups=true \
            $DAEMON_CLASS start

3. Restart the vmware-stsd service

service-control –stop vmware-stsd
service-control –start vmware-stsd

Identity Management Service

1. Back up and edit the the vmware-sts-idmd file

cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak vi /etc/rc.d/init.d/vmware-sts-idmd

2. Insert a new line near line 177 before “$DEBUG_OPTS \” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
          -wait 120 \
          -server \
          <snip>
          -Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
          -Dlog4j2.formatMsgNoLookups=true \
          $DEBUG_OPTS \
          $DAEMON_CLASS

3. Restart the vmware-sts-idmd service

service-control –stop vmware-sts-idmd
service-control –start vmware-sts-idmd

PSC Client Service

1. Back up and edit the vmware-psc-client file

cp -rfp /etc/rc.d/init.d/vmware-psc-client /root/vmware-psc-client.bak
vi /etc/rc.d/init.d/vmware-psc-client

2. Insert a new line near line 300, just before “$DAEMON_CLASS start” with “-Dlog4j2.formatMsgNoLookups=true \” as seen in the example:

$JSVC_BIN -procname $SERVICE_NAME \
        -home $JAVA_HOME \
        -server \
        <snip>    
        -Djava.io.tmpdir=”$CATALINA_BASE/temp” \
        -Dlog4j2.formatMsgNoLookups=true \
        $DAEMON_CLASS start

3. Restart the vmware-psc-client service

service-control –stop vmware-psc-client
service-control –start vmware-psc-client  

Verify the changes

Once all sections are complete, use the following steps to confirm if they were implemented successfully.

1. Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:

ps auxww | grep formatMsgNoLookups

Check if the processes include -Dlog4j2.formatMsgNoLookups=true

2. Verify the CM Service changes:

grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l

This should return 0 lines

vCenter Server Appliance 6.0 U3j Workaround

vCenter Server Appliance 6.0 U3j is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. Mitigation steps have been identified as follows:

1. Back up and edit /usr/lib/vmware-perfcharts/wrapper/conf/wrapper.conf on the appliance and add a new line just below “wrapper.java.additional.13=-Dlog4j.configurationFile=file:/etc/vmware-perfcharts/log4j2.xml” (line 72) with the following content:

wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true

2. Save the file, stop the service and then start it through service-control:

service-control –stop vmware-perfcharts
service-control –start vmware-perfcharts

Note: vCenter Server Appliance versions 6.0GA – 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.

• /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
• /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
• /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
• /opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar

Related Information To revert the workarounds, replace the modified files with the backups created in each step.
VCHA needs to be disabled before executing the steps in this KB.

Change log:

• December 12th 2021 – 11:20 PST: Updated Knowledge Base article advising workarounds for 6.5/6.7 partially address the vulnerability.
• December 12th 2021 – 16:00 PST: Added steps for stsd, idmd and psc-client services to the workaround sections for 6.5 and 6.7.
• December 12th 2021 – 17:00 PST: Added workaround for VCSA 6.0U3j.
• December 12th 2021 – 18:30 PST: Added additional information for older versions of VCSA 6.0.
• December 12th 2021 – 20:30 PST: Updated versions with vulnerable jar files in VCSA 6.0.
• December 13th 2021 – 01:57 PST: Updated the steps in the vMON service commands for permission validation
• December 13th 2021 – 16:30 PST: Simplified and re-arranged beginning of workaround section (no new content). Added clarification to 6.7U3p analytics verification steps. Updated cm service restart commands. Added highlighting to changelog.
• December 13th 2021 – 20:05 PST: Removed DBCC Utility Workaround step from this article. VMware has tested this and does not have any impact or risk hence the workaround section was removed.