Updating the vRealize Log Insight Internal Certificate
search cancel

Updating the vRealize Log Insight Internal Certificate

book

Article ID: 337277

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

This article is to inform the reader that the vRealize Log Insight internal certificate of all vRealize Log Insight 8.10 and lower deployments will expire on April 30th 2023.

Symptoms:
  • If vRealize Log Insight is running on default (out of box) certificates, the issue can present as the external certificate expiring, but in reality it's the internal certificate backing it that is expiring:
Number1.jpg
  • To validate whether the internal certificate is expiring, run the following command from the vRealize Log Insight Primary node:

openssl x509 -noout -in /storage/core/loginsight/cidata/cassandra/config/cacert.pem -enddate

Example:
image.png

Note: If using the default certificate, you can also navigate to the vRealize Log Insight UI and click the Warning or Warning padlock icon in the URL to view the certificate details.
image.png image.png

Note: If you receive a file not found error for the above command, you can check the internal cert's validity with the following keytool command:

keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc | openssl x509 -noout -enddate

Example:
image.png



Environment

VMware vRealize Log Insight 8.x

Resolution

This is a known issue affecting all versions of vRealize Log Insight.

To resolve this problem and avoid any issues, either perform the workaround detailed in this article, or upgrade to VMware Aria Operations for Logs 8.12 before the April 30th deadline.

Important: Run the following command on the vRealize Log Insight Primary node to confirm if the current certificate allows for Client Auth as one of the extended key use:
echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -purpose | grep 'SSL client :'

If the above command returns SSL client : Yes, then you can upgrade to VMware Aria Operations for Logs 8.12.  Please read Important Upgrade Notes on Certificates from the release notes before upgrading.

If the above command returns a value of SSL client : No, DO NOT attempt the VMware Aria Operations for Logs 8.12 upgrade.  Instead, create and apply a new custom certificate that allows Client Auth or revert to the default certificate by following the Revert to the default certificate instructions.  A custom certificate can be applied after upgrading to VMware Aria Operations for Logs 8.12, however the custom certificate must allow ClientAuth.

Revert to the default certificate

  1. Log into the vRealize Log Insight UI as the local admin user.
  2. Navigate to Configuration > SSL.
  3. Click RESET TO DEFAULTS....
image.png
  1. Click Reset.
image.png
 

If you are unable to upgrade to VMware Aria Operations for Logs 8.12, you must update/replace the existing certificates using the steps documented in this article.
To see which steps are required, see which of the Scenarios bellow apply to you.
  • Scenario 1: Environment is using the default self-signed certificate that is expiring on April 30th 2023.The default certificate will need to be replaced by either a new custom certificate or by a new self-signed certificate. Once this is completed, the internal certificate will then need to be updated. The KB documents all of these steps.
  • Scenario 2: Environment is using a updated self-signed certificate with an expiry date set after April 30th 2023. The internal certificate needs to be replaced using the steps in this KB. (Workaround Option 1 or Workaround Option 2 using the attached update_default_cert_v3.tar file.)
  • Scenario 3: Environment is already using a custom certificate with an expiry date set after April 30th 2023. The internal certificate needs to be replaced using the steps in this Article.  (Workaround Option 1 or Workaround Option 2 using the attached update_default_cert_v3.tar file.)
  • Scenario 4: Environment is using a VMware vRealize Lifecycle Manager generated certificate with an expiration date set after April 30th 2023. The internal certificate needs to be replaced using the steps in this KB. (Workaround Option 1 or Workaround Option 2 using the attached update_default_cert_v3.tar file.)
     


Workaround:

Quick Links:

 

Prerequisites

  • Prior to proceeding, please preform a Guest Shutdown of all vRealize Log Insight nodes in the cluster and take snapshots.
  • The update_default_cert.sh script attached to this article has been tested and validated on all supported versions of vRealize Log Insight.  If the environment is running a version that is passed its EOGs date, then please upgrade to a supported version before executing the script.  Refer to the Lifecycle Matrix for a list of supported versions.
  • The workaround assumes that the SSL certificate is already updated from the default self-signed certificate to a custom, or a new self-signed certificate, with an expiration date later than April 30 2023.  This can be checked via a browser pointed to the vRealize Log Insight instance as noted in the Symptoms section.
If that's not the case, see Install a Custom SSL Certificate to create/install a new custom certificate, or use the following steps to generate a self-signed certificate, before executing the workaround steps.
 

Generate a self-signed certificate

  1. Log into the Primary node as root via SSH or Console.
  2. Run the following command to generate a self-signed certificate:
openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 3650 -out domain.crt -nodes

Note: This command will generate a self-signed certificate that is valid for 3650 days (10 years). You may alter the -days value as needed per your organization's security requirements.

Note: When prompted by openssl, provide the required values for your company.  If you want to use the default certificate options, enter the following values:
 
PromptValue
CountryUS
State Or ProvinceCalifornia
LocalityPalo Alto
OrganizationVMware, Inc.
Organization UnitvCenter Log Insight
Common NameVMware vCenter Log Insight
  1. Run the following command to concatenate the key and certificate into a pem file
cat domain.key domain.crt > /tmp/cert.pem
  1. Using an SCP utility like WinSCP, connect to the primary node as root, and download the /tmp/cert.pem file to your desktop.
  2. Log into the vRealize Log Insight UI as the local admin, expand the main menu and navigate to Configuration > SSL.
  3. Click Choose File, browse to the cert.pem file downloaded from step 4 and click Open.
  4. Click Save. This will automatically distribute the new certificate across all nodes in the vRealize Log Insight cluster.  Wait for the SSL certificate to be updated.
  5. Refresh the vRealize Log Insight UI and validate whether the new certificate was applied via the browser.
  • After you either have a self-signed certificate in place by following the above steps, or you have implemented a CA cert, with an expiration date later than April 30 2023, proceed to copy the update_default_cert_v3.tar file attached to this article on the right under the Attachments section to your vRealize Log Insight node(s).
  1. Download then copy the update_default_cert_v3.tar file to the /root directory using an SCP utility like WinSCP.
  2. Log into the vRealize Log Insight node as root via SSH or Console.
  3. Run the following command to extract the tar file:
tar xvf /root/update_default_cert_v3.tar
  1. Run the following command to update the permissions of the script:
chmod 744 /root/update_default_cert.sh

Examples:
image.png
  1. Repeat steps 1-4 for all nodes in the vRealize Log Insight cluster.
 
Follow one of the two options below to implement the workaround.

Workaround Option 1

This option requires a simultaneous restart of all of the nodes, which is faster but doesn't guarantee ingestion and UI HA through VIP(s) during the procedure.

For each vRealize Log Insight node, complete the following:

  1. Log into the vRealize Log Insight node as root via SSH or Console.
  2. Run the following command to navigate to the root directory:
cd /root
  1. Run the following command on all nodes to execute the script with the --all option:
./update_default_cert.sh --all
  1. Once completed on all of the nodes, stop the loginsight service on all of the nodes by running the following command:
systemctl stop loginsight

Important: After running this, wait until all nodes in the cluster have returned to the command prompt prior to proceeding.
  1. Start the loginsight service on all of the nodes by running the following command:
systemctl start loginsight
 

Examples:
number8.jpg

Note: If you receive the following output from running the script, you must generate and apply a certificate including the ClientAuth extended key usage as previously described in this article, and then re-attempt the workaround.
Current certificate does not have SSL client purpose
Please upload a new one from UI and retry

Workaround Option 2

This option requires more steps, but guarantees ingestion and UI HA through VIP(s) during the procedure.
 

Stage 1

For each vRealize Log Insight node, complete the following:
  1. Log into the vRealize Log Insight node as root via SSH or Console.
  2. Run the following command to navigate to the root directory:
cd /root
  1. Run the following command to execute the script with the --stage1 option:
./update_default_cert.sh --stage1
  1. Run the following command to restart the loginsight service and wait until it's up and running before proceeding to the next node:
systemctl restart loginsight
 
Examples:
number9.jpg

Once Stage 1 has been implemented on all nodes in the vRealize Log Insight cluster, proceed to stage 2.
 

Stage 2

Ensure the steps in Stage 1 have been completed for each node, then for each vRealize Log Insight node, complete the following:
  1. Log into the vRealize Log Insight node as root via SSH or Console.
  2. Run the following command to navigate to the root directory:
cd /root
  1. Run the following command to execute the script with the --stage2 option:
./update_default_cert.sh --stage2
  1. Run the following command to restart the loginsight service and wait until it's up and running before proceeding to the next node:
systemctl restart loginsight

Examples:
number10.jpg

Note: If you receive the following output from running the script, you must generate and apply a certificate including the ClientAuth extended key usage as previously described in this article, and then re-attempt the workaround.

Current certificate does not have SSL client purpose
Please upload a new one from UI and retry

 



Additional Information

Verifying the Workaround

For each Log Insight node, complete the following:
  1. Log into the vRealize Log Insight node as root via SSH or Console.
  2. Run the following command to navigate to the root directory:
cd /root
  1. Run the command below to execute the script with the verify option:
./update_default_cert.sh --verify

Example:
number11.jpg
  1. Alternatively, you can run the following command to validate that the new certificate is in place:
openssl x509 -noout -in /storage/core/loginsight/cidata/cassandra/config/cacert.pem -enddate

Note: After the procedure has been verified as successful, please remember to remove the snapshots within ~72 hours to avoid performance issues within Log Insight.

Impact/Risks:
Before applying the resolution steps, it is recommended to perform a Guest Shutdown of all vRealize Log Insight nodes in the cluster and take snapshots.

Note: After the procedure has been verified as successful, please remember to remove the snapshots within ~72 hours to avoid performance issues within vRealize Log Insight.

If a new web certificate is applied to vRealize Log Insight, the Log Insight agents may disconnect from vRealize Log Insight.  See Main SSL Functions for more information.  It is recommended to Configure the vRealize Log Insight Agent SSL Parameters.

Attachments

update_default_cert_v3 get_app