vRealize Log Insight node fails to start with error "keystore was tampered with or password was incorrect"
search cancel

vRealize Log Insight node fails to start with error "keystore was tampered with or password was incorrect"

book

Article ID: 320055

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
  • One or more vRealize Log Insight nodes fail to start.
  • You may see an error message similar to:
java.io.IOException: Keystore was tampered with, or password was incorrect
  • The /storage/var/loginsight/runtime.log file may show errors similar to:
[2019-01-09 17:08:12.920+0000] ["main"/10.126.59.179 ERROR] [com.vmware.loginsight.services.EmbeddedService] [Error when starting service com.vmware.loginsight.daemon.shared.TomcatController]
java.io.FileNotFoundException: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.32/conf/keystore does not exist
  • vRealize Log Insight cluster status is Unknown.


Environment

VMware vRealize Log Insight 8.x
VMware vRealize Log Insight 4.3.x
VMware vRealize Log Insight 4.6.x
VMware vRealize Log Insight 4.5.x
VMware vRealize Log Insight 4.8.x
VMware vRealize Log Insight 4.7.x
VMware vRealize Log Insight 4.0.x

Cause

The issue occurs when the keystore is manually edited, or if there are any intermittent storage or uptime issues, such as the vRealize Log Insight node having a force shut down.

Resolution

Single Node Clusters

To resolve the issue, generate a new keystore on the problematic node.
  1. Log into the problematic node as root via SSH or Console.
  2. Stop the loginsight service:
service loginsight stop
  1. Look into the /storage/core/loginsight/config/ directory and note the loginsight-config.xml# file with the highest number.
  2. Open the loginsight-config.xml# file with the highest number, noted in step 3, in a text editor.
Example: vi loginsight-config.xml#31
  1. Find the syslog-ssl-keystore-password tag and delete it.
Example: Remove these lines:  
<importer>
...
    <syslog-ssl-keystore-password value="IC1zBFx21ecBy2oHO93PNNfY3QSm0EQK" />
...
</importer>
  1. Save and close the file.
  2. Run the following command and note the provided password:
grep syslog-ssl-keystore-password /usr/lib/loginsight/application/etc/loginsight-config-base.xml | gawk -F'"' '{print $2}'

Output Example: wXnOwLx8RTO0odq9VsR5jL7iBBEzM3xx
  1. Look into the /usr/lib/loginsight/application/3rd_party/ directory and note the apache-tomcat directory with the highest version.
Example: apache-tomcat-8.5.38
  1. Delete the old keystore file by running the following command:
rm /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore

Note: Replace apache_ver with the apache directory noted in step 8
Example: rm /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.38/conf/keystore
  1. Run the following command to generate a new keystore:
/usr/java/default/bin/keytool -genkey -alias loginsight -keyalg RSA -validity 3650 -keysize 4096 -keystore /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore -keypass keystore_password -storepass keystore_password -dname "CN=VMware vCenter Log Insight, OU=vCenter Log Insight, O=VMware\, Inc., L=Palo Alto, S=California, C=US"

Note: Replace apache_ver with the apache directory noted in step 8 and replace keystore_password with the password noted in step 7.

Example: /usr/java/default/bin/keytool -genkey -alias loginsight -keyalg RSA -validity 3650 -keysize 4096 -keystore /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.38/conf/keystore -keypass wXnOwLx8RTO0odq9VsR5jL7iBBEzM3xx -storepass wXnOwLx8RTO0odq9VsR5jL7iBBEzM3xx -dname "CN=VMware vCenter Log Insight, OU=vCenter Log Insight, O=VMware\, Inc., L=Palo Alto, S=California, C=US"

Note: All arguments can be customized as desired. CN value should be changed with fully qualified domain name associated with the IP address of the endpoint.
You can find more information about keytool usage here: Java Platform, Standard Edition Tools Reference: keytool
  1. Start the loginsight service:
service loginsight start
 

Multi Node Clusters

If some nodes have healthy keystores, copy a valid keystore from a healthy node onto the problematic node.
  1. Log into the problematic node as root via SSH or Console.
  2. Stop the loginsight service:
service loginsight stop
  1. Run the following command to copy the keystore from a healthy node:
scp root@healthy_node_address:/storage/var/loginsight/apache-tomcat/conf/keystore /storage/var/loginsight/apache-tomcat/conf/

Note: Replace healthy_node_address with the IP address of the healthy vRealize Log Insight node.

Example: scp [email protected]:/storage/var/loginsight/apache-tomcat/conf/keystore /storage/var/loginsight/apache-tomcat/conf/
  1. Start the vRealize loginsight service on the affected node:
  2. service loginsight start

If all nodes have a damaged keystore, generate a new keystore on the Primary node, and copy it to all of the worker nodes.
  1. Log into the Primary node as root via SSH or Console.
  2. Run the following command on all nodes to stop the loginsight service:
service loginsight stop
  1. Look into the /usr/lib/loginsight/application/3rd_party/ directory and note the apache-tomcat directory with the highest version.
Example: apache-tomcat-8.5.38
  1. Run the following command on all nodes to remove the damaged keystore:
rm /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore
 
Note: Replace apache_ver with the apache directory noted in step 3.

Example: rm /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.38/conf/keystore
  1. Look into the /storage/core/loginsight/config/ directory and note the loginsight-config.xml# file with the highest number.
  2. Run the following command and note the provided password:
grep syslog-ssl-keystore-password /storage/core/loginsight/config/loginsight-config.xml# | gawk -F'"' '{print $2}'

Note: Replace loginsight-config.xml# with the file noted in step 5.

Example: grep syslog-ssl-keystore-password /storage/core/loginsight/config/loginsight-config.xml#31 | gawk -F'"' '{print $2}'

Output Example: wXnOwLx8RTO0odq9VsR5jL7iBBEzM3xx
  1. Run the following command to generate a new keystore:
/usr/java/default/bin/keytool -genkey -alias loginsight -keyalg RSA -validity 3650 -keysize 4096 -keystore /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore -keypass keystore_password -storepass keystore_password -dname "CN=VMware vCenter Log Insight, OU=vCenter Log Insight, O=VMware\, Inc., L=Palo Alto, S=California, C=US"

Note: Replace apache_ver with the apache directory noted in step 3 and replace keystore_password with the password noted in step 6.

Example: usr/java/default/bin/keytool -genkey -alias loginsight -keyalg RSA -validity 3650 -keysize 4096 -keystore /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.38/conf/keystore -keypass wXnOwLx8RTO0odq9VsR5jL7iBBEzM3xx -storepass wXnOwLx8RTO0odq9VsR5jL7iBBEzM3xx -dname "CN=VMware vCenter Log Insight, OU=vCenter Log Insight, O=VMware\, Inc., L=Palo Alto, S=California, C=US"
  1. Copy the newly generated keystore to the other nodes in the cluster:
scp /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore root@worker_address:/usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore

Note: Replace apache_ver with the apache directory noted in step 3 and replace worker_address with the IP address of a worker node in the cluster.
Example: scp /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore [email protected]:/usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.38/conf/keystore
  1. Repeat step 8 for all other worker nodes in the cluster.
  2. Start the loginsight service on all nodes in the cluster:
service loginsight start


Powered by Wolken Software