Search the VMware Knowledge Base (KB)
View by Article ID

VMware vRealize Log Insight deployment with reused IP address fails to initialize (2126692)

  • 0 Ratings

Symptoms

When the IP address or FQDN used by the new Log Insight deployment was previously used by a member of a Log Insight or Apache Cassandra cluster, you experience these symptoms:

  • Starting a new deployment of VMware vRealize Log Insight fails.

  • In the /storage/var/loginsight/runtime.log file you see entries similar to:

    [yyyy-mm-dd hh:mm:ss.nnn+0000] [main/10.11.12.13 WARN] [com.vmware.loginsight.cassandra.CassandraUtil] [Cassandra cluster not yet ready, retry later, Nodes=x, Up=y, RequiredUp=z]

  • The IP address or FQDN used by the new Log Insight deployment may have previously been used by a member of a Log Insight or Apache Cassandra cluster.

  • The command /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/nodetool status produces a list with more than the one local node, similar to:

    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address       Load    Tokens  Owns   Host ID                           Rack
    UN  10.11.12.13   151.99 MB  256  19.6%  75fd0bf8bd0556fcc3b073fed6aefb36  rack1
    UN  10.11.12.101  156.46 MB  256  20.6%  f79845ca3e7fab13448340b4403ad80f  rack1
    UN  10.11.12.102  135.12 MB  256  19.2%  fe90f17b7f8ff0f45df7aed699b50fbf  rack1
    UN  10.11.12.103  134.84 MB  256  20.6%  b1ab7493430c60e9ded8644f66bc4c11  rack1
    UN  10.11.12.104  127.27 MB  256  20.0%  63e4a2884fad9907fe87347c43ba341a  rack1

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs when another Log Insight or Apache Cassandra cluster attempts to re-establish connectivity with its lost cluster node. The new Log Insight deployment reuses the IP address or FQDN of a former cluster member and the existing cluster establishes a connection with the Apache Cassandra database in the new Log Insight deployment.

Resolution

To resolve this issue, remove the reused IP address from an existing cluster or restrict access to the network segment. After remediation, start the new deployment wizard.

Remove new node from another Log Insight cluster

If the remote nodes are part of an online Log Insight cluster, use the cluster management user interface to remove the reference to the reused IP address. For more information, see the Remove a Worker Node from a vRealize Log Insight Cluster section in the Remove a Worker Node from a vRealize Log Insight Cluster.

  1. Confirm the IP address of the new Log Insight node deployment.
  2. Open the administration interface of the  existing Log Insight cluster.
  3. Under the Management heading, select Cluster.
  4. Locate the reused IP address of the new node in the list of cluster members.
  5. Click the pause button to put the node in maintenance mode.
  6. Click the red X to remove the node.
  7. Navigate to the Web user interface of the new node and select Start New Deployment.

Gracefully remove new node from a Apache Cassandra cluster

If the remote nodes are part of an Apache Cassandra cluster that is not part of Log Insight, use the Cassandra nodetool command to remove the reference to the reused IP address. For more information, see https://wiki.apache.org/cassandra/NodeTool.

  1. Open a console or SSH session to the new Log Insight node and log in as root.
  2. Change the current working directory by running this command:

    cd /usr/lib/loginsight/application/lib/apache-cassandra-*/bin

  3. Use the nodetool command to decommission the new node:

    ./nodetool decommission

  4. Open a console or SSH session to the existing Cassandra cluster.
  5. Change working directory to the Cassandra installation.
  6. Use the nodetool status command to display a list of cluster members, including the Address and UUID of each.
  7. If the reused IP address of the new node is still present in the list of cluster members, use the nodetool removenode command to remove the node:

    ./nodetool removenode UUID

  8. Navigate to the Web user interface of the new node and select Start New Deployment.

Restrict network access

If the remote nodes are unmanageable or changes to them cannot be negotiated, restrict network connectivity between the new and old nodes. Place the new node on a management network segment protected by a firewall from the rest of your internal network. For more information, see the Ports and External Interfaces that Log Insight Uses section of the Ports and External Interfaces that Log Insight Uses section of the Log Insight Security Guide.

  1. Prevent external connectivity to TCP ports 7000 and 9042 on the Log Insight virtual appliance.
  2. Delete and redeploy the new virtual appliance with the reused IP address.
  3. Navigate to the Web user interface of the new node and select Start New Deployment.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 0 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 0 Ratings
Actions
KB: