Search the VMware Knowledge Base (KB)
View by Article ID

Resuming rolling upgrade of a VMware vRealize Log Insight cluster (2123064)

  • 0 Ratings

Symptoms

VMware vRealize Log Insight 3.0 introduced the ability to upgrade all nodes of a Log Insight cluster sequentially. If a cluster-wide upgrade fails part-way through, the upgrade should roll back to a pre-upgraded state.

If rollback also fails, such as from an infrastructure problem, the cluster is left in a partially-upgraded state. This article provides steps to manually resume the upgrade.

Resolution

Engage VMware Support prior to performing the steps in this article. For more information, see How to File a Support Request.

To disable automatic cluster-wide upgrade and manually resume upgrade of a cluster:

  1. Use one of the methods in Changing internal configuration options in VMware vRealize Log Insight (2123058) to create and set options in the <upgrade> section.

    <config>
      <upgrade>
        <automated-upgrade-enabled value="false" />
        <automated-rollback-enabled value="false" />
        <automated-snapshot-enabled value="false" />
      </upgrade>
    </config>


  2. Reboot each node for this configuration change to take effect.

  3. Open a console or SSH session to the Master node of the Log Insight cluster and log in as root.

  4. Switch the current working directory using this command:

    cd /usr/lib/loginsight/application/lib/apache-cassandra-*/bin

  5. Open the Cassandra CQL shell using the command:

    ./cqlsh

  6. At the cqlsh> prompt, select the "logdb" keyspace using this command:

    cqlsh> use logdb;

  7. Determine the status of all historic upgrades using this command:

    cqlsh:logdb> select version, status from upgrade_status;

    Example output:

    version       | status
    --------------+-----------------
    2.5.1-1234567 |         Complete
    2.6.3-2345678 | CreatingSnapshot


    Identify the highest version number from this list.

  8. Mark the upgrade as failed, using this command and the version number from step 7:

    cqlsh:logdb> update upgrade_status set status='Failed' where version = 'VERSIONNUMBER';

  9. Determine the cluster nodes which were in-progress upgrading to this version, using the version number from step 7:

    cqlsh:logdb> select node_id, status from node_upgrade_status where version = 'VERSIONNUMBER';

    Example output:

    node_id                              | status
    -------------------------------------+-----------------
    00e80c91-41a5-4563-8f7e-5cf05371abb3 |          Pending
    167def1a-8034-4b3e-81ee-0fab760d7c5f | CreatingSnapshot
    1d514478-aff1-475a-b50c-a653087fb2be |          Pending


  10. Set the status of all cluster nodes from step 9 to Failed, using the version number from step 7:

    cqlsh:logdb> update node_upgrade_status set status = 'Failed' where version = 'VERSIONNUMBER' and node_id = 'NODEID1';
    cqlsh:logdb> update node_upgrade_status set status = 'Failed' where version = 'VERSIONNUMBER' and node_id = 'NODEID2';
    ...

  11. Upgrade each node in the Log Insight cluster. To check the upgrade procedure for the appropriate version, see the vRealize Log Insight Upgrade Path section of  Administering vRealize Log Insight.

    Note: If a node takes more than 10 minutes to return from Restarting, refresh your browser. Wait for the node to finish upgrading and restarting before continuing with the next node.

  12. Re-enable automated rolling upgrade so it can be used for the next upgrade. Use one of the methods in Changing internal configuration options in VMware vRealize Log Insight (2123058) to revert the options in the <upgrade> section. As the cluster is now running Log Insight 3.0 or later, use the web browser method.

    <config>
      <upgrade>
        <automated-upgrade-enabled value="true"/>
        <automated-rollback-enabled value="true"/>
        <automated-snapshot-enabled value="true"/>
      </upgrade>
    </config>


    Note: No reboot is required after making this configuration change.


See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 0 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 0 Ratings
Actions
KB: