Search the VMware Knowledge Base (KB)
View by Article ID

Upgrade to vRealize Log Insight 4.0.0 fails with error "State mismatch on bucket" (2147738)

  • 0 Ratings

Symptoms

When upgrading to vRealize Log Insight 4.0.0, you see these symptoms:
  • Upgrade fails.
  • In runtime.log file, you see entries similar to:

    • State mismatch on bucket BucketId(uuid:6a884d03-ec8e-44d9-8c36-ebfb283ebd6c, createTime:1475476031608) size = 483733765 when messages = 0
    • [2016-10-11 22:02:07.471+0000] ["main"/10.145.133.100 ERROR] [com.vmware.loginsight.services.EmbeddedService] [Error when starting service com.vmware.loginsight.analytics.distributed.LogSearchWorkerService]java.io.IOException: Set bucket commit metadata state mismatch at com.vmware.loginsight.ingestion.BucketIndex.setBucketCommitMetadata(BucketIndex.java:316) at com.vmware.loginsight.ingestion.IngestionServer.start(IngestionServer.java:220) at com.vmware.loginsight.analytics.distributed.LogSearchWorkerService.start(LogSearchWorkerService.java:545) at com.vmware.loginsight.services.EmbeddedService.startService(EmbeddedService.java:122) at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:738) at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startService$21(LogInsightDaemon.java:681) at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709) at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:680) at com.vmware.loginsight.daemon.LogInsightDaemon.startAllServices(LogInsightDaemon.java:674) at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:369) at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1005)[2016-10-11 22:02:07.472+0000] ["main"/10.145.133.100 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Failed to start Log Search Worker: Set bucket commit metadata state mismatch]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Purpose

This article provides a workaround when the upgrade to vRealize Log Insight 4.0.0 fails.

Cause

This issue occurs when there is a failure to create a new bucket under load.

Resolution

This is a known issue affecting vRealize Log Insight 4.0.0.
 
Currently, there is no resolution.
 
To work around the issue:


  1. Make a note of the bucket ID GUID that can be found in the log messages described in the symptoms section.  The GUID looks similar to: 6a884d03-ec8e-44d9-8c36-ebfb283ebd6c

    For example, use 6a884d03-ec8e-44d9-8c36-ebfb283ebd6c to represent the path with sufficient space for the remainder of the instructions.

  2. Log in to the log insight node that is reporting this error using SSH using the root credentials.
  3. Shut down log insight service by running the command:

    service loginsight stop

  4. Verify that the service is not started by running the command: 

    service loginsight status

  5. Identify a location on the appliance with sufficient space for the bucket file.  

    For example, use /tmp to represent the path with sufficient space for the remainder of the instructions.

  6. Create a copy of the bucket file by running the command:

    cp -r /storage/core/loginsight/cidata/store/<Bucket-ID> /tmp/ 

    For example: cp -r /storage/core/loginsight/cidata/store/6a884d03-ec8e-44d9-8c36-ebfb283ebd6c /tmp/

  7. Remove the bucket from the bucket index by running the command:

    /usr/lib/loginsight/application/sbin/bucket-index <Bucket-ID> delete

    For example: /usr/lib/loginsight/application/sbin/bucket-index 6a884d03-ec8e-44d9-8c36-ebfb283ebd6c delete

  8. Re-add the bucket to the bucker index by running a command:

    /usr/lib/loginsight/application/sbin/bucket-index add /tmp/<Bucket-ID>

    For example: /usr/lib/loginsight/application/sbin/bucket-index add /tmp/6a884d03-ec8e-44d9-8c36-ebfb283ebd6c

  9. Start up the loginsight service by running the command:

     service loginsight start

Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 0 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 0 Ratings
Actions
KB: