Search the VMware Knowledge Base (KB)
View by Article ID

Analytics Service Fails to Start On Multi Node Cluster in vRealize Operations Manager 6.x (2149200)

  • 0 Ratings

Symptoms

When there is more than 1 node in the cluster and the status of 1 or more nodes is Waiting for Analytics in the admin UI, you experience these symptoms:

  • In the /storage/vcops/log/analytics-UUID.log file, you see entries similar to:

    2016-10-10 15:17:03,539 INFO [Analytics Main Thread ] com.vmware.statsplatform.persistence.sqldb.SQLDBHAManagerImpl.waitForOtherNodesToPublishMasterDB - 4 out of total 5 have published its master db knowledge

  • In the  /storage/vcops/log/cassandra/system.log file, you see entries similar to:

    ERROR [WrapperSimpleAppMain] 2016-10-10 15:18:52,294 DatabaseDescriptor.java:141 - Fatal configuration error
    org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml
            at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:120) ~[apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:136) ~[apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:164) [apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.8.jar:2.1.8]        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
            at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
            at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290) [wrapper-3.5.24.jar:3.5.24]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
    Caused by: org.yaml.snakeyaml.reader.ReaderException: special characters are not allowed
            at org.yaml.snakeyaml.reader.StreamReader.checkPrintable(StreamReader.java:93) ~[snakeyaml-1.11.jar:na]
            at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:192) ~[snakeyaml-1.11.jar:na]
            at org.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:60) ~[snakeyaml-1.11.jar:na]
            at org.yaml.snakeyaml.Yaml.load(Yaml.java:412) ~[snakeyaml-1.11.jar:na]
            at org.apache.cassandra.config.YamlConfigurationLoader.logConfig(YamlConfigurationLoader.java:126) ~[apache-cassandra-2.1.8.jar:2.1.8]
            at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:104) ~[apache-cassandra-2.1.8.jar:2.1.8]
            ... 12 common frames omitted

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs because the $VCOPS_BASE/user/conf/cassandra/cassandra.yaml file has incorrect information.

Resolution

To resolve the issue, copy the $VCOPS_BASE/user/conf/cassandra/cassandra.yaml file, from a working node to the problematic node and update the information.

  1. Log in to the problematic vRealize Operations Manager node as root through SSH or Console.
  2. Run this command to rename the cassandra.yaml file:

    mv /usr/lib/vmware-vcops/user/conf/cassandra/cassandra.yaml /usr/lib/vmware-vcops/user/conf/cassandra/cassandra.yaml.bak

  3. Copy the following files from a working node, to the $VCOPS_BASE/user/conf/cassandra/ directory on the problematic node.

    $VCOPS_BASE/user/conf/cassandra/cassandra.yaml
    $VCOPS_BASE/user/conf/cassandra/cqlshrc


    Note: The scp command, or an SCP utility can be used for the REHL and vApp versions of vRealize Operations Manager.

  4. Open the $VCOPS_BASE/user/conf/cassandra/cassandra.yaml file, using a text editor.
  5. Replace the IP Address of the node from where you copied cassandra.yaml file with the IP address of node where you copied to.

    Note: The sed command can be used for the REHL and vApp versions of vRealize Operations Manager instead:

    For example:

    sed -i "s/working_node_address/problematic_node_address/g" /usr/lib/vmware-vcops/user/conf/cassandra/cassandra.yaml

    Note: Replace working_node_address with the address of the node the cassandra.yaml file was copied from, and replace problematic_node_address with the address of the problematic node.

  6. Log in to vRealize Operations Manager admin UI.
  7. Take the cluster Offline.
  8. Bring the cluster Online.

 

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 0 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 0 Ratings
Actions
KB: