VMware Cloud Foundation update fails during VRM update when Log Insight was manually upgraded
search cancel

VMware Cloud Foundation update fails during VRM update when Log Insight was manually upgraded

book

Article ID: 344828

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:
  • Log Insight was manually upgraded outside of VMware Cloud Foundation.
  • VMware Cloud Foundation upgrade fails during LCM upgrade.
  • You see similar entries in the /home/vrack/vrm/logs/evosddc-upgrade.log on the SDDC Manager Controller VM:
2018-11-29 23:01:25 [pool-22-thread-1] DEBUG c.v.v.v.u.u.vrli.VRLINtpConfigurator - Request body: {"timeReference": "NTP_SERVER","ntpServers": ["10.x.x.x","10.x.x.x"]}
2018-11-29 23:01:26 [pool-22-thread-1] ERROR c.v.v.v.u.u.vrli.VRLINtpConfigurator - POST request to vRLI loginsight-cluster.vcf.vmware.corp failed. Response code 404
2018-11-29 23:01:26 [pool-22-thread-1] ERROR c.v.v.vrm.upgrade.task.UpdateNtps - Error: Failed post request in vRLI {}
com.vmware.vrack.vrm.upgrade.NtpConfigurationException: Failed post request in vRLI
        at com.vmware.vrack.vrm.upgrade.utils.vrli.VRLINtpConfigurator.postRequest(VRLINtpConfigurator.java:104)
        at com.vmware.vrack.vrm.upgrade.task.UpdateNtps.queryAndUpdateOverApi(UpdateNtps.java:440)
        at com.vmware.vrack.vrm.upgrade.task.UpdateNtps.updateNtpsForVRLI(UpdateNtps.java:279)
        at com.vmware.vrack.vrm.upgrade.task.UpdateNtps.executeUpgradeTask(UpdateNtps.java:194)
        at com.vmware.vrack.vrm.upgrade.task.core.UpgradeTask$1.call(UpgradeTask.java:82)
        at com.vmware.vrack.vrm.upgrade.task.core.UpgradeTask$1.call(UpgradeTask.java:76)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-11-29 23:01:26 [pool-22-thread-1] ERROR c.v.v.vrm.upgrade.task.UpdateNtps - Failed to set Ntps
com.vmware.vrack.vrm.upgrade.UpgradeException: com.vmware.vrack.vrm.upgrade.NtpConfigurationException: Failed post request in vRLI

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware Cloud Foundation 2.3.x
VMware Cloud Foundation 2.2.x

Cause

Due to API changes between Log Insight versions, Cloud Foundation is not able to update NTP settings within Log Insight.

Resolution

This is a known issue affecting VMware Cloud Foundation. There is currently no resolution.

Workaround:
  1. Take a snapshot of the SDDC Manager Controller VM.
  2. SSH to the SDDC Manager Controller VM.
  3. Issue the following command to obtain the relevant Log Insight information from the cassandra database:

/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "select * from vrmkeyspace.loginsight"

Note: You will see output similar to the following:

 id                                   | adminemail | createddate              | dalversion | enhancedloggingenabled | externalloginsightserveraddress | externalsyslogserveraddress | hostname                       | ipaddress     | modifieddate             | name            | scope  | state | vipaddress    | vipfqdn                           | vmsize
--------------------------------------+------------+--------------------------+------------+------------------------+---------------------------------+-----------------------------+--------------------------------+---------------+--------------------------+-----------------+--------+-------+---------------+-----------------------------------+--------
 87748610-6a87-11e8-a2d7-55408a7195d1 |       null | 2018-06-07 22:56:33+0000 |          1 |                  False |                            null |                        null | loginsight-vm-1.vcf.corp.local | 192.x.x.x | 2018-06-07 22:56:33+0000 | loginsight-vm-1 | public |  null | 192.x.x.x | loginsight-cluster.vcf.corp.local | xsmall
 8774ad21-6a87-11e8-a2d7-55408a7195d1 |       null | 2018-06-07 22:56:33+0000 |          1 |                   null |                            null |                        null | loginsight-vm-3.vcf.corp.local | 192.x.x.x | 2018-06-07 22:56:33+0000 | loginsight-vm-3 | public |  null |          null |                              null | xsmall
 8774ad20-6a87-11e8-a2d7-55408a7195d1 |       null | 2018-06-07 22:56:33+0000 |          1 |                   null |                            null |                        null | loginsight-vm-2.vcf.corp.local | 192.x.x.x | 2018-06-07 22:56:33+0000 | loginsight-vm-2 | public |  null |          null |                              null | xsmall

Note: Make a note of the id value for each row as it will be used in a later step.

  1. Copy the output and save it somewhere as it will be used to repopulate the database in a later step.
  2. Issue commands similar to the following to remove the Log Insight records from the cassandra database:

/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "delete from loginsight where id=87748610-6a87-11e8-a2d7-55408a7195d1"
/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "delete from loginsight where id=8774ad21-6a87-11e8-a2d7-55408a7195d1"
/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "delete from loginsight where id=8774ad20-6a87-11e8-a2d7-55408a7195d1"

Note: Replace the id values in the previous commands with those noted in Step 3.

  1. Issue the command from Step 3 again to confirm that there are no records present.
  2. Start the vrm upgrade again in the SDDC Manager UI.
Note: This time vrm upgrade will not attempt to update NTP on Log insight as there are no records in the table.
  1. Once the upgrade is successful, issue commands similar to the following to repopulate the loginsight table in the cassandra database with the smae data removed in Step 5:

/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "insert into loginsight(id,createddate,dalversion,enhancedloggingenabled,hostname, ipaddress,modifieddate,name,scope,vipaddress,vipfqdn,vmsize) values (87748610-6a87-11e8-a2d7-55408a7195d1, '2018-06-07 22:56:33+0000', '1',True,'loginsight-vm-1.vcf.corp.local', '192.x.x.x', '2018-06-07 22:56:33+0000', 'loginsight-vm-1','public','192.168.16.18','loginsight-cluster.vcf.corp.local','sxmall')"
/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "insert into loginsight(id,createddate,dalversion,enhancedloggingenabled,hostname,ipaddress,modifieddate,name,scope,vmsize) values (8774ad21-6a87-11e8-a2d7-55408a7195d1, '2018-06-07 22:56:33+0000', '1',True,'loginsight-vm-2.vcf.corp.local', '192.x.x.x', '2018-06-07 22:56:33+0000', 'loginsight-vm-1','public','sxmall')"
/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "insert into loginsight(id,createddate,dalversion,enhancedloggingenabled,hostname,ipaddress,modifieddate,name,scope,vmsize) values (87748610-6a87-11e8-a2d7-55408a7195d1, '2018-06-07 22:56:33+0000', '1',True,'loginsight-vm-1.vcf.corp.local', '192.x.x.x', '2018-06-07 22:56:33+0000', 'loginsight-vm-1','public','sxmall')"

Note: Replace the environment-specific values (names, IP addresses, sizes, dates, etc...) with the values saved in Step 4.


Additional Information

To be alerted when this article is updated, click the Subscribe to Article link in the Actions box.

Powered by Wolken Software