Upgrade to vRealize Operations Manager 8.0 fails due to low space on /dev/sdc
search cancel

Upgrade to vRealize Operations Manager 8.0 fails due to low space on /dev/sdc

book

Article ID: 318677

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
You may experience one or more of the following symptoms.
  • The vRealize Operations Manager Admin UI shows the upgrade stuck on step 4 or 9 for an extended period of time.
  • The vcops-suite-vm-photon-pack-pre-installation.log file shows errors similar to:
/bin/cp: writing '/photon-live/vRealize-Operations-Manager-Appliance-8.0.0.14857692-cloud-components.iso': No space left on device
/bin/cp: failed to extend '/photon-live/vRealize-Operations-Manager-Appliance-8.0.0.14857692-cloud-components.iso': No space left on device
  • The /var/log/vmware/vcops/ vcops_upgrade_date.log file shows errors similar to:
Mon Dec 2 14:46:54 CET 2019 ERROR pre-install.sh: Copying /storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-80014857694/extracted/live-install/vRealize-Operations-Manager-Appliance-8.0.0.14857692-cloud-components.iso to /photon-live
"
2019-12-02 14:46:54,514 - b2b - DEBUG - updateCoordinator - runScript - stderr: ""
2019-12-02 14:46:54,515 - b2b - DEBUG - updateCoordinator - runScript - exit code: 2
2019-12-02 14:46:54,515 - b2b - ERROR - updateCoordinator - runScript - Script command: "/bin/sh pre-install.sh" failed with exit code: 2
  • The output of du -sh /usr/lib/vmware-vcops/user/plugins/inbound/agent_adapter/work/bundles/ or du -sh /usr/lib/vmware-vcops/user/plugins/inbound/agent_adapter/conf/plugins/agent_plugins/bundles shows the directory size as 1GB or more.
  • The output of lsblk shows the size of sdc under 4GBs.


Environment

VMware vRealize Operations 8.x

Cause

To upgrade from SLES to Photon, the pre-install stores files in the /dev/sdc disk, which has a size of 4GB.
The contents of the /usr/lib/vmware-vcops/user/plugins/inbound directory are copied to /dev/sdc during the backup step.
If the cluster is running the Endpoint Operations adapter, the agent bundles could cause a large inbound directory, which can fill up /dev/sdc.

Resolution

To resolve this issue, remove the Endpoint Operations agent bundles and ensure /dev/sdc is at least 4GBs in size.
Note: If the output of du -sh /usr/lib/vmware-vcops/user/plugins/inbound/agent_adapter/work/bundles/ or du -sh /usr/lib/vmware-vcops/user/plugins/inbound/agent_adapter/conf/plugins/agent_plugins/bundles shows the directory size as 1GB or more, follow the steps under Remove Endpoint Operations Agent Bundles.
If the output of lsblk shows the size of sdc under 4GBs, follow the steps under Resize /dev/sdc.
 

Remove Endpoint Operations Agent Bundles

  1. Revert to the snapshots taken before the upgrade.
  2. Log into the vRealize Operations Manager Primary node as root via SSH or Console.
  3. Run the following commands to delete the Endpoint Operations agents from the inbound directory:
rm -r /usr/lib/vmware-vcops/user/plugins/inbound/agent_adapter/conf/plugins/agent_plugins/bundles
rm -r /usr/lib/vmware-vcops/user/plugins/inbound/agent_adapter/work/bundles/
 
  1. Repeat steps 2 and 3 on all other vRealize Operations Manager nodes, including Remote Collectors.
  2. Take a snapshot of the vRealize Operations Manager nodes following How to take a Snapshot of vRealize Operations Manager.
  3. Start the vRealize Operations Manager 8.0 Upgrade.

 

Resize /dev/sdc

  1. Revert to the snapshots taken before the upgrade.
Note: This step can be skipped if the Remove Endpoint Operations Agent Bundles section was already followed.
  1. Log into the vRealize Operations Manager Admin UI as the local admin user.
  2. Click Take Offline under Cluster Status.
Note: Wait for Cluster Status to show as Offline.
  1. Log into the vSphere web client as an administrative user.
  2. Right-Click the vRealize Operations Manager node and choose Power > Shut Down Guest OS.
  3. Right-Click the vRealize Operations Manager node and choose Edit Settings.
  4. Set the size of Hard disk 3 to at least 4GBs.
Notes:
  • In some cases when the VM has been moved or copied, /dev/sdc may become Hard disk 1 or Hard disk 2.  In this case, assume the Hard disk with the lowest size is /dev/sdc.
  • You cannot resize the disk if there are snapshots on the VM.  To resize the disk, delete the snapshots first.
  1. Click OK.
  2. Right-Click the vRealize Operations Manager node and choose Power > Power On.
  3. Repeat steps 5-9 on all other applicable vRealize Operations Manager nodes in the cluster.
  4. Take a snapshot of the vRealize Operations Manager nodes following How to take a Snapshot of vRealize Operations Manager.
  5. Log into the vRealize Operations Manager Admin UI as the local admin user.
  6. Click Bring Online under Cluster Status.
Note: Wait for Cluster Status to show as Online.


Additional Information

Also see, Upgrade to vRealize Operations Manager 8.0 fails due to low space on /dev/sda.

Powered by Wolken Software