• You trigger VMware Aria Suite Lifecycle 8.x upgrade and after running through a few of the initial steps, the upgrade progress in UI seems to be frozen. If left alone, the UI remains at this stage for hours.
• Similar errors to the below are found in the /opt/vmware/var/log/vami/vami.log

12/09/2022 09:39:20 [INFO] Using update pre-install script
12/09/2022 09:39:20 [ERROR] Unable to install updates because package files are missing

The VMware Aria Suite Lifecycle 8.x upgrade process fails to capture the required binary files from the upgrade source. Both online and CD-ROM upgrade source methods are impacted.

Prerequisites

• Please take simultaneous non-memory snapshots of the virtual appliance(s).
• You have access to root user and password.
• You have SSH or console access to the virtual appliance.
• Download the upgrade ISO from VMware Customer Connect.

Troubleshooting Procedure

1. Revert the appliance to its pre-upgrade snapshot.
2. Enable the debug mode for the vami log by running the following command 

   sed -i 's/info/debug/g' /opt/vmware/etc/vami/vami.xml

3. Tail the vami.log using the command below, then re-run the upgrade (like it was done initially when the failure first occurred) 

   tail -f /opt/vmware/var/log/vami/vami.log

4. In debug mode, the missing packages will be listed. Record this information for use in later steps. See below for an example of a missing package

   14/09/2022 10:42:34 [DEBUG] Stat file, file missing. filename=/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/audit-2.8.5-17.ph3.x86_64.rpm

Workaround

1. As a first step check for the existence of a stale upgrade status file.

   vami.log will show below

   27/10/2022 11:17:15 [DEBUG] Lock file 1
   27/10/2022 11:17:15 [DEBUG] status file /opt/vmware/var/lib/vami/update/data/job/2/status
   27/10/2022 11:17:15 [DEBUG] cli failed
   27/10/2022 11:17:15 [DEBUG] cache path is/opt/vmware/var/lib/vami/update/data
   27/10/2022 11:17:15 [DEBUG] No next version exists
   27/10/2022 11:17:15 [DEBUG] No next version exists
   27/10/2022 11:17:15 [DEBUG] Setting download cache location: /opt/vmware/var/lib/vami/update/data
   27/10/2022 11:17:15 [DEBUG] Disabling old download cache location:
   27/10/2022 11:17:15 [DEBUG] Done setting download cache location
   27/10/2022 11:17:15 [DEBUG] skipClearDownloadCache =
   27/10/2022 11:17:15 [DEBUG] Clear Download Cache. path=/opt/vmware/var/lib/vami/update/data

   and upgrade again fails with below error:
   
   12/09/2022 09:39:20 [ERROR] Unable to install updates because package files are missing
   
   Take the erroneous job ID from above snippet and go to that folder and check contents of status file.

   less /opt/vmware/var/lib/vami/update/data/job/2/status
   

   If status value is "2", delete the folder (in this case, folder 2 so /opt/vmware/var/lib/vami/update/data/job/2/) and retry upgrade. The job ID may be different.

   If there is no stale upgrade flag set, proceed with steps below to manually copy packages.

2. Download VMware Aria Suite Lifecycle 8.x upgrade ISO from VMware customer connect portal.
3. Mount the ISO in local system.
4. Revert the VMware Aria Suite Lifecycle 8.x snapshot to pre-upgrade state.
5. In VMware Aria Suite Lifecycle 8.x appliance, create a directory 'packages' under the /data directory. 

   mkdir /data/packages

6. Enable the debug mode for the vami log by running the following command 

   sed -i 's/info/debug/g' /opt/vmware/etc/vami/vami.xml

7. Locate the missing packages (step 4 of debugging section) from the already mounted VMware Aria Suite Lifecycle 8.x upgrade ISO        
8. Copy the missing package to '/data/packages' directory inside VMware Aria Suite Lifecycle 8.x VM from the ISO.
    Note: Use FTP tool to move the file from the mounted ISO in local system to VMware Aria Suite Lifecycle 8.x VM.
9.  Take a new snapshot of VMware Aria Suite Lifecycle 8.x at this stage (even if the other clean pre-upgrade snapshot exists, taking a snapshot here may save time). 
10. Open one SSH session (let us call this ssh-session-1 for reference in this document) to vRSLCM appliance and keep the following command ready (do not execute because it will fail if executed before the upgrade starts)  

    cp -a /data/packages/. /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/

11. Launch another SSH session (let us call this session-2 for reference in this document) to same vRSLCM appliance and tail the vami log. 

    tail -f /opt/vmware/var/log/vami/vami.log

12. Start the VMware Aria Suite Lifecycle 8.x upgrade from the VMware Aria Suite Lifecycle 8.x UI.
13. In the ssh-session-1, keep trying on to execute the command (indicated in step 10). Note: Since it is difficult to execute the command exactly at the correct moment, it is better to attempt the command as soon as the upgrade is started and we need to keep on trying it (execute it once in the ssh-session-1 and then in the same session press up-arrow and press enter for a quick retry) until it executes without error or fails with a different package missing error. 
14. If the command is run properly, then VMware Aria Suite Lifecycle 8.x upgrade shall go through and it will complete successfully. In case the upgrade fails again due to same reason, we need to revert back to the snapshot taken in step (8) and retry steps (9) to (12).
15. Once the VMware Aria Suite Lifecycle 8.x upgrade completes, disable the debug mode for vami log 

    sed -i 's/debug/info/g' /opt/vmware/etc/vami/vami.xml