VMware vRealize Network Insight (vRNI) upgrade fails with Insufficient disk space toast message displayed in vRNI GUI
Article ID: 314387
Updated On:
Products
VMware Aria Operations for Networks
Issue/Introduction
This article gives information on the issues seen with vRNI On-Premises upgrades and how to workaround the issue.
Symptoms:
1. When trying to update VMware vRealize Network Insight via any method (Online, offline or vRealize Suite Lifecycle Manager), upgrade fails with Insufficient disk space toast message displayed in vRNI GUI
Refer to Screenshot below:
2. Running df -h command on vRNI platform appliance shows "/" with 11 G available.
Refer output below (highlighted in Yellow)
support@vrni-platform-release:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 16M 6.3G 1% /run
/dev/mapper/vg-root 29G 17G 11G 63% /
tmpfs 32G 16K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 1.9G 122M 1.6G 8% /boot
none 32G 0 32G 0% /run/shm
/dev/mapper/vg-tmp 25G 1.7G 22G 8% /tmp
/dev/mapper/vg-home 30G 8.7G 20G 32% /home
/dev/mapper/vg-var 852G 14G 794G 2% /var
/dev/mapper/vg-var+log 49G 18G 29G 38% /var/log
/dev/mapper/vg-var+log+audit 20G 62M 19G 1% /var/log/audit
tmpfs 6.3G 0 6.3G 0% /run/user/998
tmpfs 6.3G 0 6.3G 0% /run/user/115
tmpfs 6.3G 0 6.3G 0% /run/user/119
tmpfs 6.3G 0 6.3G 0% /run/user/117
tmpfs 6.3G 0 6.3G 0% /run/user/120
tmpfs 6.3G 0 6.3G 0% /run/user/1001
support@vrni-platform-release:~$
Above same command can be run on vRNI deployments running platform in cluster but only on platform node1.
Symptoms:
1. When trying to update VMware vRealize Network Insight via any method (Online, offline or vRealize Suite Lifecycle Manager), upgrade fails with Insufficient disk space toast message displayed in vRNI GUI
Refer to Screenshot below:
2. Running df -h command on vRNI platform appliance shows "/" with 11 G available.
Refer output below (highlighted in Yellow)
support@vrni-platform-release:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 16M 6.3G 1% /run
/dev/mapper/vg-root 29G 17G 11G 63% /
tmpfs 32G 16K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 1.9G 122M 1.6G 8% /boot
none 32G 0 32G 0% /run/shm
/dev/mapper/vg-tmp 25G 1.7G 22G 8% /tmp
/dev/mapper/vg-home 30G 8.7G 20G 32% /home
/dev/mapper/vg-var 852G 14G 794G 2% /var
/dev/mapper/vg-var+log 49G 18G 29G 38% /var/log
/dev/mapper/vg-var+log+audit 20G 62M 19G 1% /var/log/audit
tmpfs 6.3G 0 6.3G 0% /run/user/998
tmpfs 6.3G 0 6.3G 0% /run/user/115
tmpfs 6.3G 0 6.3G 0% /run/user/119
tmpfs 6.3G 0 6.3G 0% /run/user/117
tmpfs 6.3G 0 6.3G 0% /run/user/120
tmpfs 6.3G 0 6.3G 0% /run/user/1001
support@vrni-platform-release:~$
Above same command can be run on vRNI deployments running platform in cluster but only on platform node1.
Environment
VMware vRealize Network Insight 6.x
VMware vRealize Network Insight 6.5.0
VMware vRealize Network Insight 6.5.1
VMware vRealize Network Insight 6.8.0
VMware vRealize Network Insight 6.6.0
VMware vRealize Network Insight 6.7.0
VMware vRealize Network Insight 6.5.0
VMware vRealize Network Insight 6.5.1
VMware vRealize Network Insight 6.8.0
VMware vRealize Network Insight 6.6.0
VMware vRealize Network Insight 6.7.0
Cause
vRNI upgrade fails because the old bundle did not get auto cleaned up during the previous upgrades hence consuming space on "/" (root) partition.
Resolution
vRNI Engineering is aware of this issue and working on a fix for this.
Once fix is available this Knowledge base article will be updated.
Workaround:
To workaround this, follow below steps to perform a manual clean up of the older extracted bundles:
1. Take a SSH/Putty session to vRNI platform appliance. If you are running vRNI deployment with platform cluster, then take SSH/Putty session on platform Node1
2. Login with username support and run below commands:
support@vrni-platform-release:~$ ub
ubuntu@vrni-platform-release::~$ ls -lrth /usr/share/nginx/upgradebundle
total 8.0K
drwxr-xr-x 3 www-data www-data 4.0K Nov 2 06:13 6.5.1.1645037705
drwxr-xr-x 3 www-data www-data 4.0K Nov 2 06:13 6.6.0.1648892975
drwxr-xr-x 2 www-data www-data 4.0K Nov 2 08:09 6.7.0.1656233033
As seen in output above you should see older extracted bundles in your vRNI appliance.
3. Run below command for manual clean up depending on the older bundle file name you see in output of the command executed in step 2 above.
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/<bundlename>
Example commands:
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.5.1.1645037705/
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.6.0.1648892975/
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.7.0.1656233033/
5. To validate if clean/manual removal of bundle is successful run below command.
ubuntu@vrni-platform-release:~$ ls -lrth /usr/share/nginx/upgradebundle
total 0
6. Now run below command to see that vRNI platform appliance shows "/" with more than 14 G.
Refer output below (highlighted in Green)
ubuntu@vrni-platform-release:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 16M 6.3G 1% /run
/dev/mapper/vg-root 29G 9.7G 18G 36% /
tmpfs 32G 16K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 1.9G 122M 1.6G 8% /boot
none 32G 0 32G 0% /run/shm
/dev/mapper/vg-tmp 25G 1.7G 22G 8% /tmp
/dev/mapper/vg-home 30G 8.7G 20G 32% /home
/dev/mapper/vg-var 852G 14G 794G 2% /var
/dev/mapper/vg-var+log 49G 18G 29G 38% /var/log
/dev/mapper/vg-var+log+audit 20G 64M 19G 1% /var/log/audit
tmpfs 6.3G 0 6.3G 0% /run/user/998
tmpfs 6.3G 0 6.3G 0% /run/user/115
tmpfs 6.3G 0 6.3G 0% /run/user/119
tmpfs 6.3G 0 6.3G 0% /run/user/117
tmpfs 6.3G 0 6.3G 0% /run/user/120
tmpfs 6.3G 0 6.3G 0% /run/user/1001
ubuntu@vrni-platform-release:~$
ubuntu@platform1:~$ exit
logout
support@platform1:~$
Once above steps are executed for manual clean up the vRNI upgrade needs to be kickstarted again and should pass the pre check, toast message on UI would not been seen and upgrade proceeds as normal.
Once fix is available this Knowledge base article will be updated.
Workaround:
To workaround this, follow below steps to perform a manual clean up of the older extracted bundles:
1. Take a SSH/Putty session to vRNI platform appliance. If you are running vRNI deployment with platform cluster, then take SSH/Putty session on platform Node1
2. Login with username support and run below commands:
support@vrni-platform-release:~$ ub
ubuntu@vrni-platform-release::~$ ls -lrth /usr/share/nginx/upgradebundle
total 8.0K
drwxr-xr-x 3 www-data www-data 4.0K Nov 2 06:13 6.5.1.1645037705
drwxr-xr-x 3 www-data www-data 4.0K Nov 2 06:13 6.6.0.1648892975
drwxr-xr-x 2 www-data www-data 4.0K Nov 2 08:09 6.7.0.1656233033
As seen in output above you should see older extracted bundles in your vRNI appliance.
3. Run below command for manual clean up depending on the older bundle file name you see in output of the command executed in step 2 above.
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/<bundlename>
Example commands:
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.5.1.1645037705/
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.6.0.1648892975/
ubuntu@vrni-platform-release:~$ sudo rm -rf /usr/share/nginx/upgradebundle/6.7.0.1656233033/
5. To validate if clean/manual removal of bundle is successful run below command.
ubuntu@vrni-platform-release:~$ ls -lrth /usr/share/nginx/upgradebundle
total 0
6. Now run below command to see that vRNI platform appliance shows "/" with more than 14 G.
Refer output below (highlighted in Green)
ubuntu@vrni-platform-release:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 16M 6.3G 1% /run
/dev/mapper/vg-root 29G 9.7G 18G 36% /
tmpfs 32G 16K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 1.9G 122M 1.6G 8% /boot
none 32G 0 32G 0% /run/shm
/dev/mapper/vg-tmp 25G 1.7G 22G 8% /tmp
/dev/mapper/vg-home 30G 8.7G 20G 32% /home
/dev/mapper/vg-var 852G 14G 794G 2% /var
/dev/mapper/vg-var+log 49G 18G 29G 38% /var/log
/dev/mapper/vg-var+log+audit 20G 64M 19G 1% /var/log/audit
tmpfs 6.3G 0 6.3G 0% /run/user/998
tmpfs 6.3G 0 6.3G 0% /run/user/115
tmpfs 6.3G 0 6.3G 0% /run/user/119
tmpfs 6.3G 0 6.3G 0% /run/user/117
tmpfs 6.3G 0 6.3G 0% /run/user/120
tmpfs 6.3G 0 6.3G 0% /run/user/1001
ubuntu@vrni-platform-release:~$
ubuntu@platform1:~$ exit
logout
support@platform1:~$
Once above steps are executed for manual clean up the vRNI upgrade needs to be kickstarted again and should pass the pre check, toast message on UI would not been seen and upgrade proceeds as normal.
Additional Information
Refer to vRNI Documentation on the required disc space on root partition "/"
https://docs.vmware.com/en/VMware-vRealize-Network-Insight/6.6/com.vmware.vrni.install.doc/GUID-EE13DF2F-F0D0-4834-8299-5F6344DEEC9E.html
Impact/Risks:
vRNI upgrades will fail at pre-check.
Impacted vRNI On-Premises versions are 6.5.1, 6.6.0, 6.7.0 and 6.8.0
This issue is resolved in vRNI 6.9.0.
https://docs.vmware.com/en/VMware-vRealize-Network-Insight/6.6/com.vmware.vrni.install.doc/GUID-EE13DF2F-F0D0-4834-8299-5F6344DEEC9E.html
Impact/Risks:
vRNI upgrades will fail at pre-check.
Impacted vRNI On-Premises versions are 6.5.1, 6.6.0, 6.7.0 and 6.8.0
This issue is resolved in vRNI 6.9.0.
Feedback
Yes
No
Powered by
