Agentless Management Service (AMS) version 11.4.x causing a full ams-bbUsg.txt and host not responding
search cancel

Agentless Management Service (AMS) version 11.4.x causing a full ams-bbUsg.txt and host not responding

book

Article ID: 339028

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • An ESXi host upgrade fails due to /tmp being full.
  • An ESXi host is not responding in the vCenter Server inventory.
  • The ams-bbUsg.txt file consumes a large amount of /tmp
  • In the vobd.log, you see similar to:
2020-04-10T05:25:45.458Z: [VisorfsCorrelator] 9661642434308us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
2020-04-10T05:25:45.503Z: [VisorfsCorrelator] 9661642479481us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
2020-04-10T05:25:45.504Z: [VisorfsCorrelator] 9661642480515us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/ams-bbUsg.txt because its ramdisk (tmp) is full.
2020-04-10T05:26:14.662Z: [VisorfsCorrelator] 9661671637154us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /tmp/ams-sysmod.txt because its ramdisk (tmp) is full.
2020-04-10T05:26:14.662Z: [VisorfsCorrelator] 9661816145006us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'tmp' is full.  As a result, the file /tmp/ams-sysmod.txt could not be written.
  • In the vmkernel.log, you see similar to:
2020-04-10T04:57:46.064Z cpu0:246728737)WARNING: VisorFSRam: 206: Cannot extend visorfs file /tmp/bbFile2.txt because its ramdisk (tmp) is full.
2020-04-10T04:57:46.065Z cpu2:2100680)MemSchedAdmit: 470: Admission failure in path: tmp/tmp
2020-04-10T04:57:46.065Z cpu2:2100680)MemSchedAdmit: 477: tmp (2409) extraMin/extraFromParent: 1/1, tmp (2408) childEmin/eMinLimit: 65536/65536
2020-04-10T04:57:46.065Z cpu2:2100680)WARNING: VisorFSRam: 206: Cannot extend visorfs file /tmp/ams-bbUsg.txt because its ramdisk (tmp) is full.
2020-04-10T04:58:24.288Z cpu20:2098113)ScsiDeviceIO: 3068: Cmd(0x45a29fb873c0) 0x1a, CmdSN 0x30e8a1 from world 0 to dev "mpx.vmhba1:C2:T1088:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2020-04-10T04:58:31.712Z cpu6:246732134)MemSchedAdmit: 470: Admission failure in path: tmp/tmp
2020-04-10T04:58:31.712Z cpu6:246732134)MemSchedAdmit: 477: tmp (2409) extraMin/extraFromParent: 1/1, tmp (2408) childEmin/eMinLimit: 65536/65536
  • vMotion of a virtual machine fails
  • You see an error similar to:
General System Error Occurred.
Note: A similar issue can occur if the mili2d.log fills up ramdisk at /var/, see The ramdisk '/var' is full. As a result, the file /var/log/EMU/mili/mili2d.log could not be written. (65193)


Environment

VMware vSphere ESXi 6.7
VMware vSphere 6.5.x

Cause

HPE servers running VMware ESXi version 6.X with Agentless Management Service (AMS) version 11.4.x will enter not responding state because AMS data fills up /tmp.

Resolution

A resolution is available at https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=emr_na-a00073323en_us

Workaround:
To work around this issue, clear the ams-bbUsg.txt file contents:
  1. Connect to affected host through SSH session.
  2. List the files in /tmp using du -sh *
[root@VMwareESXi-2:/tmp] du -sh *
255.6M  ams-bbUsg.txt
4.0K    ams-ent.txt
4.0K    ams-fc.txt
4.0K    ams-fsys.txt
64.0K   ams-hrswrun.txt
4.0K    ams-ip.txt
0       ams-ipv4.txt
  1. Empty the ams-bbUsg.txt file with this command: echo > ams-bbUsg.txt


Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_a38161c3e8674777a8c664e05a#tab-history

Impact/Risks:
  • None of the VC features will be applicable if the host is in not responding state.
  • VM backup will get impacted.
  • Upgrade via VUM/CLI will fail.


Powered by Wolken Software