smx-provider crashes due to memory allocation issues in ESXi 6.x/7.x on HPE hosts

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

In vSphere Client you see the following event:

“An application (/bin/sfcbd) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /var/core/sfcb-smx-zdump.000.”.

when reviewing the vmkernel.log, located in /var/run/log/, you find the following entries:

2021-11-08T08:57:38.043Z cpu31:2251208)MemSchedAdmit: 478: UserWorld 'sfcb-smx' with cmdline '/bin/sfcbd'
2021-11-08T08:57:38.043Z cpu31:2251208)MemSchedAdmit: 489: uw.2251206 (655346) extraMin/extraFromParent: 329/329, smx (12096) childEmin/eMinLimit: 17739/17920
2021-11-08T08:57:38.043Z cpu31:2251208)MemSchedAdmit: 471: Admission failure in path: smx/sfcb-ProviderMa.2251206/uw.2251206
2021-11-08T08:57:38.043Z cpu31:2251208)MemSchedAdmit: 478: UserWorld 'sfcb-smx' with cmdline '/bin/sfcbd'
2021-11-08T08:57:38.043Z cpu31:2251208)MemSchedAdmit: 489: uw.2251206 (655346) extraMin/extraFromParent: 329/329, smx (12096) childEmin/eMinLimit: 17739/17920
2021-11-08T08:57:38.043Z cpu31:2251208)User: 3173: sfcb-smx: wantCoreDump:sfcb-smx signal:6 exitCode:0 coredump:enabled
2021-11-08T08:57:38.146Z cpu31:2251208)UserDump: 3110: sfcb-smx: Dumping cartel 2251206 (from world 2251208) to file /var/core/sfcb-smx-zdump.000 ...

In the syslog.log file you see errors such as:

sfcb-vmware_base[2110110]: tool_mm_realloc_or_die: memory re-allocation failed(orig=364000 new=364800 msg=Cannot allocate memory, aborting
sfcb-vmw_ipmi[2291550]: tool_mm_realloc_or_die: memory re-allocation failed(orig=909200 new=909600 msg=Cannot allocate memory, aborting

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7

Cause

This is being caused when the HPE's smx-provider runs out of memory. The smx-provider is the WBEM provider for the HPE SmartArray controller. Similar to most of the daemons on an ESXi host, this provider runs in its own specific resource pool and can only allocate as much memory as the pool configuration allows it to.

A service which does not have its own specific resource pool configuration, does get a default memory limit of 70 Mbyte. Depending on the service, this can be insufficient.

Attempts to allocate memory above the configured limit will lead to the service crashing, leading to the events mentioned above.

Resolution

Please note that Hewlett Packard Enterprise have decided to no longer maintain the smx-provider VIB. The official recommendation to solve this issue is therefore to uninstall the VIB.
Be aware that uninstalling the SMX provider VIB requires a reboot of the ESXi host, therefore a maintenance window should be planned.

To uninstall the SMX provider

connect to the ESXi host per SSH
depending on the ESXi version, run the following command to uninstall the VIB:
1. for ESXi 6.5 or 6.7:
```
# esxcli software vib remove -n smx-provider
```
2. For ESXi 7.x:
```
# esxcli software component remove -n smxProvider
```

Note: in some cases, the esxcli software component remove command can fail on ESXi 7.x. If that happens, please use the esxcli software vib remove command on ESXi 7.x similar to ESXi 6.x

To finalize the uninstallation, reboot the ESXi host

NOTE: In VMware vSphere ESXi7.0U3i version it sets the default size of vmware_base group to 120Mb. For more details refer to VMware vSphere ESXi7.0U3i release notes.

Additional Information

See https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00117054en_us for more details

Impact/Risks:
The smx-provider serves as a data provider for health information from the HPE SmartArray controller. Removing the VIB might result in certain health information for the controller not being visible in the vSphere Client hardware health tab for the ESXi hosts.