@ NSX Manager : /var/log/upgrade-coordinator/upgrade-coordinator.log

Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host 6d1d1810-8884-4f4d-83bc-54ec5a86df3a with error : [LiveInstallationError] Error in running ['/etc/init.d/nsx-datapath', 'start', 'upgrade']: Return code: 1 Output: ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. ioctl failed: No such file or directory ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath", line 1267, in <module> UnloadKernelModules() File "/etc/init.d/nsx-datapath", line 1083, in UnloadKernelModules unloadModule(modName, modName in ['nsxt-vsip']) File "/etc/init.d/nsx-datapath", line 1068, in unloadModule (moduleName, out.decode())) KernelModulesException: Failed to unload module nsxt-switch-security: vmkload_mod: Cannot remove module nsxt-switch-security: Consumed resource count of module is not zero It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details..

@ESXi host : /var/log/vmkernel.log

2019-02-21T11:33:05.659Z cpu13:2399604)Mod: 5059: Unloading module <nsxt-switch-security> ...
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: cleanup_module:386: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Starting Cleanup
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSecRtmUnsubscribe:331: [nsx@6876 comp="nsx-esx" subcomp="swsec"]SwSec Unsubscribing to RTM Event
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSec_VMotionUnregister:210: [nsx@6876 comp="nsx-esx" subcomp="swsec"]SwSec Unegistering VMotion CBs
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSec_PropCleanup:384: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Unregistered swsec property handler com.vmware.vswitch.port.swsec.security
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSec_PropCleanup:385: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Unregistered swsec property handler com.vmware.vswitch.port.swsec.discovery
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSec_PropCleanup:386: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Unregistered swsec property handler com.vmware.vswitch.port.swsec.enabled
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSecGenericCleanup:183: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Starting SwSecGenericCleanup
2019-02-21T11:33:05.659Z cpu1:2116926)nsxt-switch-security: SwSecAgingWorldCheckExit:87: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Exiting Aging World
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSecAgingWorldDestroy:411: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Error destroying PriQ (0xbad0004): Busy
2019-02-21T11:33:05.659Z cpu13:2399604)nsxt-switch-security: SwSecGenericCleanup:187: [nsx@6876 comp="nsx-esx" subcomp="swsec"]Error destroying Aging World
2019-02-21T11:33:05.659Z cpu13:2399604)WARNING: Heap: 2734: Non-empty heap (nsxt-switch-security) being destroyed (avail is 110240, should be 192336).
2019-02-21T11:33:05.659Z cpu13:2399604)ALERT: Mod: 5212: Failed to unload module nsxt-switch-security, since its consumed resource count is 1. Waiting...
2019-02-21T11:33:10.677Z cpu13:2399604)ALERT: Mod: 5241: Failed to unload module nsxt-switch-security, since its consumed resource count is 1. Giving up.

The host VIB update fails since switch security module fails to unload. The issue occurs only if IP discovery feature is enabled in switching profile and performing an in-place upgrade from NSX-T 2.3.1 to NSX-T 2.4.0 with host running ESXi-6.7EP06 (build 11675023)

This issue only occurs when upgrading from NSX-T 2.3.1 release to NSX-T 2.4.0. Future upgrades from NSX-T 2.4.0 to later releases will not experience the issue.