I was getting the following message when I would try to start up a VM on an HP DL380.

Heap globalCartel-1 already at its maximum size of 7869288. Cannot expand. Could not start VMX: msg.vmk.status.VMK_NO_MEMORY

The machine had plenty of memory and seemed to operating fine for currently running VMs. When I brought up the direct console on the machine, there was screen full of the following:

/bin/sh: can’t fork

I could not enter any commands other than “exit.” I could log in and out, and that is about it.

In VMWare KB article 2085618, it acknowledged the issue with the HP specific image and the HP AMS driver.

Updated the HP AMS driver per VMWare KB article 2085618.

To determine version running:

esxcli software vib list | grep ams

Old version:

# esxcli software vib list | grep ams
hp-ams 550.10.0.0-18.1198610 Hewlett-Packard PartnerSupported 2014-11-07

Download updated driver from HP:

Then, I used scp to copy the downloaded zip file to my ESXi 5.5 server. I copied it to a hp-ams-esxi5.5-bundle- directory I created. Directory: /vmfs/volumes/vmhost02-datastore1/Software/hp-ams-esxi5.5-bundle-10.0.1-2.

Login to the ESXi host using ssh.

# cd /vmfs/volumes/vmhost02-datastore1/Software/hp-ams-esxi5.5-bundle-10.0.1-2

Unzip the file:

# unzip hp-ams-esxi5.5-bundle-10.0.1-2.zip
Archive: hp-ams-esxi5.5-bundle-10.0.1-2.zip
inflating: index.xml
inflating: vendor-index.xml
inflating: metadata-hp-ams-esxi5.5-bundle-10.0.1-2.zip
inflating: hp-esxi-fc-enablement-550.2.1.8-1198610.vib
inflating: hp-smx-limited-550.
inflating: char-hpilo-550.
inflating: char-hpcru-
inflating: hp-ams-550.10.0.1-07.1198610.vib

Files contained in the zip:

# ls -l
total 19456
-rw-r–r– 1 root root 13590 Apr 29 12:15 char-hpcru-
-rw-r–r– 1 root root 12676 Apr 29 12:15 char-hpilo-550.
-rw-r–r– 1 root root 1271770 Apr 29 12:15 hp-ams-550.10.0.1-07.1198610.vib
-rw-r–r– 1 root root 6829546 Apr 29 12:15 hp-ams-esxi5.5-bundle-10.0.1-2.zip
-rw-r–r– 1 root root 1213780 Apr 29 12:15 hp-esxi-fc-enablement-550.2.1.8-1198610.vib
-rw-r–r– 1 root root 4482274 Apr 29 12:15 hp-smx-limited-550.
-rw——- 1 root root 328 Apr 29 12:15 index.xml
-rw-r–r– 1 root root 8531 Apr 29 12:15 metadata-hp-ams-esxi5.5-bundle-10.0.1-2.zip
-rw——- 1 root root 238 Apr 29 12:15 vendor-index.xml

Shutdown or migrate all the guests on the ESXi server, and then put the server in maintenance mode:

# esxcli system maintenanceMode set –enable on

Update the software:

# esxcli software vib update -v /vmfs/volumes/542c3aae-03b91418-ff2a-a0d3c1f0cdf0/Software/hp-ams-esxi5.5-bundle-10.0.1-2/hp
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: Hewlett-Packard_bootbank_hp-ams_550.10.0.1-07.1198610
VIBs Removed: Hewlett-Packard_bootbank_hp-ams_550.10.0.0-18.1198610
VIBs Skipped:

At this point, you need to reboot the ESXi host.

New version:
# esxcli software vib list | grep ams
hp-ams 550.10.0.1-07.1198610 Hewlett-Packard PartnerSupported 2015-04-29

Take the server out of maintenance mode:

# esxcli system maintenanceMode set –enable off

Migrate or startup guests back to the server.