Knowledge Base
An ESX host fails to boot after increasing the Service Console or system memory (1016239)
Symptoms
After increasing the amount of physical RAM or Service Console memory, you may experience these symptoms:
- An ESX fails to boot
- You may see a purple diagnostics screen that contains an error similar to:
NOT_IMPLEMENTED /build/mts/release/bora-84374/bora/vmkernel/main/util.c:83
Resolution
Cause
This issue occurs when the ESX VMkernel cannot allocate sufficient physical memory below 4GB during boot.
The ESX host requires memory below 4GB (called Low Memory) for the Service Console, drivers, and other uses. The Service Console is configured to use between 272-800MB of the lowest physical memory. The ESX VMkernel requires some low memory to keep trace of high memory pages. The exact amount of memory Reserved can be seen with cat /proc/vmware/mem.
Example Reservations are as follows:
- With 256GB of physical memory installed in a server, the ESX VMkernel reserves 1289MB of Low Memory.
- With 128GB of physical memory installed in a server, the ESX VMkernel reserves 644MB of Low Memory.
The relationship between Physical Low and High Memory is as follows:
- Maximum Console OS memory = Total Low Memory - VMkernel Reserved Low Memory - ~100MB Drivers
The total amount of low memory available is typically 4GB. However, the System BIOS can reserve some of this memory, so less than 4GB is available to the operating system. At a high level, the memory map looks like the figures below:
The following lines within the /var/log/messages log file represent the memory map as presented by the BIOS:
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 0000000000098400 (usable)
BIOS-e820: 0000000000098400 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007ff42980 (usable)
BIOS-e820: 000000007ff42980 - 000000007ff76300 (ACPI data)
BIOS-e820: 000000007ff76300 - 0000000080000000 (reserved)
BIOS-e820: 00000000c0000000 - 00000000d0000000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000004040000000 (usable)
Note: Memory ranges from 000000007ff76300 (2047MB) to 0000000100000000 (4096MB) are reserved by the BIOS and are not usable by the OS.
The following lines from /var/log/messages record the VMkernel interpretations of the RAM map:
user-defined physical RAM map:user: 0000000000000000 - 0000000000098400 (usable)
user: 0000000000098400 - 00000000000a0000 (reserved)
user: 00000000000e0000 - 0000000000100000 (reserved)
user: 0000000000100000 - 0000000011000000 (usable)
VMNIX: linux MA=[0x0,0x11000000)
VMNIX: vmkernel MPN=[0x11000,0x7ff41]
VMNIX: vmkernel MPN=[0x100000,0x403ffff]
Note: The line containing linux MA reflect the Machine Address memory ranges allocated (in bytes) to the service console. The Service Console is using the lower 272MB. The lines containing vmkernel MPN reflect the Machine Page Numbers (4k blocks) allocated for the VMkernel. The VMkernel is using memory ranges 272MB - 2047MB and 4GB - 256GB.
The table below contains several combinations of Physical Memory and Service Console Memory assignment, along with their impact on the amount Unreserved Low Memory available when 2GB of low memory is reserved by the BIOS. The combination of 256GB of Physical Memory and 800MB of Service Console Memory with only 2GB of Low Memory leaves the server with only 7MB of Unreserved Low Memory, insufficient for booting. In this case the ESX VMkernel startup stops with the purple diagnostics screen.
Impact of Memory Allocations
| Physical Memory | Physical Low Memory | Reserved Low Memory | Service Console Memory | Unreserved Low Memory |
| 256 GB | 4096 MB | 1289 MB | 272 MB | 2535 MB |
| 256 GB | 4096 MB | 1289 MB | 800 MB | 2007 MB |
| 256 GB | 2048 MB | 1289 MB | 272 MB | 487 MB |
| 256 GB | 2048 MB | 1289 MB | 800 MB | 7 MB |
| 128 GB | 4096 MB | 644 MB | 272 MB | 3180 MB |
| 128 GB | 4096 MB | 644 MB | 800 MB | 2652 MB |
| 128 GB | 2048 MB | 644 MB | 272 MB | 1132 MB |
| 128 GB | 2048 MB | 644 MB | 800 MB | 604 MB |
Resolution
For servers affected by this limitation, it is recommended that the Service Console memory be reduced to between 500MB - 650MB, or that the Physical System Memory be reduced by 16GB - 32GB. Make one of the following changes to increase the amount of Low Memory available below 4GB:
- Reduce the amount of Physical Memory in the server.
- Reduce the amount of Physical Memory reserved by the BIOS.
Note: The BIOS memory reservation may or may not be configurable in a particular server. Consult your hardware vendor for more information.
- Reduce the amount of Service Console Memory assigned by using the following steps:
-
- Restart the ESX host.
- When presented the boot menu, ensure you have the default boot option highlighted.
- Press the e key to edit the default boot option. A new list appears.
- Use the arrow keys to highlight the line which starts with kernel /vmlinuz.
- Press the e key to edit the line in the boot configuration. A prompt with the entire line will appear.
- Use the arrow key to move over the portion of the line which has a parameter similar to the following:
mem=800M
- Change the value to something between 500 and 650 megabytes. The format is mem=<value in megabytes>M.
Note: The exact threshold may vary depending on the hardware configuration. To determine the exact value, engage your hardware vendor or test different values.
- Press enter to accept the changes.
- Press the b key to boot the host using the configuration.
Note: The configuration change is not permanent. To make the change permanent, proceed to step 10.
- Wait for the host to boot.
- When the host has booted, open the vSphere Client and connect to the host.
- Under the configuration tab, change the memory properties of the ESX host to ensure that the RAM assignment has been reduced. For more information on adjusting the RAM assignment using the vSphere Client, see Increasing the amount of RAM assigned to the ESX Server Service Console (1003501).
Request a Product Feature
To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.
|
|
