"practical" memory allocation limit?

"practical" memory allocation limit?

[Laszlo Ersek]

Hi,

I've been curious about it for a long time, so I guess I might as well
ask the question:

Even if OVMF's DXE phase is built for X64, and even if I give the VM
memory above 4GB -- confirmed by the MEMMAP command in the UEFI shell
--, in the OVMF log I see practically all allocations coming from the
32-bit address space.

OVMF produces all its memory resource descriptor HOBs in PEI, all
"tested". (No memory is added in DXE.) So why aren't DXE allocations
served from 64-bit space?

Something seems to be artificially limiting run-of-the-mill pool and
page allocations in X64 OVMF; what is it? (It doesn't work like this in
ArmVirtQemu.)

Thanks!
Laszlo

Re: "practical" memory allocation limit?

[Shi, Steven]

Hi Laszlo,

I happen to have a question about how to correctly get the system memory size on Qemu.

In the QemuInitializeRam() of OvmfPkg\PlatformPei\MemDetect.c, I add debug info as below to trace the GetSystemMemorySizeBelow4gb() and GetSystemMemorySizeAbove4gb() output. But the output results seems not right if my input memory size > 4GB. See my commands and trace outputs in below.



MemDetect.c  Line 602:

  //

  // Determine total memory size available

  //

  LowerMemorySize = GetSystemMemorySizeBelow4gb ();

  UpperMemorySize = GetSystemMemorySizeAbove4gb ();

  DEBUG ((EFI_D_INFO, "LowerMemorySize= 0x%x\n", LowerMemorySize));

  DEBUG ((EFI_D_INFO, "UpperMemorySize= 0x%x\n", UpperMemorySize));



My test commands and trace outputs:

$ /opt/qemu/bin/qemu-system-x86_64 -m 5120 -enable-kvm -hda /home/jshi19/workspace/simics5-project/images/luv-v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial file:serial.log

LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?

UpperMemorySize= 0xC0000000 //3GB, but should not it be 1GB?





$ /opt/qemu/bin/qemu-system-x86_64 -m 6144 -enable-kvm -hda /home/jshi19/workspace/simics5-project/images/luv-v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial file:serial.log

LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?

UpperMemorySize= 0x0                 // 0GB, but should not it be 2GB?









Steven Shi

Intel\SSG\STO\UEFI Firmware



Tel: +86 021-61166522

iNet: 821-6522



> -----Original Message-----

> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of

> Laszlo Ersek

> Sent: Monday, August 28, 2017 4:43 PM

> To: edk2-devel-01 <edk2-devel@lists.01.org>

> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming

> <liming.gao@intel.com>; Ard Biesheuvel <ard.biesheuvel@linaro.org>

> Subject: [edk2] "practical" memory allocation limit?

>

> Hi,

>

> I've been curious about it for a long time, so I guess I might as well

> ask the question:

>

> Even if OVMF's DXE phase is built for X64, and even if I give the VM

> memory above 4GB -- confirmed by the MEMMAP command in the UEFI shell

> --, in the OVMF log I see practically all allocations coming from the

> 32-bit address space.

>

> OVMF produces all its memory resource descriptor HOBs in PEI, all

> "tested". (No memory is added in DXE.) So why aren't DXE allocations

> served from 64-bit space?

>

> Something seems to be artificially limiting run-of-the-mill pool and

> page allocations in X64 OVMF; what is it? (It doesn't work like this in

> ArmVirtQemu.)

>

> Thanks!

> Laszlo

> _______________________________________________

> edk2-devel mailing list

> edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>

> https://lists.01.org/mailman/listinfo/edk2-devel

Re: "practical" memory allocation limit?

[Laszlo Ersek]

On 08/28/17 16:25, Shi, Steven wrote:
> Hi Laszlo,
> 
> I happen to have a question about how to correctly get the system memory size on Qemu.
> 
> In the QemuInitializeRam() of OvmfPkg\PlatformPei\MemDetect.c, I add debug info as below to trace the GetSystemMemorySizeBelow4gb() and GetSystemMemorySizeAbove4gb() output. But the output results seems not right if my input memory size > 4GB. See my commands and trace outputs in below.
> 
> 
> 
> MemDetect.c  Line 602:
> 
>   //
> 
>   // Determine total memory size available
> 
>   //
> 
>   LowerMemorySize = GetSystemMemorySizeBelow4gb ();
> 
>   UpperMemorySize = GetSystemMemorySizeAbove4gb ();
> 
>   DEBUG ((EFI_D_INFO, "LowerMemorySize= 0x%x\n", LowerMemorySize));
> 
>   DEBUG ((EFI_D_INFO, "UpperMemorySize= 0x%x\n", UpperMemorySize));
> 
> 
> 
> My test commands and trace outputs:
> 
> $ /opt/qemu/bin/qemu-system-x86_64 -m 5120 -enable-kvm -hda /home/jshi19/workspace/simics5-project/images/luv-v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial file:serial.log
> 
> LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?
> 
> UpperMemorySize= 0xC0000000 //3GB, but should not it be 1GB?

No, this is correct; the values returned are system memory *amounts*.
(On the QEMU command line, the "-m" switch says how much system memory
you have, not the highest address in system memory.)

In the 4GB address space, you don't have *just* system memory, you have
a whole bunch of MMIO too. (Most of that is used for 32-bit MMIO PCI BAR
allocation, some is used for the pflash chip range, LAPICs, etc.) On Q35
you also have 256MB assigned to the MMCONFIG / ECAM area (which provides
direct MMIO access to PCI config space for PCI 256 buses).

The physical memory map also depends on the board model; on i440fx, the
32-bit system memory can go up to 3GB, while on Q35, it only goes up to 2GB.

So, if you pass 5GB to a Q35 machine, 2GB of that are placed in the
address space at [0, 2G), and the other 3GB are placed in the address
space at [4G, 7G).

> $ /opt/qemu/bin/qemu-system-x86_64 -m 6144 -enable-kvm -hda /home/jshi19/workspace/simics5-project/images/luv-v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial file:serial.log
> 
> LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?
> 
> UpperMemorySize= 0x0                 // 0GB, but should not it be 2GB?

With 6G DRAM, 2G go low, and 4G go high.

(Your output is misleading because you used the %x format specifier in
your debug message -- %Lx would be correct, for printing a UINT64 value.)

Laszlo

Re: "practical" memory allocation limit?

[Shi, Steven]

OK, got it. Thanks.



For the why the 64bits DXE usually prefer allocations below 4GB, there is a good white paper elaborate the memory service initialization flows and can explain the reason. Please see the page 23 in below white paper (I have mark the answer in red).



https://github.com/tianocore-docs/Docs/blob/master/White_Papers/A_Tour_Beyond_BIOS_Securiy_Enhancement_to_Mitigate_Buffer_Overflow_in_UEFI.pdf

Heap Management in EDK II
In UEFI, the DxeCore maintains the heap usage. The UEFI driver or application may call
AllocatePages/FreePages/AllocatePool/FreePool to allocate or free the resource, or call
GetMemoryMap() to review all of the memory usage.
[Heap Initialization]
When DxeIpl transfers control to the DxeCore, all of the resource information is reported in a
Hand-off-Block (HOB) [PI] list. The DxeCore constructs the heap based upon the HOB
information. See figure 4-2 Heap Initialization.
1) The DxeCore needs to find one region to serve as the initial memory in
CoreInitializeMemoryServices()
(https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Dxe/Gcd/G
cd.c). The function is responsible for priming the memory map so that memory allocations
and resource allocations can be made. If the memory region described by the PHIT HOB is
big enough to hold BIN and minimum initial memory, this memory region is used as
highest priority. It can make the memory BIN allocation to be at the same memory region
with PHIT that has better compatibility to avoid memory fragmentation. Usually the BIN
size is already considered by platform PEIM when the platform PEIM calls
InstallPeiMemory() to PEI core.








Steven Shi

Intel\SSG\STO\UEFI Firmware



Tel: +86 021-61166522

iNet: 821-6522





> -----Original Message-----

> From: Laszlo Ersek [mailto:lersek@redhat.com]

> Sent: Monday, August 28, 2017 10:40 PM

> To: Shi, Steven <steven.shi@intel.com>; edk2-devel-01 <edk2-

> devel@lists.01.org>

> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming

> <liming.gao@intel.com>; Ard Biesheuvel <ard.biesheuvel@linaro.org>

> Subject: Re: [edk2] "practical" memory allocation limit?

>

> On 08/28/17 16:25, Shi, Steven wrote:

> > Hi Laszlo,

> >

> > I happen to have a question about how to correctly get the system memory

> size on Qemu.

> >

> > In the QemuInitializeRam() of OvmfPkg\PlatformPei\MemDetect.c, I add

> debug info as below to trace the GetSystemMemorySizeBelow4gb() and

> GetSystemMemorySizeAbove4gb() output. But the output results seems not

> right if my input memory size > 4GB. See my commands and trace outputs in

> below.

> >

> >

> >

> > MemDetect.c  Line 602:

> >

> >   //

> >

> >   // Determine total memory size available

> >

> >   //

> >

> >   LowerMemorySize = GetSystemMemorySizeBelow4gb ();

> >

> >   UpperMemorySize = GetSystemMemorySizeAbove4gb ();

> >

> >   DEBUG ((EFI_D_INFO, "LowerMemorySize= 0x%x\n", LowerMemorySize));

> >

> >   DEBUG ((EFI_D_INFO, "UpperMemorySize= 0x%x\n", UpperMemorySize));

> >

> >

> >

> > My test commands and trace outputs:

> >

> > $ /opt/qemu/bin/qemu-system-x86_64 -m 5120 -enable-kvm -hda

> /home/jshi19/workspace/simics5-project/images/luv-

> v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial

> file:serial.log

> >

> > LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?

> >

> > UpperMemorySize= 0xC0000000 //3GB, but should not it be 1GB?

>

> No, this is correct; the values returned are system memory *amounts*.

> (On the QEMU command line, the "-m" switch says how much system

> memory

> you have, not the highest address in system memory.)

>

> In the 4GB address space, you don't have *just* system memory, you have

> a whole bunch of MMIO too. (Most of that is used for 32-bit MMIO PCI BAR

> allocation, some is used for the pflash chip range, LAPICs, etc.) On Q35

> you also have 256MB assigned to the MMCONFIG / ECAM area (which

> provides

> direct MMIO access to PCI config space for PCI 256 buses).

>

> The physical memory map also depends on the board model; on i440fx, the

> 32-bit system memory can go up to 3GB, while on Q35, it only goes up to

> 2GB.

>

> So, if you pass 5GB to a Q35 machine, 2GB of that are placed in the

> address space at [0, 2G), and the other 3GB are placed in the address

> space at [4G, 7G).

>

> > $ /opt/qemu/bin/qemu-system-x86_64 -m 6144 -enable-kvm -hda

> /home/jshi19/workspace/simics5-project/images/luv-

> v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial

> file:serial.log

> >

> > LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?

> >

> > UpperMemorySize= 0x0                 // 0GB, but should not it be 2GB?

>

> With 6G DRAM, 2G go low, and 4G go high.

>

> (Your output is misleading because you used the %x format specifier in

> your debug message -- %Lx would be correct, for printing a UINT64 value.)

>

> Laszlo

Re: "practical" memory allocation limit?

[Andrew Fish]

For X64 (x86-64) systems it is common for PEI to run in 32-bit mode with paging disabled. This means the DXE Core gets loaded under 4GB and and the HOBs and such are < 4GB. So having the initial memory map < 4GB helps with preventing fragmentation. 

There is also architectural cruft on x86. Historically some of the PCI devices did not support DMA > 4GB and the system needed memory under for DMA buffers. The other problem is processor mode transitions as real mode code needs to be < 1MB and 32-bit protected mode needs to be under 4GB. To get into the 64-bit long mode you start in real mode, transition to protected mode, and then enter long mode. Even in 2017 there are still real mode entry points required for x86, mainly the reset vector, and IPI for the AP (how you wake up the other CPUs even in the OS). If you are using a CSM and need to get into real mode I think the thunk code assumes it runs < 4G (at least it did a long time ago). 

Not to mention just general cruft in the code that has assumptions about running < 4G (things like AsmThunk16(), CPU drivers etc.). 

Thanks,

Andrew Fish 


> On Aug 28, 2017, at 8:31 AM, Shi, Steven <steven.shi@intel.com> wrote:
> 
> OK, got it. Thanks.
> 
> 
> 
> For the why the 64bits DXE usually prefer allocations below 4GB, there is a good white paper elaborate the memory service initialization flows and can explain the reason. Please see the page 23 in below white paper (I have mark the answer in red).
> 
> 
> 
> https://github.com/tianocore-docs/Docs/blob/master/White_Papers/A_Tour_Beyond_BIOS_Securiy_Enhancement_to_Mitigate_Buffer_Overflow_in_UEFI.pdf
> 
> Heap Management in EDK II
> In UEFI, the DxeCore maintains the heap usage. The UEFI driver or application may call
> AllocatePages/FreePages/AllocatePool/FreePool to allocate or free the resource, or call
> GetMemoryMap() to review all of the memory usage.
> [Heap Initialization]
> When DxeIpl transfers control to the DxeCore, all of the resource information is reported in a
> Hand-off-Block (HOB) [PI] list. The DxeCore constructs the heap based upon the HOB
> information. See figure 4-2 Heap Initialization.
> 1) The DxeCore needs to find one region to serve as the initial memory in
> CoreInitializeMemoryServices()
> (https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Dxe/Gcd/G
> cd.c). The function is responsible for priming the memory map so that memory allocations
> and resource allocations can be made. If the memory region described by the PHIT HOB is
> big enough to hold BIN and minimum initial memory, this memory region is used as
> highest priority. It can make the memory BIN allocation to be at the same memory region
> with PHIT that has better compatibility to avoid memory fragmentation. Usually the BIN
> size is already considered by platform PEIM when the platform PEIM calls
> InstallPeiMemory() to PEI core.
> 
> 
> 
> 
> 
> 
> 
> 
> Steven Shi
> 
> Intel\SSG\STO\UEFI Firmware
> 
> 
> 
> Tel: +86 021-61166522
> 
> iNet: 821-6522
> 
> 
> 
> 
> 
>> -----Original Message-----
> 
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
> 
>> Sent: Monday, August 28, 2017 10:40 PM
> 
>> To: Shi, Steven <steven.shi@intel.com>; edk2-devel-01 <edk2-
> 
>> devel@lists.01.org>
> 
>> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming
> 
>> <liming.gao@intel.com>; Ard Biesheuvel <ard.biesheuvel@linaro.org>
> 
>> Subject: Re: [edk2] "practical" memory allocation limit?
> 
>> 
> 
>> On 08/28/17 16:25, Shi, Steven wrote:
> 
>>> Hi Laszlo,
> 
>>> 
> 
>>> I happen to have a question about how to correctly get the system memory
> 
>> size on Qemu.
> 
>>> 
> 
>>> In the QemuInitializeRam() of OvmfPkg\PlatformPei\MemDetect.c, I add
> 
>> debug info as below to trace the GetSystemMemorySizeBelow4gb() and
> 
>> GetSystemMemorySizeAbove4gb() output. But the output results seems not
> 
>> right if my input memory size > 4GB. See my commands and trace outputs in
> 
>> below.
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> MemDetect.c  Line 602:
> 
>>> 
> 
>>>  //
> 
>>> 
> 
>>>  // Determine total memory size available
> 
>>> 
> 
>>>  //
> 
>>> 
> 
>>>  LowerMemorySize = GetSystemMemorySizeBelow4gb ();
> 
>>> 
> 
>>>  UpperMemorySize = GetSystemMemorySizeAbove4gb ();
> 
>>> 
> 
>>>  DEBUG ((EFI_D_INFO, "LowerMemorySize= 0x%x\n", LowerMemorySize));
> 
>>> 
> 
>>>  DEBUG ((EFI_D_INFO, "UpperMemorySize= 0x%x\n", UpperMemorySize));
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> My test commands and trace outputs:
> 
>>> 
> 
>>> $ /opt/qemu/bin/qemu-system-x86_64 -m 5120 -enable-kvm -hda
> 
>> /home/jshi19/workspace/simics5-project/images/luv-
> 
>> v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial
> 
>> file:serial.log
> 
>>> 
> 
>>> LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?
> 
>>> 
> 
>>> UpperMemorySize= 0xC0000000 //3GB, but should not it be 1GB?
> 
>> 
> 
>> No, this is correct; the values returned are system memory *amounts*.
> 
>> (On the QEMU command line, the "-m" switch says how much system
> 
>> memory
> 
>> you have, not the highest address in system memory.)
> 
>> 
> 
>> In the 4GB address space, you don't have *just* system memory, you have
> 
>> a whole bunch of MMIO too. (Most of that is used for 32-bit MMIO PCI BAR
> 
>> allocation, some is used for the pflash chip range, LAPICs, etc.) On Q35
> 
>> you also have 256MB assigned to the MMCONFIG / ECAM area (which
> 
>> provides
> 
>> direct MMIO access to PCI config space for PCI 256 buses).
> 
>> 
> 
>> The physical memory map also depends on the board model; on i440fx, the
> 
>> 32-bit system memory can go up to 3GB, while on Q35, it only goes up to
> 
>> 2GB.
> 
>> 
> 
>> So, if you pass 5GB to a Q35 machine, 2GB of that are placed in the
> 
>> address space at [0, 2G), and the other 3GB are placed in the address
> 
>> space at [4G, 7G).
> 
>> 
> 
>>> $ /opt/qemu/bin/qemu-system-x86_64 -m 6144 -enable-kvm -hda
> 
>> /home/jshi19/workspace/simics5-project/images/luv-
> 
>> v2.1_diskboot_gpt_x86_64_.img -machine pc-q35-2.9 -bios OVMF.fd -serial
> 
>> file:serial.log
> 
>>> 
> 
>>> LowerMemorySize= 0x80000000 //2GB, but should not it be 4GB?
> 
>>> 
> 
>>> UpperMemorySize= 0x0                 // 0GB, but should not it be 2GB?
> 
>> 
> 
>> With 6G DRAM, 2G go low, and 4G go high.
> 
>> 
> 
>> (Your output is misleading because you used the %x format specifier in
> 
>> your debug message -- %Lx would be correct, for printing a UINT64 value.)
> 
>> 
> 
>> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: "practical" memory allocation limit?

[Laszlo Ersek]

On 08/28/17 18:23, Andrew Fish wrote:
> For X64 (x86-64) systems it is common for PEI to run in 32-bit mode
> with paging disabled. This means the DXE Core gets loaded under 4GB
> and and the HOBs and such are < 4GB. So having the initial memory map
> < 4GB helps with preventing fragmentation.

Hmm OK. That makes sense, thanks.

>
> There is also architectural cruft on x86. Historically some of the PCI
> devices did not support DMA > 4GB and the system needed memory under
> for DMA buffers. The other problem is processor mode transitions as
> real mode code needs to be < 1MB and 32-bit protected mode needs to be
> under 4GB. To get into the 64-bit long mode you start in real mode,
> transition to protected mode, and then enter long mode. Even in 2017
> there are still real mode entry points required for x86, mainly the
> reset vector, and IPI for the AP (how you wake up the other CPUs even
> in the OS). If you are using a CSM and need to get into real mode I
> think the thunk code assumes it runs < 4G (at least it did a long time
> ago).
>
> Not to mention just general cruft in the code that has assumptions
> about running < 4G (things like AsmThunk16(), CPU drivers etc.).

Right, I've seen several examples for this in edk2; I just didn't
understand why the "normal" stuff would get allocated by default under
4G. Mitigating fragmentation explains it though, thanks!

>> On Aug 28, 2017, at 8:31 AM, Shi, Steven <steven.shi@intel.com>
>> wrote:
>>
>> OK, got it. Thanks.

(Um... looks like your email was technically in response to Steven's...
but I got only yours (thus far), not Steven's.)

>> For the why the 64bits DXE usually prefer allocations below 4GB,
>> there is a good white paper elaborate the memory service
>> initialization flows and can explain the reason. Please see the page
>> 23 in below white paper (I have mark the answer in red).
>>
>> https://github.com/tianocore-docs/Docs/blob/master/White_Papers/A_Tour_Beyond_BIOS_Securiy_Enhancement_to_Mitigate_Buffer_Overflow_in_UEFI.pdf
>>
>> Heap Management in EDK II
>> In UEFI, the DxeCore maintains the heap usage. The UEFI driver or
>> application may call AllocatePages/FreePages/AllocatePool/FreePool to
>> allocate or free the resource, or call GetMemoryMap() to review all
>> of the memory usage.
>> [Heap Initialization]
>> When DxeIpl transfers control to the DxeCore, all of the resource
>> information is reported in a Hand-off-Block (HOB) [PI] list. The
>> DxeCore constructs the heap based upon the HOB information. See
>> figure 4-2 Heap Initialization.
>> 1) The DxeCore needs to find one region to serve as the initial
>> memory in CoreInitializeMemoryServices()
>> (https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Dxe/Gcd/Gcd.c).
>> The function is responsible for priming the memory map so that memory
>> allocations and resource allocations can be made. If the memory
>> region described by the PHIT HOB is big enough to hold BIN and
>> minimum initial memory, this memory region is used as highest
>> priority. It can make the memory BIN allocation to be at the same
>> memory region with PHIT that has better compatibility to avoid memory
>> fragmentation. Usually the BIN size is already considered by platform
>> PEIM when the platform PEIM calls InstallPeiMemory() to PEI core.

The top-posting got even more confusing here (I honestly can't tell who
quoted this from the whitepaper), but it answers my question. To
whomever I have to thank for it: thank you. :)

Laszlo

Re: "practical" memory allocation limit?

[Rebecca Cran]

On 08/28/2017 10:23 AM, Andrew Fish wrote:
> For X64 (x86-64) systems it is common for PEI to run in 32-bit mode with paging disabled. This means the DXE Core gets loaded under 4GB and and the HOBs and such are < 4GB. So having the initial memory map < 4GB helps with preventing fragmentation.

This is something I ran into with a driver I was working on: when being 
Load'd from a PCIe optrom only 4GB RAM could be seen; later, in the 
Start function, all 32GB or however much could be accessed.

Then there was still the problem of certain machines limiting 
AllocatePool calls to 4GB, which since our driver at times needed many 
times that amount required us to iterate over the memory map and call 
AllocatePages.

-- 
Rebecca Cran