* Clarification of Memory management in PEI phase @ 2022-06-09 17:28 Ayush Singh 2022-06-09 20:26 ` [edk2-devel] " Andrew Fish 0 siblings, 1 reply; 11+ messages in thread From: Ayush Singh @ 2022-06-09 17:28 UTC (permalink / raw) To: edk2-devel-groups-io Hello everyone, Can anyone help me with understanding dynamic memory management in PEI phase? In the UEFI Platform Integration Specification, version 1.7 Errata A, Section 4.6, PEI Memory services are given which include: 1. InstallPeiMemory() 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() 6. FreePages() However, no `FreePool()` service seems to be present. So how is the memory allocated using `AllocatePool()` freed? Ayush Singh ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-09 17:28 Clarification of Memory management in PEI phase Ayush Singh @ 2022-06-09 20:26 ` Andrew Fish 2022-06-10 5:22 ` Ayush Singh 0 siblings, 1 reply; 11+ messages in thread From: Andrew Fish @ 2022-06-09 20:26 UTC (permalink / raw) To: devel, ayushdevel1325 > On Jun 9, 2022, at 10:28 AM, Ayush Singh <ayushdevel1325@gmail.com> wrote: > > Hello everyone, > > Can anyone help me with understanding dynamic memory management in PEI > phase? In the UEFI Platform Integration Specification, version 1.7 > Errata A, Section 4.6, PEI Memory services are given which include: > > 1. InstallPeiMemory() This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, MemoryBegin, MemoryLength); This is how you tell the PEI Core the location of the memory that will can be used in PEI. > 2. AllocatePages() > 3. AllocatePool() > 4. CopyMem() > 5. SetMem() > 6. FreePages() > > However, no `FreePool()` service seems to be present. So how is the > memory allocated using `AllocatePool()` freed? > It basically gets Freed when you transition to the DXE phase. To step back for a minute I think it is important to remember that the main job of PEI is to initialize DRAM, and deal with S3 (resuming from suspend to RAM). So as soon as you have DRAM you are kind done and ready for the DXE IPL so you can load the DXE Phase and start up EFI. Remember PEI is Pre EFI. The reality is programming DRAM is complex and lots of code got written, then lots more code got written and PEI has become large for some ports. That was never the intent. PEI is designed as a way to run C code when you code is running from ROM and you don’t have any DRAM. For x86 not having DRAM means you are using the cache as RAM. For some SoCs there is actually an SRAM you can use. Thus the PEI memory allocation scheme is designed to deal with this very constrained environment. You start PEI with a heap and stack. You can also allocate HOBs (Hand Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There is no way to free a HOB. So the AllocatePool() kind of leaks into DXE too as an entry in the HOB list. But when the OS called gBS->ExitBootServices() that frees all non runtime memory back to the OS. If you look a AllocatePages/FreePages you will see AllocatePages creates a HOB that points to the memory region, and FreePages just marks that HOB as not used. That code is also in this file [1]. TL;DR there is no pool manager in PEI. [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 Thanks, Andrew Fish > Ayush Singh > > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-09 20:26 ` [edk2-devel] " Andrew Fish @ 2022-06-10 5:22 ` Ayush Singh 2022-06-22 19:39 ` Brian J. Johnson 2022-06-22 22:58 ` [edk2-devel] Clarification of Memory management in PEI phase Nate DeSimone 0 siblings, 2 replies; 11+ messages in thread From: Ayush Singh @ 2022-06-10 5:22 UTC (permalink / raw) To: Andrew Fish; +Cc: devel [-- Attachment #1: Type: text/plain, Size: 2750 bytes --] Thanks for the wonderful answer. Ayush Singh On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com> wrote: > >> On Jun 9, 2022, at 10:28 AM, Ayush Singh <ayushdevel1325@gmail.com >> <mailto:ayushdevel1325@gmail.com>> wrote: >> >> Hello everyone, >> >> Can anyone help me with understanding dynamic memory management in >> PEI >> phase? In the UEFI Platform Integration Specification, version 1.7 >> Errata A, Section 4.6, PEI Memory services are given which include: >> >> 1. InstallPeiMemory() > > This is basically: > (*PeiServices)->InstallPeiMemory (PeiServices, MemoryBegin, > MemoryLength); > > This is how you tell the PEI Core the location of the memory that > will can be used in PEI. > >> 2. AllocatePages() >> 3. AllocatePool() >> 4. CopyMem() >> 5. SetMem() >> 6. FreePages() >> >> However, no `FreePool()` service seems to be present. So how is the >> memory allocated using `AllocatePool()` freed? >> > > It basically gets Freed when you transition to the DXE phase. > > To step back for a minute I think it is important to remember that > the main job of PEI is to initialize DRAM, and deal with S3 (resuming > from suspend to RAM). So as soon as you have DRAM you are kind done > and ready for the DXE IPL so you can load the DXE Phase and start up > EFI. Remember PEI is Pre EFI. The reality is programming DRAM is > complex and lots of code got written, then lots more code got written > and PEI has become large for some ports. That was never the intent. > PEI is designed as a way to run C code when you code is running from > ROM and you donā��t have any DRAM. For x86 not having DRAM means you > are using the cache as RAM. For some SoCs there is actually an SRAM > you can use. Thus the PEI memory allocation scheme is designed to > deal with this very constrained environment. > > You start PEI with a heap and stack. You can also allocate HOBs (Hand > Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There > is no way to free a HOB. So the AllocatePool() kind of leaks into DXE > too as an entry in the HOB list. But when the OS called > gBS->ExitBootServices() that frees all non runtime memory back to the > OS. > > If you look a AllocatePages/FreePages you will see AllocatePages > creates a HOB that points to the memory region, and FreePages just > marks that HOB as not used. That code is also in this file [1]. > > TL;DR there is no pool manager in PEI. > > [1] > <https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878> > > > Thanks, > > Andrew Fish > >> Ayush Singh >> >> >> >> >> > [-- Attachment #2: Type: text/html, Size: 2978 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-10 5:22 ` Ayush Singh @ 2022-06-22 19:39 ` Brian J. Johnson 2022-06-22 20:54 ` Andrew Fish 2022-06-22 22:58 ` [edk2-devel] Clarification of Memory management in PEI phase Nate DeSimone 1 sibling, 1 reply; 11+ messages in thread From: Brian J. Johnson @ 2022-06-22 19:39 UTC (permalink / raw) To: devel, ayushdevel1325, Andrew Fish [-- Attachment #1: Type: text/plain, Size: 3981 bytes --] Sorry for the late response to this thread... PEI has grown greatly in complexity, as Andrew pointed out. Do we (TianoCore community) think it's time to add a real pool manager to PEI? There's getting to be quite a bit of post-DRAM-initialization PEI code on some platforms. And really, having a limited amount of pre-DRAM memory available is an even better reason for having an effective pool allocator, so the memory can be freed and reused. FWIW we (HPE) needed to add a private pool allocator to manage memory for a proprietary h/w initialization module which needs to run in post-DRAM PEI, and depends on allocating and freeing memory in different phases of its operation. Brian J. Johnson ------------------------------------------------------------------------ *From:* Ayush Singh [mailto:ayushdevel1325@gmail.com] *Sent:* Friday, June 10, 2022, 12:22 AM *To:* Andrew Fish <afish@apple.com> *Cc:* devel@edk2.groups.io *Subject:* [edk2-devel] Clarification of Memory management in PEI phase > Thanks for the wonderful answer. > > Ayush Singh > > On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com> > wrote: >> >> On Jun 9, 2022, at 10:28 AM, Ayush Singh >> <ayushdevel1325@gmail.comwrote: Hello everyone, Can anyone help >> me with understanding dynamic memory management in PEI phase? In >> the UEFI Platform Integration Specification, version 1.7 Errata >> A, Section 4.6, PEI Memory services are given which include: 1. >> InstallPeiMemory() >> >> This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, >> MemoryBegin, MemoryLength); This is how you tell the PEI Core the >> location of the memory that will can be used in PEI. >> >> 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() 6. >> FreePages() However, no `FreePool()` service seems to be present. >> So how is the memory allocated using `AllocatePool()` freed? >> >> It basically gets Freed when you transition to the DXE phase. To step >> back for a minute I think it is important to remember that the main >> job of PEI is to initialize DRAM, and deal with S3 (resuming from >> suspend to RAM). So as soon as you have DRAM you are kind done and >> ready for the DXE IPL so you can load the DXE Phase and start up EFI. >> Remember PEI is Pre EFI. The reality is programming DRAM is complex >> and lots of code got written, then lots more code got written and PEI >> has become large for some ports. That was never the intent. PEI is >> designed as a way to run C code when you code is running from ROM and >> you donā��t have any DRAM. For x86 not having DRAM means you are >> using the cache as RAM. For some SoCs there is actually an SRAM you >> can use. Thus the PEI memory allocation scheme is designed to deal >> with this very constrained environment. You start PEI with a heap and >> stack. You can also allocate HOBs (Hand Off Blocks). A pool >> allocation in PEI is just a HOB. See [1]. There is no way to free a >> HOB. So the AllocatePool() kind of leaks into DXE too as an entry in >> the HOB list. But when the OS called gBS->ExitBootServices() that >> frees all non runtime memory back to the OS. If you look a >> AllocatePages/FreePages you will see AllocatePages creates a HOB that >> points to the memory region, and FreePages just marks that HOB as not >> used. That code is also in this file [1]. TL;DR there is no pool >> manager in PEI. [1] >> https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 >> Thanks, Andrew Fish >> >> Ayush Singh >> > Brian ------------------------------------------------------------------------------------------------------------------------------ "It's OK to be stuck. 99% of the time in your own [research] work, you're stuck." -- Mark Lawrence [-- Attachment #2: Type: text/html, Size: 7291 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-22 19:39 ` Brian J. Johnson @ 2022-06-22 20:54 ` Andrew Fish 2022-06-22 21:41 ` Brian J. Johnson 0 siblings, 1 reply; 11+ messages in thread From: Andrew Fish @ 2022-06-22 20:54 UTC (permalink / raw) To: Brian J. Johnson; +Cc: devel, ayushdevel1325 [-- Attachment #1: Type: text/plain, Size: 4822 bytes --] Brian, I think all the PEI Allocate Pool does is make a HOB [1]. I don’t think we can remove HOBs when memory is freed as that would change pointers to pool. It may be possible to mark a region as free and allocate from that list batch 1st, and just over allocate space if needed. It could be as simple as just adding a new HOB type to represent free pool. [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 Thanks, Andrew Fish > On Jun 22, 2022, at 12:39 PM, Brian J. Johnson <brian.johnson@hpe.com> wrote: > > Sorry for the late response to this thread... > > PEI has grown greatly in complexity, as Andrew pointed out. Do we (TianoCore community) think it's time to add a real pool manager to PEI? There's getting to be quite a bit of post-DRAM-initialization PEI code on some platforms. And really, having a limited amount of pre-DRAM memory available is an even better reason for having an effective pool allocator, so the memory can be freed and reused. > > FWIW we (HPE) needed to add a private pool allocator to manage memory for a proprietary h/w initialization module which needs to run in post-DRAM PEI, and depends on allocating and freeing memory in different phases of its operation. > > Brian J. Johnson > > From: Ayush Singh [mailto:ayushdevel1325@gmail.com <mailto:ayushdevel1325@gmail.com>] > Sent: Friday, June 10, 2022, 12:22 AM > To: Andrew Fish <afish@apple.com> <mailto:afish@apple.com> > Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io> > Subject: [edk2-devel] Clarification of Memory management in PEI phase > >> Thanks for the wonderful answer. >> >> Ayush Singh >> >> On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com> <mailto:afish@apple.com> wrote: >>> >>> On Jun 9, 2022, at 10:28 AM, Ayush Singh <ayushdevel1325@gmail.com <mailto:ayushdevel1325@gmail.com>wrote: >>> >>> Hello everyone, >>> >>> Can anyone help me with understanding dynamic memory management in PEI >>> phase? In the UEFI Platform Integration Specification, version 1.7 >>> Errata A, Section 4.6, PEI Memory services are given which include: >>> >>> 1. InstallPeiMemory() >>> >>> This is basically: >>> (*PeiServices)->InstallPeiMemory (PeiServices, MemoryBegin, MemoryLength); >>> >>> This is how you tell the PEI Core the location of the memory that will can be used in PEI. >>> >>> 2. AllocatePages() >>> 3. AllocatePool() >>> 4. CopyMem() >>> 5. SetMem() >>> 6. FreePages() >>> >>> However, no `FreePool()` service seems to be present. So how is the >>> memory allocated using `AllocatePool()` freed? >>> >>> >>> It basically gets Freed when you transition to the DXE phase. >>> >>> To step back for a minute I think it is important to remember that the main job of PEI is to initialize DRAM, and deal with S3 (resuming from suspend to RAM). So as soon as you have DRAM you are kind done and ready for the DXE IPL so you can load the DXE Phase and start up EFI. Remember PEI is Pre EFI. The reality is programming DRAM is complex and lots of code got written, then lots more code got written and PEI has become large for some ports. That was never the intent. PEI is designed as a way to run C code when you code is running from ROM and you donā��t have any DRAM. For x86 not having DRAM means you are using the cache as RAM. For some SoCs there is actually an SRAM you can use. Thus the PEI memory allocation scheme is designed to deal with this very constrained environment. >>> >>> You start PEI with a heap and stack. You can also allocate HOBs (Hand Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There is no way to free a HOB. So the AllocatePool() kind of leaks into DXE too as an entry in the HOB list. But when the OS called gBS->ExitBootServices() that frees all non runtime memory back to the OS. >>> >>> If you look a AllocatePages/FreePages you will see AllocatePages creates a HOB that points to the memory region, and FreePages just marks that HOB as not used. That code is also in this file [1]. >>> >>> TL;DR there is no pool manager in PEI. >>> >>> [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 <https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878> >>> >>> >>> Thanks, >>> >>> Andrew Fish >>> >>> Ayush Singh >>> >>> >>> >>> >>> >>> >> >> > Brian > > ------------------------------------------------------------------------------------------------------------------------------ > "It's OK to be stuck. 99% of the time in your own [research] work, > you're stuck." > -- Mark Lawrence [-- Attachment #2: Type: text/html, Size: 8854 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-22 20:54 ` Andrew Fish @ 2022-06-22 21:41 ` Brian J. Johnson 2022-06-22 21:59 ` vincent zimmer 0 siblings, 1 reply; 11+ messages in thread From: Brian J. Johnson @ 2022-06-22 21:41 UTC (permalink / raw) To: Andrew Fish; +Cc: devel, ayushdevel1325 [-- Attachment #1: Type: text/plain, Size: 5566 bytes --] Andrew, Yes, adding a new HOB type to represent free pool would probably be the easiest. Or we could write or borrow a traditional malloc() implementation, similar to DXE's pool allocator, and back it with memory from AllocatePages(). That would probably have better performance, and would avoid fragmenting the pool memory when non-pool HOBs are added. It would probably be more code, though. Brian J. Johnson ------------------------------------------------------------------------ *From:* Andrew Fish [mailto:afish@apple.com] *Sent:* Wednesday, June 22, 2022, 3:54 PM *To:* Brian J. Johnson <brian.johnson@hpe.com> *Cc:* devel@edk2.groups.io, ayushdevel1325@gmail.com *Subject:* [edk2-devel] Clarification of Memory management in PEI phase > Brian, > > I think all the PEI Allocate Pool does is make a HOB [1]. I don’t > think we can remove HOBs when memory is freed as that would change > pointers to pool. It may be possible to mark a region as free and > allocate from that list batch 1st, and just over allocate space if > needed. It could be as simple as just adding a new HOB type to > represent free pool. > > > [1] > https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 > > Thanks, > > Andrew Fish > >> On Jun 22, 2022, at 12:39 PM, Brian J. Johnson >> <brian.johnson@hpe.comwrote: >> >> Sorry for the late response to this thread... >> >> PEI has grown greatly in complexity, as Andrew pointed out. Do we >> (TianoCore community) think it's time to add a real pool manager to >> PEI? There's getting to be quite a bit of post-DRAM-initialization >> PEI code on some platforms. And really, having a limited amount of >> pre-DRAM memory available is an even better reason for having an >> effective pool allocator, so the memory can be freed and reused. >> >> FWIW we (HPE) needed to add a private pool allocator to manage memory >> for a proprietary h/w initialization module which needs to run in >> post-DRAM PEI, and depends on allocating and freeing memory in >> different phases of its operation. >> >> Brian J. Johnson >> >> ------------------------------------------------------------------------ >> *From:* Ayush Singh [mailto:ayushdevel1325@gmail.com] >> *Sent:* Friday, June 10, 2022, 12:22 AM >> *To:* Andrew Fish <afish@apple.com> >> *Cc:* devel@edk2.groups.io >> *Subject:* [edk2-devel] Clarification of Memory management in PEI phase >> >>> Thanks for the wonderful answer. >>> >>> Ayush Singh >>> >>> On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish >>> <afish@apple.com> wrote: >>>> >>>> On Jun 9, 2022, at 10:28 AM, Ayush Singh >>>> <ayushdevel1325@gmail.comwrote: Hello everyone, Can anyone help >>>> me with understanding dynamic memory management in PEI phase? >>>> In the UEFI Platform Integration Specification, version 1.7 >>>> Errata A, Section 4.6, PEI Memory services are given which >>>> include: 1. InstallPeiMemory() >>>> >>>> This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, >>>> MemoryBegin, MemoryLength); This is how you tell the PEI Core the >>>> location of the memory that will can be used in PEI. >>>> >>>> 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() >>>> 6. FreePages() However, no `FreePool()` service seems to be >>>> present. So how is the memory allocated using `AllocatePool()` >>>> freed? >>>> >>>> It basically gets Freed when you transition to the DXE phase. To >>>> step back for a minute I think it is important to remember that the >>>> main job of PEI is to initialize DRAM, and deal with S3 (resuming >>>> from suspend to RAM). So as soon as you have DRAM you are kind done >>>> and ready for the DXE IPL so you can load the DXE Phase and start >>>> up EFI. Remember PEI is Pre EFI. The reality is programming DRAM is >>>> complex and lots of code got written, then lots more code got >>>> written and PEI has become large for some ports. That was never the >>>> intent. PEI is designed as a way to run C code when you code is >>>> running from ROM and you donā��t have any DRAM. For x86 not having >>>> DRAM means you are using the cache as RAM. For some SoCs there is >>>> actually an SRAM you can use. Thus the PEI memory allocation scheme >>>> is designed to deal with this very constrained environment. You >>>> start PEI with a heap and stack. You can also allocate HOBs (Hand >>>> Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There >>>> is no way to free a HOB. So the AllocatePool() kind of leaks into >>>> DXE too as an entry in the HOB list. But when the OS called >>>> gBS->ExitBootServices() that frees all non runtime memory back to >>>> the OS. If you look a AllocatePages/FreePages you will see >>>> AllocatePages creates a HOB that points to the memory region, and >>>> FreePages just marks that HOB as not used. That code is also in >>>> this file [1]. TL;DR there is no pool manager in PEI. [1] >>>> https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 >>>> Thanks, Andrew Fish >>>> >>>> Ayush Singh >>>> >>> > Brian ------------------------------------------------------------------------------------------------------------------------------ "There is the greatest practical benefit in making a few failures early in life." -- Thomas Henry Huxley [-- Attachment #2: Type: text/html, Size: 12843 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-22 21:41 ` Brian J. Johnson @ 2022-06-22 21:59 ` vincent zimmer 2022-07-01 17:00 ` FreePool in PEI (was: Clarification of Memory management in PEI phase) Brian J. Johnson 0 siblings, 1 reply; 11+ messages in thread From: vincent zimmer @ 2022-06-22 21:59 UTC (permalink / raw) To: devel, brian.johnson sounds like a good idea. As a next step, perhaps a candidate activity for https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Code-First-Process in anticipation of future inclusion in PI? Vincent On Wed, Jun 22, 2022 at 2:41 PM Brian J. Johnson <brian.johnson@hpe.com> wrote: > > Andrew, > > Yes, adding a new HOB type to represent free pool would probably be the easiest. > > Or we could write or borrow a traditional malloc() implementation, similar to DXE's pool allocator, and back it with memory from AllocatePages(). That would probably have better performance, and would avoid fragmenting the pool memory when non-pool HOBs are added. It would probably be more code, though. > > Brian J. Johnson > ________________________________ > From: Andrew Fish [mailto:afish@apple.com] > Sent: Wednesday, June 22, 2022, 3:54 PM > To: Brian J. Johnson <brian.johnson@hpe.com> > Cc: devel@edk2.groups.io, ayushdevel1325@gmail.com > Subject: [edk2-devel] Clarification of Memory management in PEI phase > > Brian, > > I think all the PEI Allocate Pool does is make a HOB [1]. I don’t think we can remove HOBs when memory is freed as that would change pointers to pool. It may be possible to mark a region as free and allocate from that list batch 1st, and just over allocate space if needed. It could be as simple as just adding a new HOB type to represent free pool. > > > [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 > > Thanks, > > Andrew Fish > > On Jun 22, 2022, at 12:39 PM, Brian J. Johnson <brian.johnson@hpe.comwrote: > > Sorry for the late response to this thread... > > PEI has grown greatly in complexity, as Andrew pointed out. Do we (TianoCore community) think it's time to add a real pool manager to PEI? There's getting to be quite a bit of post-DRAM-initialization PEI code on some platforms. And really, having a limited amount of pre-DRAM memory available is an even better reason for having an effective pool allocator, so the memory can be freed and reused. > > FWIW we (HPE) needed to add a private pool allocator to manage memory for a proprietary h/w initialization module which needs to run in post-DRAM PEI, and depends on allocating and freeing memory in different phases of its operation. > > Brian J. Johnson > > ________________________________ > From: Ayush Singh [mailto:ayushdevel1325@gmail.com] > Sent: Friday, June 10, 2022, 12:22 AM > To: Andrew Fish <afish@apple.com> > Cc: devel@edk2.groups.io > Subject: [edk2-devel] Clarification of Memory management in PEI phase > > Thanks for the wonderful answer. > > Ayush Singh > > On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com> wrote: > > On Jun 9, 2022, at 10:28 AM, Ayush Singh <ayushdevel1325@gmail.comwrote: Hello everyone, Can anyone help me with understanding dynamic memory management in PEI phase? In the UEFI Platform Integration Specification, version 1.7 Errata A, Section 4.6, PEI Memory services are given which include: 1. InstallPeiMemory() > > This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, MemoryBegin, MemoryLength); This is how you tell the PEI Core the location of the memory that will can be used in PEI. > > 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() 6. FreePages() However, no `FreePool()` service seems to be present. So how is the memory allocated using `AllocatePool()` freed? > > It basically gets Freed when you transition to the DXE phase. To step back for a minute I think it is important to remember that the main job of PEI is to initialize DRAM, and deal with S3 (resuming from suspend to RAM). So as soon as you have DRAM you are kind done and ready for the DXE IPL so you can load the DXE Phase and start up EFI. Remember PEI is Pre EFI. The reality is programming DRAM is complex and lots of code got written, then lots more code got written and PEI has become large for some ports. That was never the intent. PEI is designed as a way to run C code when you code is running from ROM and you donā��t have any DRAM. For x86 not having DRAM means you are using the cache as RAM. For some SoCs there is actually an SRAM you can use. Thus the PEI memory allocation scheme is designed to deal with this very constrained environment. You start PEI with a heap and stack. You can also allocate HOBs (Hand Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There is no way to free a HOB. So the AllocatePool() kind of leaks into DXE too as an entry in the HOB list. But when the OS called gBS->ExitBootServices() that frees all non runtime memory back to the OS. If you look a AllocatePages/FreePages you will see AllocatePages creates a HOB that points to the memory region, and FreePages just marks that HOB as not used. That code is also in this file [1]. TL;DR there is no pool manager in PEI. [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 Thanks, Andrew Fish > > Ayush Singh > > > Brian > > ------------------------------------------------------------------------------------------------------------------------------ > > "There is the greatest practical benefit in making a few failures > early in life." > -- Thomas Henry Huxley > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* FreePool in PEI (was: Clarification of Memory management in PEI phase) 2022-06-22 21:59 ` vincent zimmer @ 2022-07-01 17:00 ` Brian J. Johnson 0 siblings, 0 replies; 11+ messages in thread From: Brian J. Johnson @ 2022-07-01 17:00 UTC (permalink / raw) To: Vincent Zimmer, devel I've been looking at this a bit. It appears that implementing PEI FreePool in this way would require: * Adding a new HOB type EFI_HOB_TYPE_FREE_POOL to PiHob.h * Adding a new FreePool PEI service to PiPeiCis.h * Implementing the service in PeiMain.c and MemoryServices.c * Adding PeiServicesFreePool() to PeiServicesLib * Modifying PeiMemoryAllocationLib to call PeiServicesFreePool() when freeing EfiBootServicesData, while ignoring free requests for other memory types The last step is a bit problematic, since PeiMemoryAllocationLib implements non-EfiBootServicesData pool allocations using AllocatePages(). FreePool() doesn't take a MemoryType parameter, so it just has to trust that it has been given a EfiBootServicesData buffer which was allocated from the HOB list. If a caller tries to free non-EfiBootServicesData memory, eg. memory allocated via AllocateRuntimePool(), the PEI service will return an error. PeiMemoryAllocationLib can ignore the error, since a failed free is essentially a no-op, which is the traditional implementation of FreePool() in PEI anyway. Does that sound like an acceptable level of complexity and impact to the specifications? I'd rather not start the code first process if there's no chance it will go through. An alternative would be to implement pool allocation and freeing entirely in PeiMemoryAllocationLib, instead of adding a new PEI service. If the code is too big, PeiMemoryAllocationLib could wrap a PPI. Then we could have different implementations, either backed by the HOB list like the current allocator, or backed by pages with a more sophisticated allocator. Brian -------- Original Message -------- From: Vincent Zimmer [mailto:vincent.zimmer@gmail.com] Sent: Wednesday, June 22, 2022, 4:59 PM To: devel@edk2.groups.io, brian.johnson@hpe.com Subject: [edk2-devel] Clarification of Memory management in PEI phase sounds like a good idea. As a next step, perhaps a candidate activity for https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Code-First-Process in anticipation of future inclusion in PI? Vincent On Wed, Jun 22, 2022 at 2:41 PM Brian J. Johnson <brian.johnson@hpe.com> wrote: > > Andrew, > > Yes, adding a new HOB type to represent free pool would probably be the easiest. > > Or we could write or borrow a traditional malloc() implementation, similar to DXE's pool allocator, and back it with memory from AllocatePages(). That would probably have better performance, and would avoid fragmenting the pool memory when non-pool HOBs are added. It would probably be more code, though. > > Brian J. Johnson > ________________________________ > From: Andrew Fish [mailto:afish@apple.com] > Sent: Wednesday, June 22, 2022, 3:54 PM > To: Brian J. Johnson <brian.johnson@hpe.com> > Cc: devel@edk2.groups.io, ayushdevel1325@gmail.com > Subject: [edk2-devel] Clarification of Memory management in PEI phase > > Brian, > > I think all the PEI Allocate Pool does is make a HOB [1]. I don’t think we can remove HOBs when memory is freed as that would change pointers to pool. It may be possible to mark a region as free and allocate from that list batch 1st, and just over allocate space if needed. It could be as simple as just adding a new HOB type to represent free pool. > > > [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 > > Thanks, > > Andrew Fish > > On Jun 22, 2022, at 12:39 PM, Brian J. Johnson <brian.johnson@hpe.comwrote: > > Sorry for the late response to this thread... > > PEI has grown greatly in complexity, as Andrew pointed out. Do we (TianoCore community) think it's time to add a real pool manager to PEI? There's getting to be quite a bit of post-DRAM-initialization PEI code on some platforms. And really, having a limited amount of pre-DRAM memory available is an even better reason for having an effective pool allocator, so the memory can be freed and reused. > > FWIW we (HPE) needed to add a private pool allocator to manage memory for a proprietary h/w initialization module which needs to run in post-DRAM PEI, and depends on allocating and freeing memory in different phases of its operation. > > Brian J. Johnson > -- Brian J. Johnson Enterprise X86 Lab Hewlett Packard Enterprise brian.johnson@hpe.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-10 5:22 ` Ayush Singh 2022-06-22 19:39 ` Brian J. Johnson @ 2022-06-22 22:58 ` Nate DeSimone 2022-06-23 0:10 ` Michael Kubacki 2022-06-23 5:30 ` Ayush Singh 1 sibling, 2 replies; 11+ messages in thread From: Nate DeSimone @ 2022-06-22 22:58 UTC (permalink / raw) To: devel@edk2.groups.io, ayushdevel1325@gmail.com, Andrew Fish [-- Attachment #1: Type: text/plain, Size: 3315 bytes --] Hi Ayush, For your work to make Rust run in PEI I would recommend writing a generic heap manager that uses the PEI services AllocatePage() and FreePage(). PEI does allow you to truly free up memory but only when allocated in 4KB increments. Your heap manager can allow the Rust program to go to a smaller granularity. In parallel, I can see merit in the argument for adding proper heap management to PEI, but that would be a PI specification update that is way outside the scope of your GSoC project and won’t happen fast enough for the work you need to do this summer 😊. Thanks, Nate From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ayush Singh Sent: Thursday, June 9, 2022 10:23 PM To: Andrew Fish <afish@apple.com> Cc: devel@edk2.groups.io Subject: Re: [edk2-devel] Clarification of Memory management in PEI phase Thanks for the wonderful answer. Ayush Singh On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com<mailto:afish@apple.com>> wrote: On Jun 9, 2022, at 10:28 AM, Ayush Singh <ayushdevel1325@gmail.com<mailto:ayushdevel1325@gmail.com>> wrote: Hello everyone, Can anyone help me with understanding dynamic memory management in PEI phase? In the UEFI Platform Integration Specification, version 1.7 Errata A, Section 4.6, PEI Memory services are given which include: 1. InstallPeiMemory() This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, MemoryBegin, MemoryLength); This is how you tell the PEI Core the location of the memory that will can be used in PEI. 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() 6. FreePages() However, no `FreePool()` service seems to be present. So how is the memory allocated using `AllocatePool()` freed? It basically gets Freed when you transition to the DXE phase. To step back for a minute I think it is important to remember that the main job of PEI is to initialize DRAM, and deal with S3 (resuming from suspend to RAM). So as soon as you have DRAM you are kind done and ready for the DXE IPL so you can load the DXE Phase and start up EFI. Remember PEI is Pre EFI. The reality is programming DRAM is complex and lots of code got written, then lots more code got written and PEI has become large for some ports. That was never the intent. PEI is designed as a way to run C code when you code is running from ROM and you donā��t have any DRAM. For x86 not having DRAM means you are using the cache as RAM. For some SoCs there is actually an SRAM you can use. Thus the PEI memory allocation scheme is designed to deal with this very constrained environment. You start PEI with a heap and stack. You can also allocate HOBs (Hand Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There is no way to free a HOB. So the AllocatePool() kind of leaks into DXE too as an entry in the HOB list. But when the OS called gBS->ExitBootServices() that frees all non runtime memory back to the OS. If you look a AllocatePages/FreePages you will see AllocatePages creates a HOB that points to the memory region, and FreePages just marks that HOB as not used. That code is also in this file [1]. TL;DR there is no pool manager in PEI. [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 Thanks, Andrew Fish Ayush Singh [-- Attachment #2: Type: text/html, Size: 6802 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-22 22:58 ` [edk2-devel] Clarification of Memory management in PEI phase Nate DeSimone @ 2022-06-23 0:10 ` Michael Kubacki 2022-06-23 5:30 ` Ayush Singh 1 sibling, 0 replies; 11+ messages in thread From: Michael Kubacki @ 2022-06-23 0:10 UTC (permalink / raw) To: devel, nathaniel.l.desimone, ayushdevel1325@gmail.com, Andrew Fish I agree with other comments here. I mainly want to add that if you're new to PI/UEFI, you might find the following resource helpful for tying together some pieces from the code and documents like the PI Spec. https://github.com/tianocore-docs/Docs/raw/master/White_Papers/A_Tour_Beyond_BIOS_Memory_Map_And_Practices_in_UEFI_BIOS_V2.pdf Thanks, Michael On 6/22/2022 6:58 PM, Nate DeSimone wrote: > Hi Ayush, > > For your work to make Rust run in PEI I would recommend writing a > generic heap manager that uses the PEI services AllocatePage() and > FreePage(). PEI does allow you to truly free up memory but only when > allocated in 4KB increments. Your heap manager can allow the Rust > program to go to a smaller granularity. > > In parallel, I can see merit in the argument for adding proper heap > management to PEI, but that would be a PI specification update that is > way outside the scope of your GSoC project and won’t happen fast enough > for the work you need to do this summer 😊. > > Thanks, > > Nate > > *From:* devel@edk2.groups.io <devel@edk2.groups.io> *On Behalf Of *Ayush > Singh > *Sent:* Thursday, June 9, 2022 10:23 PM > *To:* Andrew Fish <afish@apple.com> > *Cc:* devel@edk2.groups.io > *Subject:* Re: [edk2-devel] Clarification of Memory management in PEI phase > > Thanks for the wonderful answer. > > Ayush Singh > > > On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com > <mailto:afish@apple.com>> wrote: > > On Jun 9, 2022, at 10:28 AM, Ayush Singh > <ayushdevel1325@gmail.com <mailto:ayushdevel1325@gmail.com>> > wrote: Hello everyone, Can anyone help me with understanding > dynamic memory management in PEI phase? In the UEFI Platform > Integration Specification, version 1.7 Errata A, Section 4.6, > PEI Memory services are given which include: 1. InstallPeiMemory() > > This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, > MemoryBegin, MemoryLength); This is how you tell the PEI Core the > location of the memory that will can be used in PEI. > > 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() 6. > FreePages() However, no `FreePool()` service seems to be > present. So how is the memory allocated using `AllocatePool()` > freed? > > It basically gets Freed when you transition to the DXE phase. To > step back for a minute I think it is important to remember that the > main job of PEI is to initialize DRAM, and deal with S3 (resuming > from suspend to RAM). So as soon as you have DRAM you are kind done > and ready for the DXE IPL so you can load the DXE Phase and start up > EFI. Remember PEI is Pre EFI. The reality is programming DRAM is > complex and lots of code got written, then lots more code got > written and PEI has become large for some ports. That was never the > intent. PEI is designed as a way to run C code when you code is > running from ROM and you donā��t have any DRAM. For x86 not having > DRAM means you are using the cache as RAM. For some SoCs there is > actually an SRAM you can use. Thus the PEI memory allocation scheme > is designed to deal with this very constrained environment. You > start PEI with a heap and stack. You can also allocate HOBs (Hand > Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There > is no way to free a HOB. So the AllocatePool() kind of leaks into > DXE too as an entry in the HOB list. But when the OS called > gBS->ExitBootServices() that frees all non runtime memory back to > the OS. If you look a AllocatePages/FreePages you will see > AllocatePages creates a HOB that points to the memory region, and > FreePages just marks that HOB as not used. That code is also in this > file [1]. TL;DR there is no pool manager in PEI. [1] > https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 > <https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878> > Thanks, Andrew Fish > > Ayush Singh > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] Clarification of Memory management in PEI phase 2022-06-22 22:58 ` [edk2-devel] Clarification of Memory management in PEI phase Nate DeSimone 2022-06-23 0:10 ` Michael Kubacki @ 2022-06-23 5:30 ` Ayush Singh 1 sibling, 0 replies; 11+ messages in thread From: Ayush Singh @ 2022-06-23 5:30 UTC (permalink / raw) To: edk2-devel-groups-io, Nate DeSimone Hello everyone, Thanks for the helpful answers. After discussion with the mentors, it was decided that we should first get the Rust std to work in DXE phase. So for now, that is the goal. Once DXE is complete and there is still time left (or I can just do it after GSoC as well), then I might look into implementing std for the PEI. Yours Sincerely Ayush Singh On Thu, Jun 23, 2022 at 4:28 AM Nate DeSimone <nathaniel.l.desimone@intel.com> wrote: > > Hi Ayush, > > > > For your work to make Rust run in PEI I would recommend writing a generic heap manager that uses the PEI services AllocatePage() and FreePage(). PEI does allow you to truly free up memory but only when allocated in 4KB increments. Your heap manager can allow the Rust program to go to a smaller granularity. > > > > In parallel, I can see merit in the argument for adding proper heap management to PEI, but that would be a PI specification update that is way outside the scope of your GSoC project and won’t happen fast enough for the work you need to do this summer 😊. > > > > Thanks, > > Nate > > > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ayush Singh > Sent: Thursday, June 9, 2022 10:23 PM > To: Andrew Fish <afish@apple.com> > Cc: devel@edk2.groups.io > Subject: Re: [edk2-devel] Clarification of Memory management in PEI phase > > > > Thanks for the wonderful answer. > > > > Ayush Singh > > > On Thu, Jun 9 2022 at 01:26:58 PM -0700, Andrew Fish <afish@apple.com> wrote: > > On Jun 9, 2022, at 10:28 AM, Ayush Singh <ayushdevel1325@gmail.com> wrote: Hello everyone, Can anyone help me with understanding dynamic memory management in PEI phase? In the UEFI Platform Integration Specification, version 1.7 Errata A, Section 4.6, PEI Memory services are given which include: 1. InstallPeiMemory() > > This is basically: (*PeiServices)->InstallPeiMemory (PeiServices, MemoryBegin, MemoryLength); This is how you tell the PEI Core the location of the memory that will can be used in PEI. > > 2. AllocatePages() 3. AllocatePool() 4. CopyMem() 5. SetMem() 6. FreePages() However, no `FreePool()` service seems to be present. So how is the memory allocated using `AllocatePool()` freed? > > It basically gets Freed when you transition to the DXE phase. To step back for a minute I think it is important to remember that the main job of PEI is to initialize DRAM, and deal with S3 (resuming from suspend to RAM). So as soon as you have DRAM you are kind done and ready for the DXE IPL so you can load the DXE Phase and start up EFI. Remember PEI is Pre EFI. The reality is programming DRAM is complex and lots of code got written, then lots more code got written and PEI has become large for some ports. That was never the intent. PEI is designed as a way to run C code when you code is running from ROM and you donā��t have any DRAM. For x86 not having DRAM means you are using the cache as RAM. For some SoCs there is actually an SRAM you can use. Thus the PEI memory allocation scheme is designed to deal with this very constrained environment. You start PEI with a heap and stack. You can also allocate HOBs (Hand Off Blocks). A pool allocation in PEI is just a HOB. See [1]. There is no way to free a HOB. So the AllocatePool() kind of leaks into DXE too as an entry in the HOB list. But when the OS called gBS->ExitBootServices() that frees all non runtime memory back to the OS. If you look a AllocatePages/FreePages you will see AllocatePages creates a HOB that points to the memory region, and FreePages just marks that HOB as not used. That code is also in this file [1]. TL;DR there is no pool manager in PEI. [1] https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Pei/Memory/MemoryServices.c#L878 Thanks, Andrew Fish > > Ayush Singh > > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-07-01 17:00 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-06-09 17:28 Clarification of Memory management in PEI phase Ayush Singh 2022-06-09 20:26 ` [edk2-devel] " Andrew Fish 2022-06-10 5:22 ` Ayush Singh 2022-06-22 19:39 ` Brian J. Johnson 2022-06-22 20:54 ` Andrew Fish 2022-06-22 21:41 ` Brian J. Johnson 2022-06-22 21:59 ` vincent zimmer 2022-07-01 17:00 ` FreePool in PEI (was: Clarification of Memory management in PEI phase) Brian J. Johnson 2022-06-22 22:58 ` [edk2-devel] Clarification of Memory management in PEI phase Nate DeSimone 2022-06-23 0:10 ` Michael Kubacki 2022-06-23 5:30 ` Ayush Singh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox