Re: [edk2-discuss] GSoC Proposal

Re: [edk2-discuss] GSoC Proposal

[Marvin Häuser]

CC Nate (GSoC admin)
CC Steven (task mentor)
CC edk2-devel (you picked the logically correct list, but it’s pretty dead and barely anyone reads it)

Hey Ada,

Out of mere curiosity, why did you pick this item? :)

Hey Steven,

I feel like there is more to your proposal than is given on the task page. Why is it “ELF first”, is it something useful for UefiPayloadPkg or Linux somehow?

As for supporting it in the EDK II core, I personally feel like this is much too late. The entire ecosystem is centred around protocols (and the services tables) already. “Loading only when necessary” doesn’t sound very important to me personally, as the firmware image is already supposed to be fairly minimal. I’d rather like to see the introduction of “lazy protocols” (which do not require any new fundamental concepts), e.g., for network and HID stuff like mice and touch, which go through the driver connection procedure only when a protocol function is called for the first time. A big issue with this of course are non-function pointers in the protocol structure.

This will not only require a dynamic linker in the firmware to maybe double the size of the already disgusting and vastly unmaintained PE loader, it will also require further format conversion from ELF and Mach-O, both of which already are buggy (the former much more so than the latter). This is a tremendous effort in my opinion and introducing partial support will cause more awkward toolchain limitations.

Can you please outline why this (in my opinion, big) tradeoff is worth it? Just curious. :)

Best regards,
Marvin

> On 13. Apr 2022, at 03:05, Ada Christine <adachristine18@gmail.com> wrote:
> Hello, edk2 developers!
> 
> I've registered as a contributor candidate for GSoC 2022 and am interested in working on one of the items from the Tasks list here https://github.com/tianocore/tianocore.github.io/wiki/Tasks. Specifically, adding dynamic linking support caught my attention as this is something i've been investigating and learning more about in one of my own personal projects. As a little background, my personal project is an experiment in OS development and I use a very small subset of the boot services to get started and loaded. It can be found here: https://github.com/adachristine/sophia. Recently I've started investigating (and begun to implement) using ELF's dynamic facilities to dynamically load kernel modules. I know PE is slightly different to ELF, but the principles seem similar enough.
> 
> I've had a few glances at the EDKII source code in the past and have a general idea of how it all fits together. What I have in mind to implement this would be the following:
> - create a dynamic linker as a module package to be compiled into the main application
>  (alternatively, implement dynamic linking as a runtime service driver?)
> - adjust the build system to enable building as DLLs and dynamic linking of module packages to the main application
>  (module packages could be per-application and optionally site packages in a subdirectory of the ESP?)
> 
> I know the details of how this would all fit together are a little more involved, but this is just the rough first idea that came to my mind. Happy to hear feedback, and if my idea seems feasible I can get to work on a more in-depth plan to put this together.
> 
> Thanks!
> 
> - Ada
> 
> 
> 

Re: [edk2-discuss] GSoC Proposal

[Rebecca Cran]

On 4/13/22 00:54, Marvin Häuser wrote:
> CC edk2-devel (you picked the logically correct list, but it’s pretty dead and barely anyone reads it)
Just a minor correction - it's edk2-discuss (discuss@edk2.groups.io) 
that was cc'd but that's pretty dead.
edk2-devel (devel@edk2.groups.io) is much more active!

-- 
Rebecca Cran

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Rebecca Cran]

Oh I see what was meant - ignore me!
I do think it would be good to have more discussions on edk2-discuss 
though :)

-- 
Rebecca Cran

On 4/13/22 08:44, Rebecca Cran wrote:
> On 4/13/22 00:54, Marvin Häuser wrote:
>> CC edk2-devel (you picked the logically correct list, but it’s pretty 
>> dead and barely anyone reads it)
> Just a minor correction - it's edk2-discuss (discuss@edk2.groups.io) 
> that was cc'd but that's pretty dead.
> edk2-devel (devel@edk2.groups.io) is much more active!
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 6741 bytes --]

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI as a
complete operating system, but to try to eliminate the static linking that
may be needlessly duplicating
code that could instead be put in a single dynamic library. For instance,
MdePkg and MdeModulePkg are linked into a *lot* of .efi, instead of being
just a library. It'd be nice to see some
numbers on this (something like Google's bloaty could be run on every .efi
file, in order to understand how much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser <mhaeuser@posteo.de> wrote:

>
> On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@gmail.com> wrote:
>
> i was replying via the groups.io web interface, I'm guessing that messed
> up
> the thread? i haven't used mailing lists before and don't know how they
> work. I'll use my mail client from here on.
>
> I'm on board with not treating EFI as an operating system. the more i think
> about it the more it looks like scope creep.
>
>
> Agreed.
>
> I'm not quite as enthusiastic
> about it as i was at first glance.
>
> I'm still keen on doing my gsoc proposal for edk, though, and even if this
> task and the acpica application are decided to be out of scope unit
> testing,
>
>
> How about fuzz-testing? This is also something edk2 needs quite badly. At
> Acidanthera, we compile edk2 code in userspace outside the edk2 build
> system and fuzz with dummy applications.
>
> Note: fuzzing is also part of the LLVM instrumentation suite (see
https://llvm.org/docs/LibFuzzer.html) and is something I could happily
mentor.

> clang integration
>
>
> Pedro and Vitaly are looking for someone to finish ASan:
> https://edk2.groups.io/g/devel/topic/90010978#87991
> There are working UBSan concepts, but they also need to be mainlined.
>

Is Vitaly going to be a mentor? I was assuming it was going to be me and
some other, more senior, mentor (possibly Steven Shi, which I included in
the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly some
other sanitizer it's quite possible that it could be considered a large
project (which means more hours but a larger stipend too). Fuzzing +
coverage could
be very nice additions to this project idea.
Also, is stress-testing a decent idea?

>
> and source-level debugging are all relevant to
> my interests.
>
> how about your ideas for security stuff?
>
>
> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work
> on MMRAM. Also evaluation of how to best avoid pointers in MM communication
> buffers would be nice.
>
> There also is a bunch of other stuff, like working out moving a part of
> CpuDxe into DxeCore to have memory protection live immediately, memory
> protection in PEI, a replacement for the TE format (it’s buggy and most
> platforms mostly abandoned it over various issues), and alternatives to
> guarding critical code with SMM (like allowing NVRAM commits only as part
> of a reboot).
>
> I personally find all of those projects very important, but I cannot
> promise many people agree. Especially those that impose global changes
> (most notably the TE replacement) may be very tedious to submit. Gladly, I
> believe you can submit multiple proposals (?)
>
> Best regards,
> Marvin
>
> I'm not very knowledgeable about
> trusted platform or secure boot but I'm willing to learn whatever is
> necessary to get something spun up for my proposal.
>
> On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@posteo.de> wrote:
>
> Do you use the “reply all” option in your mail client? Looks like my CCs
>
> have been dropped again. Comments inline.
>
>
> On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@gmail.com>
>
> wrote:
>
> Hi, Marvin
>
>
> Its similarity to my own latest experiment is the key to what grabbed my
>
> attention. I have no particular use case in mind for it, but I see its
>
> potential for anybody developing larger applications in that when a library
>
> is changed there's no need to distribute a new version of the whole binary,
>
> just the relevant library module.
>
>
> I really do not like the trend of treating UEFI as a full-fledged OS - it
>
> is not. The most used UEFI applications, OS loaders, are really not that
>
> huge and are distributed as part of the OS image anyway. Even for less used
>
> applications, you will always get a full snapshot anyhow. Gladly we don’t
>
> have auto-update and package management yet. :)
>
>
>
> I slept on it and it occurred to me that the whole thing could operate
>
> similarly to the shell protocol in that the linker/loader is itself an
>
> application that does a LoadImage() on the application needing dynamic
>
> linking facilities.
>
>
> That would mean the linker itself is shipped with every application that
>
> requires it? Otherwise it doesn’t make much sense for it to be an app and
>
> below’s problems apply.
>
>
> If however the whole plan is making the linker as a DXE and including it
>
> with the firmware, that I'm not quite as sure about. That would necessarily
>
> tie any applications using dynamic linking to TianoCore or any firmware
>
> distribution that derives from it.
>
>
> I think that was the idea referred to as “edk2 core” by Steven, but I’d
>
> like to hear his proposal to be sure. Virtually everyone uses edk2, so that
>
> itself is not the problem, but versioning is. Vendors are slow to update
>
> their snapshots or have just given up doing that entirely. Distributing it
>
> for external applications like OS loaders would mean this can be leveraged
>
> probably no earlier than 10 years from now. And for in-firmware things, I
>
> have a hard time thinking about a use-case that outweighs the drawbacks.
>
>
>
> To shift the topic slightly back to GSoC, however, I'm willing to work
>
> on other items on the task list. Unit testing and an ACPICA application are
>
> the alternative projects I had thought about. I need to choose fairly soon
>
> as the proposal deadline is next Tuesday. I know a tiny bit about porting
>
> ACPICA as I also have plans to incorporate it into my own project.
>
>
> I have a few more ideas for security stuff, but Nate did not confirm them
>
> as appropriate yet and I’m not here to drive you away from this specific
>
> task (or the others). However, I’m still curious and concerned. :)
>
>
> Best regards,
>
> Marvin
>
>
>
>
>
>
>
> 
>
>

-- 
Pedro Falcato

[-- Attachment #2: Type: text/html, Size: 12050 bytes --]

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 6445 bytes --]

GSoC tip: Try to spread your proposals out. If we don't have enough project
slots for you, all your 3 proposals get rejected and your GSoC
participation goes out the window.
Take a look at our project ideas and ask questions, and take a look at
other orgs if you want. Also, don't forget that these project ideas are
just suggestions, and you're free (and encouraged) to
build on top of them, or write your own proposal from scratch.

Best regards,
Pedro

On Wed, Apr 13, 2022 at 6:53 PM Ada Christine <adachristine18@gmail.com>
wrote:

> i can submit up to three proposals. I'll give all of this some thought and
> a little research and start putting at least two proposals together
> starting Friday. :)
>
> On Wed, Apr 13, 2022, 15:15 Marvin Häuser <mhaeuser@posteo.de> wrote:
>
>>
>> On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@gmail.com>
>> wrote:
>>
>> i was replying via the groups.io web interface, I'm guessing that
>> messed up
>> the thread? i haven't used mailing lists before and don't know how they
>> work. I'll use my mail client from here on.
>>
>> I'm on board with not treating EFI as an operating system. the more i
>> think
>> about it the more it looks like scope creep.
>>
>>
>> Agreed.
>>
>> I'm not quite as enthusiastic
>> about it as i was at first glance.
>>
>> I'm still keen on doing my gsoc proposal for edk, though, and even if this
>> task and the acpica application are decided to be out of scope unit
>> testing,
>>
>>
>> How about fuzz-testing? This is also something edk2 needs quite badly. At
>> Acidanthera, we compile edk2 code in userspace outside the edk2 build
>> system and fuzz with dummy applications.
>>
>> clang integration
>>
>>
>> Pedro and Vitaly are looking for someone to finish ASan:
>> https://edk2.groups.io/g/devel/topic/90010978#87991
>> There are working UBSan concepts, but they also need to be mainlined.
>>
>> and source-level debugging are all relevant to
>> my interests.
>>
>> how about your ideas for security stuff?
>>
>>
>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work
>> on MMRAM. Also evaluation of how to best avoid pointers in MM communication
>> buffers would be nice.
>>
>> There also is a bunch of other stuff, like working out moving a part of
>> CpuDxe into DxeCore to have memory protection live immediately, memory
>> protection in PEI, a replacement for the TE format (it’s buggy and most
>> platforms mostly abandoned it over various issues), and alternatives to
>> guarding critical code with SMM (like allowing NVRAM commits only as part
>> of a reboot).
>>
>> I personally find all of those projects very important, but I cannot
>> promise many people agree. Especially those that impose global changes
>> (most notably the TE replacement) may be very tedious to submit. Gladly, I
>> believe you can submit multiple proposals (?)
>>
>> Best regards,
>> Marvin
>>
>> I'm not very knowledgeable about
>> trusted platform or secure boot but I'm willing to learn whatever is
>> necessary to get something spun up for my proposal.
>>
>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@posteo.de> wrote:
>>
>> Do you use the “reply all” option in your mail client? Looks like my CCs
>>
>> have been dropped again. Comments inline.
>>
>>
>> On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@gmail.com>
>>
>> wrote:
>>
>> Hi, Marvin
>>
>>
>> Its similarity to my own latest experiment is the key to what grabbed my
>>
>> attention. I have no particular use case in mind for it, but I see its
>>
>> potential for anybody developing larger applications in that when a
>> library
>>
>> is changed there's no need to distribute a new version of the whole
>> binary,
>>
>> just the relevant library module.
>>
>>
>> I really do not like the trend of treating UEFI as a full-fledged OS - it
>>
>> is not. The most used UEFI applications, OS loaders, are really not that
>>
>> huge and are distributed as part of the OS image anyway. Even for less
>> used
>>
>> applications, you will always get a full snapshot anyhow. Gladly we don’t
>>
>> have auto-update and package management yet. :)
>>
>>
>>
>> I slept on it and it occurred to me that the whole thing could operate
>>
>> similarly to the shell protocol in that the linker/loader is itself an
>>
>> application that does a LoadImage() on the application needing dynamic
>>
>> linking facilities.
>>
>>
>> That would mean the linker itself is shipped with every application that
>>
>> requires it? Otherwise it doesn’t make much sense for it to be an app and
>>
>> below’s problems apply.
>>
>>
>> If however the whole plan is making the linker as a DXE and including it
>>
>> with the firmware, that I'm not quite as sure about. That would
>> necessarily
>>
>> tie any applications using dynamic linking to TianoCore or any firmware
>>
>> distribution that derives from it.
>>
>>
>> I think that was the idea referred to as “edk2 core” by Steven, but I’d
>>
>> like to hear his proposal to be sure. Virtually everyone uses edk2, so
>> that
>>
>> itself is not the problem, but versioning is. Vendors are slow to update
>>
>> their snapshots or have just given up doing that entirely. Distributing it
>>
>> for external applications like OS loaders would mean this can be leveraged
>>
>> probably no earlier than 10 years from now. And for in-firmware things, I
>>
>> have a hard time thinking about a use-case that outweighs the drawbacks.
>>
>>
>>
>> To shift the topic slightly back to GSoC, however, I'm willing to work
>>
>> on other items on the task list. Unit testing and an ACPICA application
>> are
>>
>> the alternative projects I had thought about. I need to choose fairly soon
>>
>> as the proposal deadline is next Tuesday. I know a tiny bit about porting
>>
>> ACPICA as I also have plans to incorporate it into my own project.
>>
>>
>> I have a few more ideas for security stuff, but Nate did not confirm them
>>
>> as appropriate yet and I’m not here to drive you away from this specific
>>
>> task (or the others). However, I’m still curious and concerned. :)
>>
>>
>> Best regards,
>>
>> Marvin
>>
>>
>>
>>
>>
>>
>>
>> 
>
>

-- 
Pedro Falcato

[-- Attachment #2: Type: text/html, Size: 11370 bytes --]

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

[-- Attachment #1: Type: text/plain, Size: 7696 bytes --]


> On 13. Apr 2022, at 20:43, Pedro Falcato <pedro.falcato@gmail.com> wrote:
> 
> 
> Hi Marvin, Ada,
> 
> Some comments:
> 
> I don't think the purpose of the dynamic linker is to treat EFI as a complete operating system, but to try to eliminate the static linking that may be needlessly duplicating
> code that could instead be put in a single dynamic library.

Which is already possible as part of protocols. In fact, there are such for string and device path operations. What about that model justifies a fundamental change?

> For instance, MdePkg and MdeModulePkg are linked into a *lot* of .efi, instead of being just a library. It'd be nice to see some
> numbers on this (something like Google's bloaty could be run on every .efi file, in order to understand how much file space we would actually save).

The worst issue is PEI because of XIP. This should be out of scope anyway. Also needs numbers of how much new space is occupied by the loader duplicated into DXE and SMM.

> 
> Other comments inline.
> 
>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser <mhaeuser@posteo.de> wrote:
>> 
>>>> On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@gmail.com> wrote:
>>>> 
>>> i was replying via the groups.io web interface, I'm guessing that messed up
>>> the thread? i haven't used mailing lists before and don't know how they
>>> work. I'll use my mail client from here on.
>>> 
>>> I'm on board with not treating EFI as an operating system. the more i think
>>> about it the more it looks like scope creep.
>> 
>> Agreed.
>> 
>>> I'm not quite as enthusiastic
>>> about it as i was at first glance.
>>> 
>>> I'm still keen on doing my gsoc proposal for edk, though, and even if this
>>> task and the acpica application are decided to be out of scope unit
>>> testing,
>> 
>> How about fuzz-testing? This is also something edk2 needs quite badly. At Acidanthera, we compile edk2 code in userspace outside the edk2 build system and fuzz with dummy applications.
>> 
> Note: fuzzing is also part of the LLVM instrumentation suite (see https://llvm.org/docs/LibFuzzer.html) and is something I could happily mentor.

Yep :)

>>> clang integration
>> 
>> Pedro and Vitaly are looking for someone to finish ASan: https://edk2.groups.io/g/devel/topic/90010978#87991
>> There are working UBSan concepts, but they also need to be mainlined.
> 
> Is Vitaly going to be a mentor? I was assuming it was going to be me and some other, more senior, mentor (possibly Steven Shi, which I included in the task).

Looking was not specifically meant for mentoring, mostly he just wants it done. He offered to mentor after we saw that every Meeting Minutes mail asked for new mentors. Albeit his little activity on the list, I can assure you he is plenty senior for about any task in the list. :)

> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly some other sanitizer it's quite possible that it could be considered a large project (which means more hours but a larger stipend too). Fuzzing + coverage could
> be very nice additions to this project idea.
> Also, is stress-testing a decent idea?

Literally any kind of sophisticated testing… please

Best regards,
Marvin

>> 
>>> and source-level debugging are all relevant to
>>> my interests.
>>> 
>>> how about your ideas for security stuff?
>> 
>> I want the entirety of MM to leverage SmmMemLib and to support SMAP. SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work on MMRAM. Also evaluation of how to best avoid pointers in MM communication buffers would be nice.
>> 
>> There also is a bunch of other stuff, like working out moving a part of CpuDxe into DxeCore to have memory protection live immediately, memory protection in PEI, a replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues), and alternatives to guarding critical code with SMM (like allowing NVRAM commits only as part of a reboot).
>> 
>> I personally find all of those projects very important, but I cannot promise many people agree. Especially those that impose global changes (most notably the TE replacement) may be very tedious to submit. Gladly, I believe you can submit multiple proposals (?)
>> 
>> Best regards,
>> Marvin
>> 
>>> I'm not very knowledgeable about
>>> trusted platform or secure boot but I'm willing to learn whatever is
>>> necessary to get something spun up for my proposal.
>>> 
>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@posteo.de> wrote:
>>>> 
>>>> Do you use the “reply all” option in your mail client? Looks like my CCs
>>>> have been dropped again. Comments inline.
>>>> 
>>>>> On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@gmail.com>
>>>> wrote:
>>>>> Hi, Marvin
>>>>> 
>>>>> Its similarity to my own latest experiment is the key to what grabbed my
>>>> attention. I have no particular use case in mind for it, but I see its
>>>> potential for anybody developing larger applications in that when a library
>>>> is changed there's no need to distribute a new version of the whole binary,
>>>> just the relevant library module.
>>>> 
>>>> I really do not like the trend of treating UEFI as a full-fledged OS - it
>>>> is not. The most used UEFI applications, OS loaders, are really not that
>>>> huge and are distributed as part of the OS image anyway. Even for less used
>>>> applications, you will always get a full snapshot anyhow. Gladly we don’t
>>>> have auto-update and package management yet. :)
>>>> 
>>>>> 
>>>>> I slept on it and it occurred to me that the whole thing could operate
>>>> similarly to the shell protocol in that the linker/loader is itself an
>>>> application that does a LoadImage() on the application needing dynamic
>>>> linking facilities.
>>>> 
>>>> That would mean the linker itself is shipped with every application that
>>>> requires it? Otherwise it doesn’t make much sense for it to be an app and
>>>> below’s problems apply.
>>>> 
>>>>> If however the whole plan is making the linker as a DXE and including it
>>>> with the firmware, that I'm not quite as sure about. That would necessarily
>>>> tie any applications using dynamic linking to TianoCore or any firmware
>>>> distribution that derives from it.
>>>> 
>>>> I think that was the idea referred to as “edk2 core” by Steven, but I’d
>>>> like to hear his proposal to be sure. Virtually everyone uses edk2, so that
>>>> itself is not the problem, but versioning is. Vendors are slow to update
>>>> their snapshots or have just given up doing that entirely. Distributing it
>>>> for external applications like OS loaders would mean this can be leveraged
>>>> probably no earlier than 10 years from now. And for in-firmware things, I
>>>> have a hard time thinking about a use-case that outweighs the drawbacks.
>>>> 
>>>>> 
>>>>> To shift the topic slightly back to GSoC, however, I'm willing to work
>>>> on other items on the task list. Unit testing and an ACPICA application are
>>>> the alternative projects I had thought about. I need to choose fairly soon
>>>> as the proposal deadline is next Tuesday. I know a tiny bit about porting
>>>> ACPICA as I also have plans to incorporate it into my own project.
>>>> 
>>>> I have a few more ideas for security stuff, but Nate did not confirm them
>>>> as appropriate yet and I’m not here to drive you away from this specific
>>>> task (or the others). However, I’m still curious and concerned. :)
>>>> 
>>>> Best regards,
>>>> Marvin
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> -- 
> Pedro Falcato

[-- Attachment #2: Type: text/html, Size: 13795 bytes --]

FW: [edk2-discuss] GSoC Proposal

[Nate DeSimone]

Hi Ada,

Great to meet you and welcome to the TianoCore project! Great to hear you are interested! Despite Marvin's misgivings, I think dynamic linking would be an excellent addition to EDK II! Marvin is right that we would not want to use it in UEFI spec compliant applications or OpROMs at least for now, but even if you make the assumption that this feature would only be used with EDK II it is still valuable. The primary value I see is reducing the size of BIOS images. Every DXE driver and PEIM includes a lot of statically linked libraries from MdePkg and/or MdeModulePkg depending on the exact driver and build configuration, and it adds up very quickly. On debug builds, the infrastructure for debug messages becomes particularly large. Every PEIM/DXE driver needs to have a copy of the parser for printf() style format strings. The overhead works out to ~12KB for every PEIM and DXE driver. Most BIOS images have ~250 DXE drivers and ~30 PEIMs these days. Uncompressed that works out to >3MB of overhead. The DXE drivers are typically compressed, but LZMA isn't perfect at deduplication and as optimizing compliers have gotten better the deduplication has gotten worse, which is causing this to become more important over time. I did some experiments with the Intel reference BIOS and I found that we can save roughly 1.5-2MB of space with dynamic linking, which is >10% of the flash budget. For PEI where there is no compression, the flash budget savings is >20%, and it has the extra benefit of reducing usage of precious NEM memory. Another thing to consider is that linking OpenSSL for doing code signing checks add ~65KB to the size of the PEIM doing the signature check. When you consider that moving PEI from 32-bit to 64-bit is going to increase code size by ~20%, this feature is extremely valuable for many reasons.

More forward looking, if the project to add Rust is successful then suddenly the rich set of libraries available in Rust crates becomes available to EDK II. Those are going to be big and the only way we will be able to use them is with some deduplication facility like dynamic linking. That will require having some way of using type safe Rust objects across driver boundaries... which is completely out of scope for this GSoC project, but at the same time this GSoC project would be a necessary prerequisite before we could even start thinking about that 😊.

I agree with Marvin that PE/COFF should be the preferred starting point. I believe ELF to be a security risk. It looks easy to parse on the surface but it has been well documented that the devil is in the details and as the CVE history shows it is very easy to build a ELF parser with unintended overflows and other security vulnerabilities. Perhaps most worrying to me is that I don't believe the security community has done enough research on ELF yet, whereas PE/COFF has been very heavily researched and has a mature literature.

Let me know if any of this piques your interest, I would be happy to answer any questions you have!

Hope this helps and welcome to the project!

With Best Regards,
Nate

> -----Original Message-----
> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Ada 
> Christine
> Sent: Tuesday, April 12, 2022 11:09 AM
> To: discuss@edk2.groups.io
> Subject: [edk2-discuss] GSoC Proposal
> 
> Hello, edk2 developers!
> 
> I've registered as a contributor candidate for GSoC 2022 and am 
> interested in working on one of the items from the Tasks list here 
> https://github.com/tianocore/tianocore.github.io/wiki/Tasks. 
> Specifically, adding dynamic linking support caught my attention as 
> this is something i've been investigating and learning more about in 
> one of my own personal projects. As a little background, my personal 
> project is an experiment in OS development and I use a very small 
> subset of the boot services to get started and loaded. It can be found here: https://github.com/adachristine/sophia.
> Recently I've started investigating (and begun to implement) using 
> ELF's dynamic facilities to dynamically load kernel modules. I know PE 
> is slightly different to ELF, but the principles seem similar enough.
> 
> I've had a few glances at the EDKII source code in the past and have a 
> general idea of how it all fits together. What I have in mind to 
> implement this would be the following:
> - create a dynamic linker as a module package to be compiled into the 
> main application
>   (alternatively, implement dynamic linking as a runtime service 
> driver?)
> - adjust the build system to enable building as DLLs and dynamic 
> linking of module packages to the main application
>   (module packages could be per-application and optionally site 
> packages in a subdirectory of the ESP?)
> 
> I know the details of how this would all fit together are a little 
> more involved, but this is just the rough first idea that came to my 
> mind. Happy to hear feedback, and if my idea seems feasible I can get 
> to work on a more in-depth plan to put this together.
> 
> Thanks!
> 
> - Ada
> 
> 
> 
> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Nate DeSimone]

[-- Attachment #1: Type: text/plain, Size: 7403 bytes --]

Hi All,

Pedro is 100% correct. The primary use case and the reason I added this is to remove library duplication across all the .efi files. This is actually super valuable because LZMA compression is becoming ineffective because compiler optimization is getting so good that the patterns for a library across binaries don’t match very well anymore. For XIP PEI code… it will really help and would be very timely since the transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by ~20%.

Thanks,
Nate

From: Pedro Falcato <pedro.falcato@gmail.com>
Sent: Wednesday, April 13, 2022 11:43 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser <mhaeuser@posteo.de>
Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven <steven.shi@intel.com>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI as a complete operating system, but to try to eliminate the static linking that may be needlessly duplicating
code that could instead be put in a single dynamic library. For instance, MdePkg and MdeModulePkg are linked into a *lot* of .efi, instead of being just a library. It'd be nice to see some
numbers on this (something like Google's bloaty could be run on every .efi file, in order to understand how much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:

On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
i was replying via the groups.io<http://groups.io> web interface, I'm guessing that messed up
the thread? i haven't used mailing lists before and don't know how they
work. I'll use my mail client from here on.

I'm on board with not treating EFI as an operating system. the more i think
about it the more it looks like scope creep.

Agreed.


I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and even if this
task and the acpica application are decided to be out of scope unit
testing,

How about fuzz-testing? This is also something edk2 needs quite badly. At Acidanthera, we compile edk2 code in userspace outside the edk2 build system and fuzz with dummy applications.

Note: fuzzing is also part of the LLVM instrumentation suite (see https://llvm.org/docs/LibFuzzer.html) and is something I could happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan: https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be me and some other, more senior, mentor (possibly Steven Shi, which I included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly some other sanitizer it's quite possible that it could be considered a large project (which means more hours but a larger stipend too). Fuzzing + coverage could
be very nice additions to this project idea.
Also, is stress-testing a decent idea?


and source-level debugging are all relevant to
my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support SMAP. SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work on MMRAM. Also evaluation of how to best avoid pointers in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a part of CpuDxe into DxeCore to have memory protection live immediately, memory protection in PEI, a replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues), and alternatives to guarding critical code with SMM (like allowing NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I cannot promise many people agree. Especially those that impose global changes (most notably the TE replacement) may be very tedious to submit. Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin


I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever is
necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:


Do you use the “reply all” option in your mail client? Looks like my CCs
have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what grabbed my
attention. I have no particular use case in mind for it, but I see its
potential for anybody developing larger applications in that when a library
is changed there's no need to distribute a new version of the whole binary,
just the relevant library module.

I really do not like the trend of treating UEFI as a full-fledged OS - it
is not. The most used UEFI applications, OS loaders, are really not that
huge and are distributed as part of the OS image anyway. Even for less used
applications, you will always get a full snapshot anyhow. Gladly we don’t
have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could operate
similarly to the shell protocol in that the linker/loader is itself an
application that does a LoadImage() on the application needing dynamic
linking facilities.

That would mean the linker itself is shipped with every application that
requires it? Otherwise it doesn’t make much sense for it to be an app and
below’s problems apply.

If however the whole plan is making the linker as a DXE and including it
with the firmware, that I'm not quite as sure about. That would necessarily
tie any applications using dynamic linking to TianoCore or any firmware
distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but I’d
like to hear his proposal to be sure. Virtually everyone uses edk2, so that
itself is not the problem, but versioning is. Vendors are slow to update
their snapshots or have just given up doing that entirely. Distributing it
for external applications like OS loaders would mean this can be leveraged
probably no earlier than 10 years from now. And for in-firmware things, I
have a hard time thinking about a use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to work
on other items on the task list. Unit testing and an ACPICA application are
the alternative projects I had thought about. I need to choose fairly soon
as the proposal deadline is next Tuesday. I know a tiny bit about porting
ACPICA as I also have plans to incorporate it into my own project.

I have a few more ideas for security stuff, but Nate did not confirm them
as appropriate yet and I’m not here to drive you away from this specific
task (or the others). However, I’m still curious and concerned. :)

Best regards,
Marvin








--
Pedro Falcato

[-- Attachment #2: Type: text/html, Size: 20632 bytes --]

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Ada Christine]

[-- Attachment #1: Type: text/plain, Size: 8153 bytes --]

i understand now, thank you for clarifying! I'm on board and I'll get to
work on a more complete plan as soon as possible. i can probably have a
draft proposal done by EOD Saturday. i know it's a holiday weekend, so i
don't expect anybody to hop to (pun intended) to critique :)

On Wed, Apr 13, 2022, 22:57 Nate DeSimone <nathaniel.l.desimone@intel.com>
wrote:

> Hi All,
>
> Pedro is 100% correct. The primary use case and the reason I added this is
> to remove library duplication across all the .efi files. This is actually
> super valuable because LZMA compression is becoming ineffective because
> compiler optimization is getting so good that the patterns for a library
> across binaries don’t match very well anymore. For XIP PEI code… it will
> really help and would be very timely since the transition of PEI from
> 32-bit to 64-bit is going to increase the size of PEI by ~20%.
>
> Thanks,
> Nate
>
> From: Pedro Falcato <pedro.falcato@gmail.com>
> Sent: Wednesday, April 13, 2022 11:43 AM
> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser <
> mhaeuser@posteo.de>
> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone, Nathaniel
> L <nathaniel.l.desimone@intel.com>; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>
> Hi Marvin, Ada,
>
> Some comments:
>
> I don't think the purpose of the dynamic linker is to treat EFI as a
> complete operating system, but to try to eliminate the static linking that
> may be needlessly duplicating
> code that could instead be put in a single dynamic library. For instance,
> MdePkg and MdeModulePkg are linked into a *lot* of .efi, instead of being
> just a library. It'd be nice to see some
> numbers on this (something like Google's bloaty could be run on every .efi
> file, in order to understand how much file space we would actually save).
>
> Other comments inline.
>
> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser <mhaeuser@posteo.de<mailto:
> mhaeuser@posteo.de>> wrote:
>
> On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@gmail.com<mailto:
> adachristine18@gmail.com>> wrote:
> i was replying via the groups.io<http://groups.io> web interface, I'm
> guessing that messed up
> the thread? i haven't used mailing lists before and don't know how they
> work. I'll use my mail client from here on.
>
> I'm on board with not treating EFI as an operating system. the more i think
> about it the more it looks like scope creep.
>
> Agreed.
>
>
> I'm not quite as enthusiastic
> about it as i was at first glance.
>
> I'm still keen on doing my gsoc proposal for edk, though, and even if this
> task and the acpica application are decided to be out of scope unit
> testing,
>
> How about fuzz-testing? This is also something edk2 needs quite badly. At
> Acidanthera, we compile edk2 code in userspace outside the edk2 build
> system and fuzz with dummy applications.
>
> Note: fuzzing is also part of the LLVM instrumentation suite (see
> https://llvm.org/docs/LibFuzzer.html) and is something I could happily
> mentor.
> clang integration
>
> Pedro and Vitaly are looking for someone to finish ASan:
> https://edk2.groups.io/g/devel/topic/90010978#87991
> There are working UBSan concepts, but they also need to be mainlined.
>
> Is Vitaly going to be a mentor? I was assuming it was going to be me and
> some other, more senior, mentor (possibly Steven Shi, which I included in
> the task).
> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly some
> other sanitizer it's quite possible that it could be considered a large
> project (which means more hours but a larger stipend too). Fuzzing +
> coverage could
> be very nice additions to this project idea.
> Also, is stress-testing a decent idea?
>
>
> and source-level debugging are all relevant to
> my interests.
>
> how about your ideas for security stuff?
>
> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work
> on MMRAM. Also evaluation of how to best avoid pointers in MM communication
> buffers would be nice.
>
> There also is a bunch of other stuff, like working out moving a part of
> CpuDxe into DxeCore to have memory protection live immediately, memory
> protection in PEI, a replacement for the TE format (it’s buggy and most
> platforms mostly abandoned it over various issues), and alternatives to
> guarding critical code with SMM (like allowing NVRAM commits only as part
> of a reboot).
>
> I personally find all of those projects very important, but I cannot
> promise many people agree. Especially those that impose global changes
> (most notably the TE replacement) may be very tedious to submit. Gladly, I
> believe you can submit multiple proposals (?)
>
> Best regards,
> Marvin
>
>
> I'm not very knowledgeable about
> trusted platform or secure boot but I'm willing to learn whatever is
> necessary to get something spun up for my proposal.
>
> On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@posteo.de<mailto:
> mhaeuser@posteo.de>> wrote:
>
>
> Do you use the “reply all” option in your mail client? Looks like my CCs
> have been dropped again. Comments inline.
>
> On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@gmail.com<mailto:
> adachristine18@gmail.com>>
> wrote:
> Hi, Marvin
>
> Its similarity to my own latest experiment is the key to what grabbed my
> attention. I have no particular use case in mind for it, but I see its
> potential for anybody developing larger applications in that when a library
> is changed there's no need to distribute a new version of the whole binary,
> just the relevant library module.
>
> I really do not like the trend of treating UEFI as a full-fledged OS - it
> is not. The most used UEFI applications, OS loaders, are really not that
> huge and are distributed as part of the OS image anyway. Even for less used
> applications, you will always get a full snapshot anyhow. Gladly we don’t
> have auto-update and package management yet. :)
>
>
> I slept on it and it occurred to me that the whole thing could operate
> similarly to the shell protocol in that the linker/loader is itself an
> application that does a LoadImage() on the application needing dynamic
> linking facilities.
>
> That would mean the linker itself is shipped with every application that
> requires it? Otherwise it doesn’t make much sense for it to be an app and
> below’s problems apply.
>
> If however the whole plan is making the linker as a DXE and including it
> with the firmware, that I'm not quite as sure about. That would necessarily
> tie any applications using dynamic linking to TianoCore or any firmware
> distribution that derives from it.
>
> I think that was the idea referred to as “edk2 core” by Steven, but I’d
> like to hear his proposal to be sure. Virtually everyone uses edk2, so that
> itself is not the problem, but versioning is. Vendors are slow to update
> their snapshots or have just given up doing that entirely. Distributing it
> for external applications like OS loaders would mean this can be leveraged
> probably no earlier than 10 years from now. And for in-firmware things, I
> have a hard time thinking about a use-case that outweighs the drawbacks.
>
>
> To shift the topic slightly back to GSoC, however, I'm willing to work
> on other items on the task list. Unit testing and an ACPICA application are
> the alternative projects I had thought about. I need to choose fairly soon
> as the proposal deadline is next Tuesday. I know a tiny bit about porting
> ACPICA as I also have plans to incorporate it into my own project.
>
> I have a few more ideas for security stuff, but Nate did not confirm them
> as appropriate yet and I’m not here to drive you away from this specific
> task (or the others). However, I’m still curious and concerned. :)
>
> Best regards,
> Marvin
>
>
>
>
>
>
>
>
> --
> Pedro Falcato
>
>
> 
>
>
>

[-- Attachment #2: Type: text/html, Size: 10316 bytes --]

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

> On 14. Apr 2022, at 00:57, Nate DeSimone <nathaniel.l.desimone@intel.com> wrote:
> 
> Hi All,
> 
> Pedro is 100% correct. The primary use case and the reason I added this is to remove library duplication across all the .efi files. This is actually super valuable because LZMA compression is becoming ineffective because compiler optimization is getting so good that the patterns for a library across binaries don’t match very well anymore.

I feel like there are so many much easier solutions to this problem that are at most limited by the clear specification. The UEFI specification with regards to booting and all of that obviously is of utmost importance. The UEFI PI specification parts that deal about internal structure, as far as I know, are only in place to make it easy to integrate Intel IP. In fact, I don’t *know*, but I’m pretty sure the very strict separation between PEI and DXE was preserved mostly because MRC was 32-bit-only for a long time. Glad that seems to have been resolved, AMD does memory init by PSP nowadays.

For many good reasons, Linux does not provide a stable kernel API. This allows to easily deploy breaking changes across the entire codebase. Obviously, this is infeasible at a large scale when you need to integrate IP blobs, but it would already help to break the expectation that UEFI PI is a perfectly forwards- and backwards-compatible system. DXE has SetMem and CopyMem as part of gBS. Maybe I don’t like it that much as part of the spec itself, but it’s a good idea technically. I’d probably opt to have a quickly accessible implementation detail of important function pointers appended to PEI and DXE services, used by in-tree modules. This may break both forwards- and backwards-compatibility, but as it only applies to in-tree modules, that is fine so long as we let go of the compatibility notions. PPIs and protocols are an option too of course, but they have a lookup performance penalty. Compared to dynamic linking, that should hopefully be negligible however.

Absolutely optional to read, I don’t intend to waste anyone’s time much, some philosophical stuff about my rationale:

If you started UEFI from scratch right now, would it have strictly separated PEI and DXE? The duplication between PEI and DXE core, and by extension MM core, would be my most obvious place to start reducing size. I would probably opt for a PEI/DXE hybrid where it starts in „minimal mode“ (maybe think of PEI more like a microkernel here) and after memory is initialised, the rest of DXE is loaded. And MM core would get no loading at all, this requirement has gladly been dropped ages ago. Just one prelinked snapshot of the address space with a relocation table and a safe self-relocator on entry (this is needed at the very least for ARM).

Ironically, with my idea for MM, dynamic loading would be free as everything is prelinked anyway. The same is true for PEI XIP, it is prelinked by nature. What I do not like is the additional dynamic linking code at load-time for non-XIP modules. Though, the more I think about it, the more I wonder whether not the entirety of UEFI could be composed of prelinked, relocatable memory snapshots without traditional image loading entirely (for in-FW stuff). macOS has a similar concept with its “Kernel Collections”. Well, way too much off-topic now. :)

Why am I explaining all this despite the fact everyone knows this will never happen? Because I don’t like the notion of fixing issues of an already overcomplicated system by adding even more complicated stuff. Especially when the existing overcomplicated stuff is already uncomfortably broken.

Best regards,
Marvin

> For XIP PEI code… it will really help and would be very timely since the transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by ~20%.
> 
> Thanks,
> Nate
> 
> From: Pedro Falcato <pedro.falcato@gmail.com>
> Sent: Wednesday, April 13, 2022 11:43 AM
> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser <mhaeuser@posteo.de>
> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> 
> Hi Marvin, Ada,
> 
> Some comments:
> 
> I don't think the purpose of the dynamic linker is to treat EFI as a complete operating system, but to try to eliminate the static linking that may be needlessly duplicating
> code that could instead be put in a single dynamic library. For instance, MdePkg and MdeModulePkg are linked into a *lot* of .efi, instead of being just a library. It'd be nice to see some
> numbers on this (something like Google's bloaty could be run on every .efi file, in order to understand how much file space we would actually save).
> 
> Other comments inline.
> 
> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> 
> On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
> i was replying via the groups.io<http://groups.io> web interface, I'm guessing that messed up
> the thread? i haven't used mailing lists before and don't know how they
> work. I'll use my mail client from here on.
> 
> I'm on board with not treating EFI as an operating system. the more i think
> about it the more it looks like scope creep.
> 
> Agreed.
> 
> 
> I'm not quite as enthusiastic
> about it as i was at first glance.
> 
> I'm still keen on doing my gsoc proposal for edk, though, and even if this
> task and the acpica application are decided to be out of scope unit
> testing,
> 
> How about fuzz-testing? This is also something edk2 needs quite badly. At Acidanthera, we compile edk2 code in userspace outside the edk2 build system and fuzz with dummy applications.
> 
> Note: fuzzing is also part of the LLVM instrumentation suite (see https://llvm.org/docs/LibFuzzer.html) and is something I could happily mentor.
> clang integration
> 
> Pedro and Vitaly are looking for someone to finish ASan: https://edk2.groups.io/g/devel/topic/90010978#87991
> There are working UBSan concepts, but they also need to be mainlined.
> 
> Is Vitaly going to be a mentor? I was assuming it was going to be me and some other, more senior, mentor (possibly Steven Shi, which I included in the task).
> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly some other sanitizer it's quite possible that it could be considered a large project (which means more hours but a larger stipend too). Fuzzing + coverage could
> be very nice additions to this project idea.
> Also, is stress-testing a decent idea?
> 
> 
> and source-level debugging are all relevant to
> my interests.
> 
> how about your ideas for security stuff?
> 
> I want the entirety of MM to leverage SmmMemLib and to support SMAP. SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work on MMRAM. Also evaluation of how to best avoid pointers in MM communication buffers would be nice.
> 
> There also is a bunch of other stuff, like working out moving a part of CpuDxe into DxeCore to have memory protection live immediately, memory protection in PEI, a replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues), and alternatives to guarding critical code with SMM (like allowing NVRAM commits only as part of a reboot).
> 
> I personally find all of those projects very important, but I cannot promise many people agree. Especially those that impose global changes (most notably the TE replacement) may be very tedious to submit. Gladly, I believe you can submit multiple proposals (?)
> 
> Best regards,
> Marvin
> 
> 
> I'm not very knowledgeable about
> trusted platform or secure boot but I'm willing to learn whatever is
> necessary to get something spun up for my proposal.
> 
> On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> 
> 
> Do you use the “reply all” option in your mail client? Looks like my CCs
> have been dropped again. Comments inline.
> 
> On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
> wrote:
> Hi, Marvin
> 
> Its similarity to my own latest experiment is the key to what grabbed my
> attention. I have no particular use case in mind for it, but I see its
> potential for anybody developing larger applications in that when a library
> is changed there's no need to distribute a new version of the whole binary,
> just the relevant library module.
> 
> I really do not like the trend of treating UEFI as a full-fledged OS - it
> is not. The most used UEFI applications, OS loaders, are really not that
> huge and are distributed as part of the OS image anyway. Even for less used
> applications, you will always get a full snapshot anyhow. Gladly we don’t
> have auto-update and package management yet. :)
> 
> 
> I slept on it and it occurred to me that the whole thing could operate
> similarly to the shell protocol in that the linker/loader is itself an
> application that does a LoadImage() on the application needing dynamic
> linking facilities.
> 
> That would mean the linker itself is shipped with every application that
> requires it? Otherwise it doesn’t make much sense for it to be an app and
> below’s problems apply.
> 
> If however the whole plan is making the linker as a DXE and including it
> with the firmware, that I'm not quite as sure about. That would necessarily
> tie any applications using dynamic linking to TianoCore or any firmware
> distribution that derives from it.
> 
> I think that was the idea referred to as “edk2 core” by Steven, but I’d
> like to hear his proposal to be sure. Virtually everyone uses edk2, so that
> itself is not the problem, but versioning is. Vendors are slow to update
> their snapshots or have just given up doing that entirely. Distributing it
> for external applications like OS loaders would mean this can be leveraged
> probably no earlier than 10 years from now. And for in-firmware things, I
> have a hard time thinking about a use-case that outweighs the drawbacks.
> 
> 
> To shift the topic slightly back to GSoC, however, I'm willing to work
> on other items on the task list. Unit testing and an ACPICA application are
> the alternative projects I had thought about. I need to choose fairly soon
> as the proposal deadline is next Tuesday. I know a tiny bit about porting
> ACPICA as I also have plans to incorporate it into my own project.
> 
> I have a few more ideas for security stuff, but Nate did not confirm them
> as appropriate yet and I’m not here to drive you away from this specific
> task (or the others). However, I’m still curious and concerned. :)
> 
> Best regards,
> Marvin
> 
> 
> 
> 
> 
> 
> 
> 
> --
> Pedro Falcato
> 
> 
> 
> 
> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Nate DeSimone]

Hi Marvin,

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin
> Häuser
> Sent: Thursday, April 14, 2022 12:56 AM
> To: discuss@edk2.groups.io; Desimone, Nathaniel L
> <nathaniel.l.desimone@intel.com>
> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
> <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> 
> I feel like there are so many much easier solutions to this problem that are at
> most limited by the clear specification. The UEFI specification with regards to
> booting and all of that obviously is of utmost importance.

If you have a better idea that retains compatibility with the existing UEFI PI then I would be happy to hear it. Ultimately anything we do needs to be a pure extension that retains compatibility with old code. Given that restriction having the ability to coalesce all the LibraryClasses into a single module and have cross-module linking seems like the best way to handle it to me.

> The UEFI PI specification parts that deal about internal structure, as far as I know, are
> only in place to make it easy to integrate Intel IP.

Its not just for Intel. The biggest reason for it to increase the standardization of the boot flow across the PC ecosystem. We have learned from experience that firmware is super critical to get a product out the door but it is also difficult to write. So we try to make it as reusable as humanly possible.

> In fact, I don’t *know*, but I’m pretty sure the very strict separation between PEI and DXE was
> preserved mostly because MRC was 32-bit-only for a long time. Glad that
> seems to have been resolved, AMD does memory init by PSP nowadays.

Having less complex early stages chain load more complex later stages is a common design pattern in firmware, not just UEFI. For example, your typical ARM system loads kinda like this:

PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor --> EDK II or U-Boot or LittleKernel (which runs android fastboot)

Comparing relative complexity I believe the Intel UEFI PI design is actually pretty simple when you consider how much it gets done:

Ucode ROM --> SEC + PEI --> DXE + SMM + BDS

My biggest criticism of the design is that the strict separation between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

There are a few reasons why PEI was 32-bit for quite some time. The biggest one is the code size increase, 64-bit x86 code is 20-30% larger than 32-bit x86 code. Since the only RAM Pre-Memory code has access to is the cache onboard the processor and for security reasons all that code has to fit within that RAM we generally do everything we can to make that image as small as possible. Second, 64-bit requires a page table and since we desired to keep PEI simple we tried to avoid that. Finally, the PI spec didn't allow a 64-bit PEI until recently. MRC is 32-bit code just because that is what PEI happens to be. Porting it to 64-bit is not terribly difficult.

Ultimately the mix of 32/64-bit does cause some difficulties. Any data structures that get shared between PEI and DXE (HOBs, PCDs, etc.) need to resolve to the same size and packing. LibraryClasses need to be written to compile properly in both modes. In the case of FSP API mode you need to resort to thunking between 32 and 64-bit modes during DXE. More or less we decided that the costs are starting to outweigh the advantages.

> 
> For many good reasons, Linux does not provide a stable kernel API. This
> allows to easily deploy breaking changes across the entire codebase.
> Obviously, this is infeasible at a large scale when you need to integrate IP
> blobs, but it would already help to break the expectation that UEFI PI is a
> perfectly forwards- and backwards-compatible system. DXE has SetMem and
> CopyMem as part of gBS. Maybe I don’t like it that much as part of the spec
> itself, but it’s a good idea technically. I’d probably opt to have a quickly
> accessible implementation detail of important function pointers appended to
> PEI and DXE services, used by in-tree modules. This may break both
> forwards- and backwards-compatibility, but as it only applies to in-tree
> modules, that is fine so long as we let go of the compatibility notions. PPIs
> and protocols are an option too of course, but they have a lookup
> performance penalty. Compared to dynamic linking, that should hopefully be
> negligible however.
> 
> Absolutely optional to read, I don’t intend to waste anyone’s time much,
> some philosophical stuff about my rationale:
> 
> If you started UEFI from scratch right now, would it have strictly separated
> PEI and DXE?

For sure a clean slate design started today would look a lot different than PEI/DXE... which was a clean slate design circa 1998 😊. In my opinion, if we were to suddenly go back to the drawing board we would build something that is much closer to a full OS now than we did back then. There have been cases where not being able to use interrupt handlers and not having thread scheduling has prevented implementation of desired features. The ARM guys built LittleKernel (https://github.com/littlekernel/lk) for a lot of these reasons. In the data center world some have decided to go to the extreme of putting an entire copy of Linux in SPI so they can do a network boot that downloads the OS image using BitTorrent!

> The duplication between PEI and DXE core, and by extension
> MM core, would be my most obvious place to start reducing size. I would
> probably opt for a PEI/DXE hybrid where it starts in „minimal mode“ (maybe
> think of PEI more like a microkernel here) and after memory is initialised, the
> rest of DXE is loaded. And MM core would get no loading at all, this
> requirement has gladly been dropped ages ago. Just one prelinked snapshot
> of the address space with a relocation table and a safe self-relocator on entry
> (this is needed at the very least for ARM).
> 
> Ironically, with my idea for MM, dynamic loading would be free as everything
> is prelinked anyway. The same is true for PEI XIP, it is prelinked by nature.

Actually Post-Memory PEI can have non-prelinked PEIMs. And that does get used for the PEI GOP driver.

> What I do not like is the additional dynamic linking code at load-time for non-
> XIP modules. Though, the more I think about it, the more I wonder whether
> not the entirety of UEFI could be composed of prelinked, relocatable
> memory snapshots without traditional image loading entirely (for in-FW
> stuff). macOS has a similar concept with its “Kernel Collections”. Well, way
> too much off-topic now. :)

If you make the assumption that 100% of the code is compiled all at once then yes that works. UEFI was designed so that assumption does not need to be true. There are good use cases for it: OpROMs, generic OS loaders, network boot, etc. 

Thanks,
Nate

> 
> Why am I explaining all this despite the fact everyone knows this will never
> happen? Because I don’t like the notion of fixing issues of an already
> overcomplicated system by adding even more complicated stuff. Especially
> when the existing overcomplicated stuff is already uncomfortably broken.
> 
> Best regards,
> Marvin
> 
> > For XIP PEI code… it will really help and would be very timely since the
> transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by
> ~20%.
> >
> > Thanks,
> > Nate
> >
> > From: Pedro Falcato <pedro.falcato@gmail.com>
> > Sent: Wednesday, April 13, 2022 11:43 AM
> > To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
> > <mhaeuser@posteo.de>
> > Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
> > Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
> > <steven.shi@intel.com>
> > Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> >
> > Hi Marvin, Ada,
> >
> > Some comments:
> >
> > I don't think the purpose of the dynamic linker is to treat EFI as a
> > complete operating system, but to try to eliminate the static linking
> > that may be needlessly duplicating code that could instead be put in a
> single dynamic library. For instance, MdePkg and MdeModulePkg are linked
> into a *lot* of .efi, instead of being just a library. It'd be nice to see some
> numbers on this (something like Google's bloaty could be run on every .efi
> file, in order to understand how much file space we would actually save).
> >
> > Other comments inline.
> >
> > On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> >
> > On 13. Apr 2022, at 16:38, Ada Christine
> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
> > i was replying via the groups.io<http://groups.io> web interface, I'm
> > guessing that messed up the thread? i haven't used mailing lists
> > before and don't know how they work. I'll use my mail client from here on.
> >
> > I'm on board with not treating EFI as an operating system. the more i
> > think about it the more it looks like scope creep.
> >
> > Agreed.
> >
> >
> > I'm not quite as enthusiastic
> > about it as i was at first glance.
> >
> > I'm still keen on doing my gsoc proposal for edk, though, and even if
> > this task and the acpica application are decided to be out of scope
> > unit testing,
> >
> > How about fuzz-testing? This is also something edk2 needs quite badly. At
> Acidanthera, we compile edk2 code in userspace outside the edk2 build
> system and fuzz with dummy applications.
> >
> > Note: fuzzing is also part of the LLVM instrumentation suite (see
> https://llvm.org/docs/LibFuzzer.html) and is something I could happily
> mentor.
> > clang integration
> >
> > Pedro and Vitaly are looking for someone to finish ASan:
> > https://edk2.groups.io/g/devel/topic/90010978#87991
> > There are working UBSan concepts, but they also need to be mainlined.
> >
> > Is Vitaly going to be a mentor? I was assuming it was going to be me and
> some other, more senior, mentor (possibly Steven Shi, which I included in
> the task).
> > Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly
> > some other sanitizer it's quite possible that it could be considered a large
> project (which means more hours but a larger stipend too). Fuzzing +
> coverage could be very nice additions to this project idea.
> > Also, is stress-testing a decent idea?
> >
> >
> > and source-level debugging are all relevant to my interests.
> >
> > how about your ideas for security stuff?
> >
> > I want the entirety of MM to leverage SmmMemLib and to support SMAP.
> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
> only work on MMRAM. Also evaluation of how to best avoid pointers in MM
> communication buffers would be nice.
> >
> > There also is a bunch of other stuff, like working out moving a part of
> CpuDxe into DxeCore to have memory protection live immediately, memory
> protection in PEI, a replacement for the TE format (it’s buggy and most
> platforms mostly abandoned it over various issues), and alternatives to
> guarding critical code with SMM (like allowing NVRAM commits only as part
> of a reboot).
> >
> > I personally find all of those projects very important, but I cannot
> > promise many people agree. Especially those that impose global changes
> > (most notably the TE replacement) may be very tedious to submit.
> > Gladly, I believe you can submit multiple proposals (?)
> >
> > Best regards,
> > Marvin
> >
> >
> > I'm not very knowledgeable about
> > trusted platform or secure boot but I'm willing to learn whatever is
> > necessary to get something spun up for my proposal.
> >
> > On Wed, Apr 13, 2022, 12:05 Marvin Häuser
> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> >
> >
> > Do you use the “reply all” option in your mail client? Looks like my
> > CCs have been dropped again. Comments inline.
> >
> > On 13. Apr 2022, at 12:54, Ada Christine
> > <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
> > wrote:
> > Hi, Marvin
> >
> > Its similarity to my own latest experiment is the key to what grabbed
> > my attention. I have no particular use case in mind for it, but I see
> > its potential for anybody developing larger applications in that when
> > a library is changed there's no need to distribute a new version of
> > the whole binary, just the relevant library module.
> >
> > I really do not like the trend of treating UEFI as a full-fledged OS -
> > it is not. The most used UEFI applications, OS loaders, are really not
> > that huge and are distributed as part of the OS image anyway. Even for
> > less used applications, you will always get a full snapshot anyhow.
> > Gladly we don’t have auto-update and package management yet. :)
> >
> >
> > I slept on it and it occurred to me that the whole thing could operate
> > similarly to the shell protocol in that the linker/loader is itself an
> > application that does a LoadImage() on the application needing dynamic
> > linking facilities.
> >
> > That would mean the linker itself is shipped with every application
> > that requires it? Otherwise it doesn’t make much sense for it to be an
> > app and below’s problems apply.
> >
> > If however the whole plan is making the linker as a DXE and including
> > it with the firmware, that I'm not quite as sure about. That would
> > necessarily tie any applications using dynamic linking to TianoCore or
> > any firmware distribution that derives from it.
> >
> > I think that was the idea referred to as “edk2 core” by Steven, but
> > I’d like to hear his proposal to be sure. Virtually everyone uses
> > edk2, so that itself is not the problem, but versioning is. Vendors
> > are slow to update their snapshots or have just given up doing that
> > entirely. Distributing it for external applications like OS loaders
> > would mean this can be leveraged probably no earlier than 10 years
> > from now. And for in-firmware things, I have a hard time thinking about a
> use-case that outweighs the drawbacks.
> >
> >
> > To shift the topic slightly back to GSoC, however, I'm willing to work
> > on other items on the task list. Unit testing and an ACPICA
> > application are the alternative projects I had thought about. I need
> > to choose fairly soon as the proposal deadline is next Tuesday. I know
> > a tiny bit about porting ACPICA as I also have plans to incorporate it into my
> own project.
> >
> > I have a few more ideas for security stuff, but Nate did not confirm
> > them as appropriate yet and I’m not here to drive you away from this
> > specific task (or the others). However, I’m still curious and
> > concerned. :)
> >
> > Best regards,
> > Marvin
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > Pedro Falcato
> >
> >
> >
> >
> >
> 
> 
> 
> 
> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Andrew Fish]

> On Apr 14, 2022, at 6:06 PM, Nate DeSimone <nathaniel.l.desimone@intel.com> wrote:
> 
> Hi Marvin,
> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin
>> Häuser
>> Sent: Thursday, April 14, 2022 12:56 AM
>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>> <nathaniel.l.desimone@intel.com>
>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>> <steven.shi@intel.com>
>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>> 
>> I feel like there are so many much easier solutions to this problem that are at
>> most limited by the clear specification. The UEFI specification with regards to
>> booting and all of that obviously is of utmost importance.
> 
> If you have a better idea that retains compatibility with the existing UEFI PI then I would be happy to hear it. Ultimately anything we do needs to be a pure extension that retains compatibility with old code. Given that restriction having the ability to coalesce all the LibraryClasses into a single module and have cross-module linking seems like the best way to handle it to me.
> 
>> The UEFI PI specification parts that deal about internal structure, as far as I know, are
>> only in place to make it easy to integrate Intel IP.
> 
> Its not just for Intel. The biggest reason for it to increase the standardization of the boot flow across the PC ecosystem. We have learned from experience that firmware is super critical to get a product out the door but it is also difficult to write. So we try to make it as reusable as humanly possible.
> 
>> In fact, I don’t *know*, but I’m pretty sure the very strict separation between PEI and DXE was
>> preserved mostly because MRC was 32-bit-only for a long time. Glad that
>> seems to have been resolved, AMD does memory init by PSP nowadays.
> 
> Having less complex early stages chain load more complex later stages is a common design pattern in firmware, not just UEFI. For example, your typical ARM system loads kinda like this:
> 
> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
> 
> Comparing relative complexity I believe the Intel UEFI PI design is actually pretty simple when you consider how much it gets done:
> 
> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
> 
> My biggest criticism of the design is that the strict separation between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
> 
> There are a few reasons why PEI was 32-bit for quite some time. The biggest one is the code size increase, 64-bit x86 code is 20-30% larger than 32-bit x86 code. Since the only RAM Pre-Memory code has access to is the cache onboard the processor and for security reasons all that code has to fit within that RAM we generally do everything we can to make that image as small as possible. Second, 64-bit requires a page table and since we desired to keep PEI simple we tried to avoid that. Finally, the PI spec didn't allow a 64-bit PEI until recently. MRC is 32-bit code just because that is what PEI happens to be. Porting it to 64-bit is not terribly difficult.
> 
> Ultimately the mix of 32/64-bit does cause some difficulties. Any data structures that get shared between PEI and DXE (HOBs, PCDs, etc.) need to resolve to the same size and packing. LibraryClasses need to be written to compile properly in both modes. In the case of FSP API mode you need to resort to thunking between 32 and 64-bit modes during DXE. More or less we decided that the costs are starting to outweigh the advantages.
> 

I’d also point out that x86 VMs use X64 (x86-64) PEI today and it works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2. 

In the PC ecosystem a single chipset family can power thousands of unique designs. So the  DRAM memory needs to be external, support lots of different chipset packages(signal integrity...), support the lowest cost through the highest cost DRAM and thousands of different board layouts. So programing DRAM takes a masters degree in antenna design. I’ve seen MRC (Memory Reference Code) with over a MiB of DEBUG prints in it, and it literally is printing histograms of what it is tuning. So all this code has to run before the system has any DRAM, thus it is running using the cache as RAM. I’ve not looked at the x86 architecture specs form the vendors in a while, but back in the day they did not support page tables in ROM or pinned cached. Now it might work, but if it breaks your CPU vendor blames you so you don’t code PEI in X64….

We contributed the 1st edk2 ARM platform, Beagle Board, and It was a long time ago but I seem to remember the mask ROM used a table in NOR FLASH to init memory and then copied an image from NOR FLASH into DRAM and jumped to it. So PEI is kind of not really needed and we implemented a PrePEI and jumped directly to DXE. 

Given I was around back in the day when all this stuff was designed I can say SEC was always a place holder for security code, as security code always has to run 1st. PEI (Pre EFI) was designed to get DRAM programmed and then jump to DXE. It kind of also fell in naturally to ACPI S3 flow since that was turning memory back on. When we designed PEI we kind of though of it more like a boot loader stage for the firmware that turned on memory and all the work would happen in DXE. Then reality strikes and the existing BIOS assembly programmers start learning C (lots of  cranky people) and they start having to learn all about PEI to turn on memory. They had to write a big chunk of code for the memory init in PEI. These folks had never written any EFI code, so to them it was easier to move a lot of the chipset init code into PEI as that is the world they had to figure out to get memory turned on. I mean why learn EFI if you don’t have to? So that is how we get so much code in IA32 (i386) on some platforms. This start kind of biased future choices and how to enable non edk2 code bases….

Thanks,

Andrew Fish


>> 
>> For many good reasons, Linux does not provide a stable kernel API. This
>> allows to easily deploy breaking changes across the entire codebase.
>> Obviously, this is infeasible at a large scale when you need to integrate IP
>> blobs, but it would already help to break the expectation that UEFI PI is a
>> perfectly forwards- and backwards-compatible system. DXE has SetMem and
>> CopyMem as part of gBS. Maybe I don’t like it that much as part of the spec
>> itself, but it’s a good idea technically. I’d probably opt to have a quickly
>> accessible implementation detail of important function pointers appended to
>> PEI and DXE services, used by in-tree modules. This may break both
>> forwards- and backwards-compatibility, but as it only applies to in-tree
>> modules, that is fine so long as we let go of the compatibility notions. PPIs
>> and protocols are an option too of course, but they have a lookup
>> performance penalty. Compared to dynamic linking, that should hopefully be
>> negligible however.
>> 
>> Absolutely optional to read, I don’t intend to waste anyone’s time much,
>> some philosophical stuff about my rationale:
>> 
>> If you started UEFI from scratch right now, would it have strictly separated
>> PEI and DXE?
> 
> For sure a clean slate design started today would look a lot different than PEI/DXE... which was a clean slate design circa 1998 😊. In my opinion, if we were to suddenly go back to the drawing board we would build something that is much closer to a full OS now than we did back then. There have been cases where not being able to use interrupt handlers and not having thread scheduling has prevented implementation of desired features. The ARM guys built LittleKernel (https://github.com/littlekernel/lk) for a lot of these reasons. In the data center world some have decided to go to the extreme of putting an entire copy of Linux in SPI so they can do a network boot that downloads the OS image using BitTorrent!
> 
>> The duplication between PEI and DXE core, and by extension
>> MM core, would be my most obvious place to start reducing size. I would
>> probably opt for a PEI/DXE hybrid where it starts in „minimal mode“ (maybe
>> think of PEI more like a microkernel here) and after memory is initialised, the
>> rest of DXE is loaded. And MM core would get no loading at all, this
>> requirement has gladly been dropped ages ago. Just one prelinked snapshot
>> of the address space with a relocation table and a safe self-relocator on entry
>> (this is needed at the very least for ARM).
>> 
>> Ironically, with my idea for MM, dynamic loading would be free as everything
>> is prelinked anyway. The same is true for PEI XIP, it is prelinked by nature.
> 
> Actually Post-Memory PEI can have non-prelinked PEIMs. And that does get used for the PEI GOP driver.
> 
>> What I do not like is the additional dynamic linking code at load-time for non-
>> XIP modules. Though, the more I think about it, the more I wonder whether
>> not the entirety of UEFI could be composed of prelinked, relocatable
>> memory snapshots without traditional image loading entirely (for in-FW
>> stuff). macOS has a similar concept with its “Kernel Collections”. Well, way
>> too much off-topic now. :)
> 
> If you make the assumption that 100% of the code is compiled all at once then yes that works. UEFI was designed so that assumption does not need to be true. There are good use cases for it: OpROMs, generic OS loaders, network boot, etc. 
> 
> Thanks,
> Nate
> 
>> 
>> Why am I explaining all this despite the fact everyone knows this will never
>> happen? Because I don’t like the notion of fixing issues of an already
>> overcomplicated system by adding even more complicated stuff. Especially
>> when the existing overcomplicated stuff is already uncomfortably broken.
>> 
>> Best regards,
>> Marvin
>> 
>>> For XIP PEI code… it will really help and would be very timely since the
>> transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by
>> ~20%.
>>> 
>>> Thanks,
>>> Nate
>>> 
>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>> <mhaeuser@posteo.de>
>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>> <steven.shi@intel.com>
>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>> 
>>> Hi Marvin, Ada,
>>> 
>>> Some comments:
>>> 
>>> I don't think the purpose of the dynamic linker is to treat EFI as a
>>> complete operating system, but to try to eliminate the static linking
>>> that may be needlessly duplicating code that could instead be put in a
>> single dynamic library. For instance, MdePkg and MdeModulePkg are linked
>> into a *lot* of .efi, instead of being just a library. It'd be nice to see some
>> numbers on this (something like Google's bloaty could be run on every .efi
>> file, in order to understand how much file space we would actually save).
>>> 
>>> Other comments inline.
>>> 
>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>> 
>>> On 13. Apr 2022, at 16:38, Ada Christine
>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>> i was replying via the groups.io<http://groups.io> web interface, I'm
>>> guessing that messed up the thread? i haven't used mailing lists
>>> before and don't know how they work. I'll use my mail client from here on.
>>> 
>>> I'm on board with not treating EFI as an operating system. the more i
>>> think about it the more it looks like scope creep.
>>> 
>>> Agreed.
>>> 
>>> 
>>> I'm not quite as enthusiastic
>>> about it as i was at first glance.
>>> 
>>> I'm still keen on doing my gsoc proposal for edk, though, and even if
>>> this task and the acpica application are decided to be out of scope
>>> unit testing,
>>> 
>>> How about fuzz-testing? This is also something edk2 needs quite badly. At
>> Acidanthera, we compile edk2 code in userspace outside the edk2 build
>> system and fuzz with dummy applications.
>>> 
>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>> https://llvm.org/docs/LibFuzzer.html) and is something I could happily
>> mentor.
>>> clang integration
>>> 
>>> Pedro and Vitaly are looking for someone to finish ASan:
>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>> There are working UBSan concepts, but they also need to be mainlined.
>>> 
>>> Is Vitaly going to be a mentor? I was assuming it was going to be me and
>> some other, more senior, mentor (possibly Steven Shi, which I included in
>> the task).
>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly
>>> some other sanitizer it's quite possible that it could be considered a large
>> project (which means more hours but a larger stipend too). Fuzzing +
>> coverage could be very nice additions to this project idea.
>>> Also, is stress-testing a decent idea?
>>> 
>>> 
>>> and source-level debugging are all relevant to my interests.
>>> 
>>> how about your ideas for security stuff?
>>> 
>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
>> only work on MMRAM. Also evaluation of how to best avoid pointers in MM
>> communication buffers would be nice.
>>> 
>>> There also is a bunch of other stuff, like working out moving a part of
>> CpuDxe into DxeCore to have memory protection live immediately, memory
>> protection in PEI, a replacement for the TE format (it’s buggy and most
>> platforms mostly abandoned it over various issues), and alternatives to
>> guarding critical code with SMM (like allowing NVRAM commits only as part
>> of a reboot).
>>> 
>>> I personally find all of those projects very important, but I cannot
>>> promise many people agree. Especially those that impose global changes
>>> (most notably the TE replacement) may be very tedious to submit.
>>> Gladly, I believe you can submit multiple proposals (?)
>>> 
>>> Best regards,
>>> Marvin
>>> 
>>> 
>>> I'm not very knowledgeable about
>>> trusted platform or secure boot but I'm willing to learn whatever is
>>> necessary to get something spun up for my proposal.
>>> 
>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>> 
>>> 
>>> Do you use the “reply all” option in your mail client? Looks like my
>>> CCs have been dropped again. Comments inline.
>>> 
>>> On 13. Apr 2022, at 12:54, Ada Christine
>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>> wrote:
>>> Hi, Marvin
>>> 
>>> Its similarity to my own latest experiment is the key to what grabbed
>>> my attention. I have no particular use case in mind for it, but I see
>>> its potential for anybody developing larger applications in that when
>>> a library is changed there's no need to distribute a new version of
>>> the whole binary, just the relevant library module.
>>> 
>>> I really do not like the trend of treating UEFI as a full-fledged OS -
>>> it is not. The most used UEFI applications, OS loaders, are really not
>>> that huge and are distributed as part of the OS image anyway. Even for
>>> less used applications, you will always get a full snapshot anyhow.
>>> Gladly we don’t have auto-update and package management yet. :)
>>> 
>>> 
>>> I slept on it and it occurred to me that the whole thing could operate
>>> similarly to the shell protocol in that the linker/loader is itself an
>>> application that does a LoadImage() on the application needing dynamic
>>> linking facilities.
>>> 
>>> That would mean the linker itself is shipped with every application
>>> that requires it? Otherwise it doesn’t make much sense for it to be an
>>> app and below’s problems apply.
>>> 
>>> If however the whole plan is making the linker as a DXE and including
>>> it with the firmware, that I'm not quite as sure about. That would
>>> necessarily tie any applications using dynamic linking to TianoCore or
>>> any firmware distribution that derives from it.
>>> 
>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>> are slow to update their snapshots or have just given up doing that
>>> entirely. Distributing it for external applications like OS loaders
>>> would mean this can be leveraged probably no earlier than 10 years
>>> from now. And for in-firmware things, I have a hard time thinking about a
>> use-case that outweighs the drawbacks.
>>> 
>>> 
>>> To shift the topic slightly back to GSoC, however, I'm willing to work
>>> on other items on the task list. Unit testing and an ACPICA
>>> application are the alternative projects I had thought about. I need
>>> to choose fairly soon as the proposal deadline is next Tuesday. I know
>>> a tiny bit about porting ACPICA as I also have plans to incorporate it into my
>> own project.
>>> 
>>> I have a few more ideas for security stuff, but Nate did not confirm
>>> them as appropriate yet and I’m not here to drive you away from this
>>> specific task (or the others). However, I’m still curious and
>>> concerned. :)
>>> 
>>> Best regards,
>>> Marvin
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> Pedro Falcato
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

> On 15. Apr 2022, at 04:42, Andrew Fish <afish@apple.com> wrote:
> 
> 
>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone <nathaniel.l.desimone@intel.com> wrote:
>> 
>> Hi Marvin,
>> 
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin
>>> Häuser
>>> Sent: Thursday, April 14, 2022 12:56 AM
>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>>> <nathaniel.l.desimone@intel.com>
>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>>> <steven.shi@intel.com>
>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>> I feel like there are so many much easier solutions to this problem that are at
>>> most limited by the clear specification. The UEFI specification with regards to
>>> booting and all of that obviously is of utmost importance.
>> 
>> If you have a better idea that retains compatibility with the existing UEFI PI then I would be happy to hear it. Ultimately anything we do needs to be a pure extension that retains compatibility with old code. Given that restriction having the ability to coalesce all the LibraryClasses into a single module and have cross-module linking seems like the best way to handle it to me.
>> 
>>> The UEFI PI specification parts that deal about internal structure, as far as I know, are
>>> only in place to make it easy to integrate Intel IP.
>> 
>> Its not just for Intel. The biggest reason for it to increase the standardization of the boot flow across the PC ecosystem. We have learned from experience that firmware is super critical to get a product out the door but it is also difficult to write. So we try to make it as reusable as humanly possible.
>> 
>>> In fact, I don’t *know*, but I’m pretty sure the very strict separation between PEI and DXE was
>>> preserved mostly because MRC was 32-bit-only for a long time. Glad that
>>> seems to have been resolved, AMD does memory init by PSP nowadays.
>> 
>> Having less complex early stages chain load more complex later stages is a common design pattern in firmware, not just UEFI. For example, your typical ARM system loads kinda like this:
>> 
>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor --> EDK II or U-Boot or LittleKernel (which runs android fastboot)

It’s not about the fact that stages of rising complexity exist (SEC for example is nice), but more so that the PEI and DXE dispatchers share quite a bunch of code. PEI has more of a “DXE light” notion to me than really being its own thing with its own distinct purpose.

>> 
>> Comparing relative complexity I believe the Intel UEFI PI design is actually pretty simple when you consider how much it gets done:
>> 
>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>> 
>> My biggest criticism of the design is that the strict separation between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

Well, there is Standalone MM and it’s badly needed. I’m not sure separate BDS would have many advantages.

>> 
>> There are a few reasons why PEI was 32-bit for quite some time. The biggest one is the code size increase, 64-bit x86 code is 20-30% larger than 32-bit x86 code. Since the only RAM Pre-Memory code has access to is the cache onboard the processor and for security reasons all that code has to fit within that RAM we generally do everything we can to make that image as small as possible. Second, 64-bit requires a page table and since we desired to keep PEI simple we tried to avoid that. Finally, the PI spec didn't allow a 64-bit PEI until recently. MRC is 32-bit code just because that is what PEI happens to be. Porting it to 64-bit is not terribly difficult.

I heard a bunch of horror stories about the MRC, but from less credible sources. Thanks for correcting. :)

>> 
>> Ultimately the mix of 32/64-bit does cause some difficulties. Any data structures that get shared between PEI and DXE (HOBs, PCDs, etc.) need to resolve to the same size and packing. LibraryClasses need to be written to compile properly in both modes. In the case of FSP API mode you need to resort to thunking between 32 and 64-bit modes during DXE. More or less we decided that the costs are starting to outweigh the advantages.
> 
> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2. 

Was there a misunderstanding? I never claimed EDK II design *limited* X64 PEI, I said I *believe* its design was noticeably influenced by the requirement of cross-architecture execution. Was it not?

> 
> In the PC ecosystem a single chipset family can power thousands of unique designs. So the  DRAM memory needs to be external, support lots of different chipset packages(signal integrity...), support the lowest cost through the highest cost DRAM and thousands of different board layouts. So programing DRAM takes a masters degree in antenna design. I’ve seen MRC (Memory Reference Code) with over a MiB of DEBUG prints in it, and it literally is printing histograms of what it is tuning. So all this code has to run before the system has any DRAM, thus it is running using the cache as RAM. I’ve not looked at the x86 architecture specs form the vendors in a while, but back in the day they did not support page tables in ROM or pinned cached. Now it might work, but if it breaks your CPU vendor blames you so you don’t code PEI in X64….
> 
> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a long time ago but I seem to remember the mask ROM used a table in NOR FLASH to init memory and then copied an image from NOR FLASH into DRAM and jumped to it. So PEI is kind of not really needed and we implemented a PrePEI and jumped directly to DXE. 
> 
> Given I was around back in the day when all this stuff was designed I can say SEC was always a place holder for security code, as security code always has to run 1st. PEI (Pre EFI) was designed to get DRAM programmed and then jump to DXE. It kind of also fell in naturally to ACPI S3 flow since that was turning memory back on. When we designed PEI we kind of though of it more like a boot loader stage for the firmware that turned on memory and all the work would happen in DXE. Then reality strikes and the existing BIOS assembly programmers start learning C (lots of  cranky people) and they start having to learn all about PEI to turn on memory. They had to write a big chunk of code for the memory init in PEI. These folks had never written any EFI code, so to them it was easier to move a lot of the chipset init code into PEI as that is the world they had to figure out to get memory turned on. I mean why learn EFI if you don’t have to? So that is how we get so much code in IA32 (i386) on some platforms. This start kind of biased future choices and how to enable non edk2 code bases….

That actually clears a lot of stuff up… there is a lot going on in PEI and I always wondered whether all of that was really needed to enable DRAM. Guess not. Thanks! :)

> 
> Thanks,
> 
> Andrew Fish
> 
> 
>>> For many good reasons, Linux does not provide a stable kernel API. This
>>> allows to easily deploy breaking changes across the entire codebase.
>>> Obviously, this is infeasible at a large scale when you need to integrate IP
>>> blobs, but it would already help to break the expectation that UEFI PI is a
>>> perfectly forwards- and backwards-compatible system. DXE has SetMem and
>>> CopyMem as part of gBS. Maybe I don’t like it that much as part of the spec
>>> itself, but it’s a good idea technically. I’d probably opt to have a quickly
>>> accessible implementation detail of important function pointers appended to
>>> PEI and DXE services, used by in-tree modules. This may break both
>>> forwards- and backwards-compatibility, but as it only applies to in-tree
>>> modules, that is fine so long as we let go of the compatibility notions. PPIs
>>> and protocols are an option too of course, but they have a lookup
>>> performance penalty. Compared to dynamic linking, that should hopefully be
>>> negligible however.
>>> Absolutely optional to read, I don’t intend to waste anyone’s time much,
>>> some philosophical stuff about my rationale:
>>> If you started UEFI from scratch right now, would it have strictly separated
>>> PEI and DXE?
>> 
>> For sure a clean slate design started today would look a lot different than PEI/DXE... which was a clean slate design circa 1998 😊. In my opinion, if we were to suddenly go back to the drawing board we would build something that is much closer to a full OS now than we did back then. There have been cases where not being able to use interrupt handlers and not having thread scheduling has prevented implementation of desired features. The ARM guys built LittleKernel (https://github.com/littlekernel/lk) for a lot of these reasons. In the data center world some have decided to go to the extreme of putting an entire copy of Linux in SPI so they can do a network boot that downloads the OS image using BitTorrent!

That last part hurts. :)

>> 
>>> The duplication between PEI and DXE core, and by extension
>>> MM core, would be my most obvious place to start reducing size. I would
>>> probably opt for a PEI/DXE hybrid where it starts in „minimal mode“ (maybe
>>> think of PEI more like a microkernel here) and after memory is initialised, the
>>> rest of DXE is loaded. And MM core would get no loading at all, this
>>> requirement has gladly been dropped ages ago. Just one prelinked snapshot
>>> of the address space with a relocation table and a safe self-relocator on entry
>>> (this is needed at the very least for ARM).
>>> Ironically, with my idea for MM, dynamic loading would be free as everything
>>> is prelinked anyway. The same is true for PEI XIP, it is prelinked by nature.
>> 
>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that does get used for the PEI GOP driver.

Yes, hence I said “XIP”. I’m not sure how many of those even are left in modern designs, but they are already “pre-loaded” (load + relocate) and thus must be prelinked in the future. Using PPIs and protocols with a stable structure (at least stable for as long as it makes sense) mitigates the issues, but as I said, harms performance.

>> 
>>> What I do not like is the additional dynamic linking code at load-time for non-
>>> XIP modules. Though, the more I think about it, the more I wonder whether
>>> not the entirety of UEFI could be composed of prelinked, relocatable
>>> memory snapshots without traditional image loading entirely (for in-FW
>>> stuff). macOS has a similar concept with its “Kernel Collections”. Well, way
>>> too much off-topic now. :)
>> 
>> If you make the assumption that 100% of the code is compiled all at once then yes that works. UEFI was designed so that assumption does not need to be true. There are good use cases for it: OpROMs, generic OS loaders, network boot, etc. 

All of those are things you declared (at least temporarily) out of scope for dynamic linking. Allowing these would have a latency of many, many years to even start taking advantage of it (especially for OSes it hardly matters, OS loaders are tiny and insignificant in the big picture) and then you’re stuck with guaranteeing compatibility virtually forever. I was strictly talking about “private code” that composes the primary UEFI image.

However the more I think about it, the more I like the idea of prelinking. Because don’t get me wrong, I don’t dislike the state of having dynamically linked, it’s nice and convenient, I mind having a dynamic linker in a bare-bones environment without memory protection (PEI) and in a highly security critical source of hazards (SMM). Well-written and tested prelinking code would not only solve all my concerns, it would also further reduce firmware code complexity and binary size and increase performance. Many issues:

1) Without the help of “Collections” I was philosophising about above, every single module may have a different base address and they need to be recorded (would be required for dynamic linking as well). Relocation code would grow more complex as it has to shift relocations that target different link targets by different deltas. Without Collections, this might only be a little less complicated than just dynamic linking, though I never wrote such a linker for PE/COFF (but did for Mach-O… it’s not nice :) ).
- How Collections solve this: Have a “Core Collection” that includes {Pei,Dxe}Core and its direct extensions (e.g., SecurityStub, CpuDxe) which can be consumed and otherwise force dynamic linking to only happen within the own collection. This means for every collection, there is at most one additional relocation delta (the Core Collection) and it doesn’t need specific identification. General interfaces would all go into the Core Collection (strings, memory, device path, …) and specific interfaces must be sorted into over-arching Collections (Networking, USB, …?).

2) Sophisticated tools are needed. However, a prelinker is required for XIP PEIMs in any scenario. Prelinking non-XIPs would require something to emit the “extended relocations” and the symbol table information should be accompanying the flash image, stripped from the PE/COFF files (prevents dynamic linking of anything outside the primary image, saves space within the image). Any modifications of executables in the image after generation with the TianoCore tools would need to be updated, I have no information on the requirements there.

3) Little flexibility. No matter what, flexibility will be suffering with prelinking. Without prelinking, there will be more space occupation (3-times duplicated dynamic linker, symbol table for every dynamic library) and more code complexity (I know of downstreams that already are so upset at the current loader that they forked and froze it with out-of-tree patches).

This prelinking idea is crazy, but it may be worth it for some scenarios. The private function array is still my favourite. :) In my opinion, this needs a clear declaration of requirements and scope with multiple approaches compared to each other wrt complexity and guarantees. Just implementing dynamic linking may mean another thing everyone is stuck with once it lands (which is not about questioning the implementation itself, but the abstract idea‘s complexity compared to the requirements). It also matters a lot whether you just want to save some space (the private function array can be abolished at any time in case something more sophisticated is needed) or build an infrastructure.

Best regards,
Marvin

>> 
>> Thanks,
>> Nate
>> 
>>> Why am I explaining all this despite the fact everyone knows this will never
>>> happen? Because I don’t like the notion of fixing issues of an already
>>> overcomplicated system by adding even more complicated stuff. Especially
>>> when the existing overcomplicated stuff is already uncomfortably broken.
>>> Best regards,
>>> Marvin
>>>> For XIP PEI code… it will really help and would be very timely since the
>>> transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by
>>> ~20%.
>>>> Thanks,
>>>> Nate
>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>>> <mhaeuser@posteo.de>
>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>>> <steven.shi@intel.com>
>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>> Hi Marvin, Ada,
>>>> Some comments:
>>>> I don't think the purpose of the dynamic linker is to treat EFI as a
>>>> complete operating system, but to try to eliminate the static linking
>>>> that may be needlessly duplicating code that could instead be put in a
>>> single dynamic library. For instance, MdePkg and MdeModulePkg are linked
>>> into a *lot* of .efi, instead of being just a library. It'd be nice to see some
>>> numbers on this (something like Google's bloaty could be run on every .efi
>>> file, in order to understand how much file space we would actually save).
>>>> Other comments inline.
>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>>> i was replying via the groups.io<http://groups.io> web interface, I'm
>>>> guessing that messed up the thread? i haven't used mailing lists
>>>> before and don't know how they work. I'll use my mail client from here on.
>>>> I'm on board with not treating EFI as an operating system. the more i
>>>> think about it the more it looks like scope creep.
>>>> Agreed.
>>>> I'm not quite as enthusiastic
>>>> about it as i was at first glance.
>>>> I'm still keen on doing my gsoc proposal for edk, though, and even if
>>>> this task and the acpica application are decided to be out of scope
>>>> unit testing,
>>>> How about fuzz-testing? This is also something edk2 needs quite badly. At
>>> Acidanthera, we compile edk2 code in userspace outside the edk2 build
>>> system and fuzz with dummy applications.
>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>> https://llvm.org/docs/LibFuzzer.html) and is something I could happily
>>> mentor.
>>>> clang integration
>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>> There are working UBSan concepts, but they also need to be mainlined.
>>>> Is Vitaly going to be a mentor? I was assuming it was going to be me and
>>> some other, more senior, mentor (possibly Steven Shi, which I included in
>>> the task).
>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly
>>>> some other sanitizer it's quite possible that it could be considered a large
>>> project (which means more hours but a larger stipend too). Fuzzing +
>>> coverage could be very nice additions to this project idea.
>>>> Also, is stress-testing a decent idea?
>>>> and source-level debugging are all relevant to my interests.
>>>> how about your ideas for security stuff?
>>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
>>> only work on MMRAM. Also evaluation of how to best avoid pointers in MM
>>> communication buffers would be nice.
>>>> There also is a bunch of other stuff, like working out moving a part of
>>> CpuDxe into DxeCore to have memory protection live immediately, memory
>>> protection in PEI, a replacement for the TE format (it’s buggy and most
>>> platforms mostly abandoned it over various issues), and alternatives to
>>> guarding critical code with SMM (like allowing NVRAM commits only as part
>>> of a reboot).
>>>> I personally find all of those projects very important, but I cannot
>>>> promise many people agree. Especially those that impose global changes
>>>> (most notably the TE replacement) may be very tedious to submit.
>>>> Gladly, I believe you can submit multiple proposals (?)
>>>> Best regards,
>>>> Marvin
>>>> I'm not very knowledgeable about
>>>> trusted platform or secure boot but I'm willing to learn whatever is
>>>> necessary to get something spun up for my proposal.
>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>> Do you use the “reply all” option in your mail client? Looks like my
>>>> CCs have been dropped again. Comments inline.
>>>> On 13. Apr 2022, at 12:54, Ada Christine
>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>> wrote:
>>>> Hi, Marvin
>>>> Its similarity to my own latest experiment is the key to what grabbed
>>>> my attention. I have no particular use case in mind for it, but I see
>>>> its potential for anybody developing larger applications in that when
>>>> a library is changed there's no need to distribute a new version of
>>>> the whole binary, just the relevant library module.
>>>> I really do not like the trend of treating UEFI as a full-fledged OS -
>>>> it is not. The most used UEFI applications, OS loaders, are really not
>>>> that huge and are distributed as part of the OS image anyway. Even for
>>>> less used applications, you will always get a full snapshot anyhow.
>>>> Gladly we don’t have auto-update and package management yet. :)
>>>> I slept on it and it occurred to me that the whole thing could operate
>>>> similarly to the shell protocol in that the linker/loader is itself an
>>>> application that does a LoadImage() on the application needing dynamic
>>>> linking facilities.
>>>> That would mean the linker itself is shipped with every application
>>>> that requires it? Otherwise it doesn’t make much sense for it to be an
>>>> app and below’s problems apply.
>>>> If however the whole plan is making the linker as a DXE and including
>>>> it with the firmware, that I'm not quite as sure about. That would
>>>> necessarily tie any applications using dynamic linking to TianoCore or
>>>> any firmware distribution that derives from it.
>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>> are slow to update their snapshots or have just given up doing that
>>>> entirely. Distributing it for external applications like OS loaders
>>>> would mean this can be leveraged probably no earlier than 10 years
>>>> from now. And for in-firmware things, I have a hard time thinking about a
>>> use-case that outweighs the drawbacks.
>>>> To shift the topic slightly back to GSoC, however, I'm willing to work
>>>> on other items on the task list. Unit testing and an ACPICA
>>>> application are the alternative projects I had thought about. I need
>>>> to choose fairly soon as the proposal deadline is next Tuesday. I know
>>>> a tiny bit about porting ACPICA as I also have plans to incorporate it into my
>>> own project.
>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>> them as appropriate yet and I’m not here to drive you away from this
>>>> specific task (or the others). However, I’m still curious and
>>>> concerned. :)
>>>> Best regards,
>>>> Marvin
>>>> --
>>>> Pedro Falcato
>> 
>> 
>> 
>> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Nate DeSimone]

Hi Andrew,

On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
>
>
>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone <nathaniel.l.desimone@intel.com> wrote:
>> 
>> Hi Marvin,
>> 
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin
>>> Häuser
>>> Sent: Thursday, April 14, 2022 12:56 AM
>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>>> <nathaniel.l.desimone@intel.com>
>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>>> <steven.shi@intel.com>
>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>> 
>>> I feel like there are so many much easier solutions to this problem that are at
>>> most limited by the clear specification. The UEFI specification with regards to
>>> booting and all of that obviously is of utmost importance.
>> 
>> If you have a better idea that retains compatibility with the existing UEFI PI then I would be happy to hear it. Ultimately anything we do needs to be a pure extension that retains compatibility with old code. Given that restriction having the ability to coalesce all the LibraryClasses into a single module and have cross-module linking seems like the best way to handle it to me.
>> 
>>> The UEFI PI specification parts that deal about internal structure, as far as I know, are
>>> only in place to make it easy to integrate Intel IP.
>> 
>> Its not just for Intel. The biggest reason for it to increase the standardization of the boot flow across the PC ecosystem. We have learned from experience that firmware is super critical to get a product out the door but it is also difficult to write. So we try to make it as reusable as humanly possible.
>> 
>>> In fact, I don’t *know*, but I’m pretty sure the very strict separation between PEI and DXE was
>>> preserved mostly because MRC was 32-bit-only for a long time. Glad that
>>> seems to have been resolved, AMD does memory init by PSP nowadays.
>> 
>> Having less complex early stages chain load more complex later stages is a common design pattern in firmware, not just UEFI. For example, your typical ARM system loads kinda like this:
>> 
>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
>> 
>> Comparing relative complexity I believe the Intel UEFI PI design is actually pretty simple when you consider how much it gets done:
>> 
>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>> 
>> My biggest criticism of the design is that the strict separation between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
>> 
>> There are a few reasons why PEI was 32-bit for quite some time. The biggest one is the code size increase, 64-bit x86 code is 20-30% larger than 32-bit x86 code. Since the only RAM Pre-Memory code has access to is the cache onboard the processor and for security reasons all that code has to fit within that RAM we generally do everything we can to make that image as small as possible. Second, 64-bit requires a page table and since we desired to keep PEI simple we tried to avoid that. Finally, the PI spec didn't allow a 64-bit PEI until recently. MRC is 32-bit code just because that is what PEI happens to be. Porting it to 64-bit is not terribly difficult.
>> 
>> Ultimately the mix of 32/64-bit does cause some difficulties. Any data structures that get shared between PEI and DXE (HOBs, PCDs, etc.) need to resolve to the same size and packing. LibraryClasses need to be written to compile properly in both modes. In the case of FSP API mode you need to resort to thunking between 32 and 64-bit modes during DXE. More or less we decided that the costs are starting to outweigh the advantages.
>> 
>
>I’d also point out that x86 VMs use X64 (x86-64) PEI today and it works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2. 
>
>In the PC ecosystem a single chipset family can power thousands of unique designs. So the  DRAM memory needs to be external, support lots of different chipset packages(signal integrity...), support the lowest cost through the highest cost DRAM and thousands of different board layouts. So programing DRAM takes a masters degree in antenna design. I’ve seen MRC (Memory Reference Code) with over a MiB of DEBUG prints in it, and it literally is printing histograms of what it is tuning. So all this code has to run before the system has any DRAM, thus it is running using the cache as RAM. I’ve not looked at the x86 architecture specs form the vendors in a while, but back in the day they did not support page tables in ROM or pinned cached. Now it might work, but if it breaks your CPU vendor blames you so you don’t code PEI in X64….
>
>We contributed the 1st edk2 ARM platform, Beagle Board, and It was a long time ago but I seem to remember the mask ROM used a table in NOR FLASH to init memory and then copied an image from NOR FLASH into DRAM and jumped to it. So PEI is kind of not really needed and we implemented a PrePEI and jumped directly to DXE. 
>
>Given I was around back in the day when all this stuff was designed I can say SEC was always a place holder for security code, as security code always has to run 1st. PEI (Pre EFI) was designed to get DRAM programmed and then jump to DXE. It kind of also fell in naturally to ACPI S3 flow since that was turning memory back on. When we designed PEI we kind of though of it more like a boot loader stage for the firmware that turned on memory and all the work would happen in DXE. Then reality strikes and the existing BIOS assembly programmers start learning C (lots of  cranky people) and they start having to learn all about PEI to turn on memory. They had to write a big chunk of code for the memory init in PEI. These folks had never written any EFI code, so to them it was easier to move a lot of the chipset init code into PEI as that is the world they had to figure out to get memory turned on. I mean why learn EFI if you don’t have to? So that is how we get so much code in IA32 (i386) on some platforms. This start kind of biased future choices and how to enable non edk2 code bases….

One of the big reasons a lot of code that should have been written in DXE ended up in PEI is unfortunately due to the FSP and its inability to support DXE code.

>
>Thanks,
>
>Andrew Fish
>
>
>>> 
>>> For many good reasons, Linux does not provide a stable kernel API. This
>>> allows to easily deploy breaking changes across the entire codebase.
>>> Obviously, this is infeasible at a large scale when you need to integrate IP
>>> blobs, but it would already help to break the expectation that UEFI PI is a
>>> perfectly forwards- and backwards-compatible system. DXE has SetMem and
>>> CopyMem as part of gBS. Maybe I don’t like it that much as part of the spec
>>> itself, but it’s a good idea technically. I’d probably opt to have a quickly
>>> accessible implementation detail of important function pointers appended to
>>> PEI and DXE services, used by in-tree modules. This may break both
>>> forwards- and backwards-compatibility, but as it only applies to in-tree
>>> modules, that is fine so long as we let go of the compatibility notions. PPIs
>>> and protocols are an option too of course, but they have a lookup
>>> performance penalty. Compared to dynamic linking, that should hopefully be
>>> negligible however.
>>> 
>>> Absolutely optional to read, I don’t intend to waste anyone’s time much,
>>> some philosophical stuff about my rationale:
>>> 
>>> If you started UEFI from scratch right now, would it have strictly separated
>>> PEI and DXE?
>> 
>> For sure a clean slate design started today would look a lot different than PEI/DXE... which was a clean slate design circa 1998 😊. In my opinion, if we were to suddenly go back to the drawing board we would build something that is much closer to a full OS now than we did back then. There have been cases where not being able to use interrupt handlers and not having thread scheduling has prevented implementation of desired features. The ARM guys built LittleKernel (https://github.com/littlekernel/lk) for a lot of these reasons. In the data center world some have decided to go to the extreme of putting an entire copy of Linux in SPI so they can do a network boot that downloads the OS image using BitTorrent!
>> 
>>> The duplication between PEI and DXE core, and by extension
>>> MM core, would be my most obvious place to start reducing size. I would
>>> probably opt for a PEI/DXE hybrid where it starts in „minimal mode“ (maybe
>>> think of PEI more like a microkernel here) and after memory is initialised, the
>>> rest of DXE is loaded. And MM core would get no loading at all, this
>>> requirement has gladly been dropped ages ago. Just one prelinked snapshot
>>> of the address space with a relocation table and a safe self-relocator on entry
>>> (this is needed at the very least for ARM).
>>> 
>>> Ironically, with my idea for MM, dynamic loading would be free as everything
>>> is prelinked anyway. The same is true for PEI XIP, it is prelinked by nature.
>> 
>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that does get used for the PEI GOP driver.
>> 
>>> What I do not like is the additional dynamic linking code at load-time for non-
>>> XIP modules. Though, the more I think about it, the more I wonder whether
>>> not the entirety of UEFI could be composed of prelinked, relocatable
>>> memory snapshots without traditional image loading entirely (for in-FW
>>> stuff). macOS has a similar concept with its “Kernel Collections”. Well, way
>>> too much off-topic now. :)
>> 
>> If you make the assumption that 100% of the code is compiled all at once then yes that works. UEFI was designed so that assumption does not need to be true. There are good use cases for it: OpROMs, generic OS loaders, network boot, etc. 
>> 
>> Thanks,
>> Nate
>> 
>>> 
>>> Why am I explaining all this despite the fact everyone knows this will never
>>> happen? Because I don’t like the notion of fixing issues of an already
>>> overcomplicated system by adding even more complicated stuff. Especially
>>> when the existing overcomplicated stuff is already uncomfortably broken.
>>> 
>>> Best regards,
>>> Marvin
>>> 
>>>> For XIP PEI code… it will really help and would be very timely since the
>>> transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by
>>> ~20%.
>>>> 
>>>> Thanks,
>>>> Nate
>>>> 
>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>>> <mhaeuser@posteo.de>
>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>>> <steven.shi@intel.com>
>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>> 
>>>> Hi Marvin, Ada,
>>>> 
>>>> Some comments:
>>>> 
>>>> I don't think the purpose of the dynamic linker is to treat EFI as a
>>>> complete operating system, but to try to eliminate the static linking
>>>> that may be needlessly duplicating code that could instead be put in a
>>> single dynamic library. For instance, MdePkg and MdeModulePkg are linked
>>> into a *lot* of .efi, instead of being just a library. It'd be nice to see some
>>> numbers on this (something like Google's bloaty could be run on every .efi
>>> file, in order to understand how much file space we would actually save).
>>>> 
>>>> Other comments inline.
>>>> 
>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>> 
>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>>> i was replying via the groups.io<http://groups.io> web interface, I'm
>>>> guessing that messed up the thread? i haven't used mailing lists
>>>> before and don't know how they work. I'll use my mail client from here on.
>>>> 
>>>> I'm on board with not treating EFI as an operating system. the more i
>>>> think about it the more it looks like scope creep.
>>>> 
>>>> Agreed.
>>>> 
>>>> 
>>>> I'm not quite as enthusiastic
>>>> about it as i was at first glance.
>>>> 
>>>> I'm still keen on doing my gsoc proposal for edk, though, and even if
>>>> this task and the acpica application are decided to be out of scope
>>>> unit testing,
>>>> 
>>>> How about fuzz-testing? This is also something edk2 needs quite badly. At
>>> Acidanthera, we compile edk2 code in userspace outside the edk2 build
>>> system and fuzz with dummy applications.
>>>> 
>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>> https://llvm.org/docs/LibFuzzer.html) and is something I could happily
>>> mentor.
>>>> clang integration
>>>> 
>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>> There are working UBSan concepts, but they also need to be mainlined.
>>>> 
>>>> Is Vitaly going to be a mentor? I was assuming it was going to be me and
>>> some other, more senior, mentor (possibly Steven Shi, which I included in
>>> the task).
>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly
>>>> some other sanitizer it's quite possible that it could be considered a large
>>> project (which means more hours but a larger stipend too). Fuzzing +
>>> coverage could be very nice additions to this project idea.
>>>> Also, is stress-testing a decent idea?
>>>> 
>>>> 
>>>> and source-level debugging are all relevant to my interests.
>>>> 
>>>> how about your ideas for security stuff?
>>>> 
>>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
>>> only work on MMRAM. Also evaluation of how to best avoid pointers in MM
>>> communication buffers would be nice.
>>>> 
>>>> There also is a bunch of other stuff, like working out moving a part of
>>> CpuDxe into DxeCore to have memory protection live immediately, memory
>>> protection in PEI, a replacement for the TE format (it’s buggy and most
>>> platforms mostly abandoned it over various issues), and alternatives to
>>> guarding critical code with SMM (like allowing NVRAM commits only as part
>>> of a reboot).
>>>> 
>>>> I personally find all of those projects very important, but I cannot
>>>> promise many people agree. Especially those that impose global changes
>>>> (most notably the TE replacement) may be very tedious to submit.
>>>> Gladly, I believe you can submit multiple proposals (?)
>>>> 
>>>> Best regards,
>>>> Marvin
>>>> 
>>>> 
>>>> I'm not very knowledgeable about
>>>> trusted platform or secure boot but I'm willing to learn whatever is
>>>> necessary to get something spun up for my proposal.
>>>> 
>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>> 
>>>> 
>>>> Do you use the “reply all” option in your mail client? Looks like my
>>>> CCs have been dropped again. Comments inline.
>>>> 
>>>> On 13. Apr 2022, at 12:54, Ada Christine
>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>> wrote:
>>>> Hi, Marvin
>>>> 
>>>> Its similarity to my own latest experiment is the key to what grabbed
>>>> my attention. I have no particular use case in mind for it, but I see
>>>> its potential for anybody developing larger applications in that when
>>>> a library is changed there's no need to distribute a new version of
>>>> the whole binary, just the relevant library module.
>>>> 
>>>> I really do not like the trend of treating UEFI as a full-fledged OS -
>>>> it is not. The most used UEFI applications, OS loaders, are really not
>>>> that huge and are distributed as part of the OS image anyway. Even for
>>>> less used applications, you will always get a full snapshot anyhow.
>>>> Gladly we don’t have auto-update and package management yet. :)
>>>> 
>>>> 
>>>> I slept on it and it occurred to me that the whole thing could operate
>>>> similarly to the shell protocol in that the linker/loader is itself an
>>>> application that does a LoadImage() on the application needing dynamic
>>>> linking facilities.
>>>> 
>>>> That would mean the linker itself is shipped with every application
>>>> that requires it? Otherwise it doesn’t make much sense for it to be an
>>>> app and below’s problems apply.
>>>> 
>>>> If however the whole plan is making the linker as a DXE and including
>>>> it with the firmware, that I'm not quite as sure about. That would
>>>> necessarily tie any applications using dynamic linking to TianoCore or
>>>> any firmware distribution that derives from it.
>>>> 
>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>> are slow to update their snapshots or have just given up doing that
>>>> entirely. Distributing it for external applications like OS loaders
>>>> would mean this can be leveraged probably no earlier than 10 years
>>>> from now. And for in-firmware things, I have a hard time thinking about a
>>> use-case that outweighs the drawbacks.
>>>> 
>>>> 
>>>> To shift the topic slightly back to GSoC, however, I'm willing to work
>>>> on other items on the task list. Unit testing and an ACPICA
>>>> application are the alternative projects I had thought about. I need
>>>> to choose fairly soon as the proposal deadline is next Tuesday. I know
>>>> a tiny bit about porting ACPICA as I also have plans to incorporate it into my
>>> own project.
>>>> 
>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>> them as appropriate yet and I’m not here to drive you away from this
>>>> specific task (or the others). However, I’m still curious and
>>>> concerned. :)
>>>> 
>>>> Best regards,
>>>> Marvin
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Pedro Falcato
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>> 
>> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Ada Christine]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

Hi Everybody

I've read all the discussion here and condensed my plan into a short
project proposal. It's a little short and light on detail at the moment
because I'm pressed for time for other matters today, but I wanted to get
something in before EOD today as requested. Anybody else's input or a
change in the overall strategy to allow for code sharing between DXE
modules, whether it be prelinking or some kind of function pointer table is
absolutely welcome and I'm not attached to any particular way of solving
the code repetition problem. You can find my proposal here
https://summerofcode.withgoogle.com/proposals/details/whGX9tXL

Looking forward to your commentary!

Thanks!
- Ada Christine

[-- Attachment #2: Type: text/html, Size: 900 bytes --]

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

CC Mike (proposal review as per announcement mail)

Hey Ada,

I can neither decide on nor even view your proposal (I think that's up 
to Nate and Mike?), but I had a brief conversation with Vitaly about the 
Amaranth downstream. There are other potentially technologically related 
topics Vitaly's team wants to deploy, including driver sandboxing and 
ASLR (both will probably significantly impact paging). The easiest route 
for these two is likely to let go of identity mapping. *If* this is 
feasible and will be accepted upstream, prelinking might become a much 
simpler matter. For memory protection, all PE/COFF image sections must 
be page-aligned anyway, so depending on how the more sophisticated 
paging would actually work, there may be a lot of wiggle room for where 
to load modules wrt virtual addresses. In *simple and naive* theory, 
they could all be assigned a virtual base address at UEFI image 
construction (which will be free from any physical memory layout 
constraints due to non-identity mapping) and ASLR could just be a slide 
value that shifts the entire executable UEFI address space around 
(randomised). With (virtual) addresses known at build-time, none of that 
"custom relocation" madness I mentioned before is relevant (gladly). Of 
course, there needs to be discussion whether fine-grained ASLR would be 
worth the trouble first.

To get more input on the "ecosystem" of security features mentioned 
(ASLR, sandboxing, prelinking), we will try to discuss it with Microsoft 
next week. If you are interested in a prelinking route, I can let you 
know. This would unlikely be quick to deploy, however, and it would need 
strong support from Intel. I think the overall pool of ideas is clear 
now and I'll leave it to you and Nate. Good luck!

Best regards,
Marvin

On 15.04.22 14:09, Ada Christine wrote:
> Hi Everybody
>
> I've read all the discussion here and condensed my plan into a short 
> project proposal. It's a little short and light on detail at the 
> moment because I'm pressed for time for other matters today, but I 
> wanted to get something in before EOD today as requested. Anybody 
> else's input or a change in the overall strategy to allow for code 
> sharing between DXE modules, whether it be prelinking or some kind of 
> function pointer table is absolutely welcome and I'm not attached to 
> any particular way of solving the code repetition problem. You can 
> find my proposal here 
> https://summerofcode.withgoogle.com/proposals/details/whGX9tXL
>
> Looking forward to your commentary!
>
> Thanks!
> - Ada Christine

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Zimmer, Vincent]

Fyi
There is a running list of some edk2 defense-in-depth work at https://github.com/jyao1/SecurityEx/blob/master/Summary.md, too, including ASLR, if you haven't already seen that material

-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Marvin Häuser
Sent: Friday, April 15, 2022 5:31 AM
To: Ada Christine <adachristine18@gmail.com>; edk2-devel-groups-io <devel@edk2.groups.io>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Andrew Fish <afish@apple.com>; discuss@edk2.groups.io; Pedro Falcato <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

CC Mike (proposal review as per announcement mail)

Hey Ada,

I can neither decide on nor even view your proposal (I think that's up to Nate and Mike?), but I had a brief conversation with Vitaly about the Amaranth downstream. There are other potentially technologically related topics Vitaly's team wants to deploy, including driver sandboxing and ASLR (both will probably significantly impact paging). The easiest route for these two is likely to let go of identity mapping. *If* this is feasible and will be accepted upstream, prelinking might become a much simpler matter. For memory protection, all PE/COFF image sections must be page-aligned anyway, so depending on how the more sophisticated paging would actually work, there may be a lot of wiggle room for where to load modules wrt virtual addresses. In *simple and naive* theory, they could all be assigned a virtual base address at UEFI image construction (which will be free from any physical memory layout constraints due to non-identity mapping) and ASLR could just be a slide value that shifts the entire executable UEFI address space around (randomised). With (virtual) addresses known at build-time, none of that "custom relocation" madness I mentioned before is relevant (gladly). Of course, there needs to be discussion whether fine-grained ASLR would be worth the trouble first.

To get more input on the "ecosystem" of security features mentioned (ASLR, sandboxing, prelinking), we will try to discuss it with Microsoft next week. If you are interested in a prelinking route, I can let you know. This would unlikely be quick to deploy, however, and it would need strong support from Intel. I think the overall pool of ideas is clear now and I'll leave it to you and Nate. Good luck!

Best regards,
Marvin

On 15.04.22 14:09, Ada Christine wrote:
> Hi Everybody
>
> I've read all the discussion here and condensed my plan into a short 
> project proposal. It's a little short and light on detail at the 
> moment because I'm pressed for time for other matters today, but I 
> wanted to get something in before EOD today as requested. Anybody 
> else's input or a change in the overall strategy to allow for code 
> sharing between DXE modules, whether it be prelinking or some kind of 
> function pointer table is absolutely welcome and I'm not attached to 
> any particular way of solving the code repetition problem. You can 
> find my proposal here 
> https://summerofcode.withgoogle.com/proposals/details/whGX9tXL
>
> Looking forward to your commentary!
>
> Thanks!
> - Ada Christine

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

Hey Vincent,

In fact I haven't, thanks a lot! Are there any known blockers for these 
outside development resources? Except for C++, they are things we'd want 
asap downstream. I guess rather than OverflowDetectionPkg, ASan has 
higher priority here.

Best regards,
Marvin

On 15.04.22 15:31, Zimmer, Vincent wrote:
> Fyi
> There is a running list of some edk2 defense-in-depth work at https://github.com/jyao1/SecurityEx/blob/master/Summary.md, too, including ASLR, if you haven't already seen that material
>
> -----Original Message-----
> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Marvin Häuser
> Sent: Friday, April 15, 2022 5:31 AM
> To: Ada Christine <adachristine18@gmail.com>; edk2-devel-groups-io <devel@edk2.groups.io>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Andrew Fish <afish@apple.com>; discuss@edk2.groups.io; Pedro Falcato <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>
> CC Mike (proposal review as per announcement mail)
>
> Hey Ada,
>
> I can neither decide on nor even view your proposal (I think that's up to Nate and Mike?), but I had a brief conversation with Vitaly about the Amaranth downstream. There are other potentially technologically related topics Vitaly's team wants to deploy, including driver sandboxing and ASLR (both will probably significantly impact paging). The easiest route for these two is likely to let go of identity mapping. *If* this is feasible and will be accepted upstream, prelinking might become a much simpler matter. For memory protection, all PE/COFF image sections must be page-aligned anyway, so depending on how the more sophisticated paging would actually work, there may be a lot of wiggle room for where to load modules wrt virtual addresses. In *simple and naive* theory, they could all be assigned a virtual base address at UEFI image construction (which will be free from any physical memory layout constraints due to non-identity mapping) and ASLR could just be a slide value that shifts the entire executable UEFI address space around (randomised). With (virtual) addresses known at build-time, none of that "custom relocation" madness I mentioned before is relevant (gladly). Of course, there needs to be discussion whether fine-grained ASLR would be worth the trouble first.
>
> To get more input on the "ecosystem" of security features mentioned (ASLR, sandboxing, prelinking), we will try to discuss it with Microsoft next week. If you are interested in a prelinking route, I can let you know. This would unlikely be quick to deploy, however, and it would need strong support from Intel. I think the overall pool of ideas is clear now and I'll leave it to you and Nate. Good luck!
>
> Best regards,
> Marvin
>
> On 15.04.22 14:09, Ada Christine wrote:
>> Hi Everybody
>>
>> I've read all the discussion here and condensed my plan into a short
>> project proposal. It's a little short and light on detail at the
>> moment because I'm pressed for time for other matters today, but I
>> wanted to get something in before EOD today as requested. Anybody
>> else's input or a change in the overall strategy to allow for code
>> sharing between DXE modules, whether it be prelinking or some kind of
>> function pointer table is absolutely welcome and I'm not attached to
>> any particular way of solving the code repetition problem. You can
>> find my proposal here
>> https://summerofcode.withgoogle.com/proposals/details/whGX9tXL
>>
>> Looking forward to your commentary!
>>
>> Thanks!
>> - Ada Christine
>
>
>
>
>
>
>
> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Zimmer, Vincent]

Historically the challenge we had w/ upstreaming some features was if the compiler intrinsic for which a particular feature was dependent didn't have public documentation or wasn't supported by all of the compilers that EDKII supported. For the former, this lack of info would lead to the patches being rejected. 

Vincent

-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Marvin Häuser
Sent: Friday, April 15, 2022 6:40 AM
To: discuss@edk2.groups.io; Zimmer, Vincent <vincent.zimmer@intel.com>; Ada Christine <adachristine18@gmail.com>; edk2-devel-groups-io <devel@edk2.groups.io>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Andrew Fish <afish@apple.com>; Pedro Falcato <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Vincent,

In fact I haven't, thanks a lot! Are there any known blockers for these outside development resources? Except for C++, they are things we'd want asap downstream. I guess rather than OverflowDetectionPkg, ASan has higher priority here.

Best regards,
Marvin

On 15.04.22 15:31, Zimmer, Vincent wrote:
> Fyi
> There is a running list of some edk2 defense-in-depth work at 
> https://github.com/jyao1/SecurityEx/blob/master/Summary.md, too, 
> including ASLR, if you haven't already seen that material
>
> -----Original Message-----
> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of 
> Marvin Häuser
> Sent: Friday, April 15, 2022 5:31 AM
> To: Ada Christine <adachristine18@gmail.com>; edk2-devel-groups-io 
> <devel@edk2.groups.io>; Desimone, Nathaniel L 
> <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>; 
> Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Andrew Fish <afish@apple.com>; discuss@edk2.groups.io; Pedro 
> Falcato <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>
> CC Mike (proposal review as per announcement mail)
>
> Hey Ada,
>
> I can neither decide on nor even view your proposal (I think that's up to Nate and Mike?), but I had a brief conversation with Vitaly about the Amaranth downstream. There are other potentially technologically related topics Vitaly's team wants to deploy, including driver sandboxing and ASLR (both will probably significantly impact paging). The easiest route for these two is likely to let go of identity mapping. *If* this is feasible and will be accepted upstream, prelinking might become a much simpler matter. For memory protection, all PE/COFF image sections must be page-aligned anyway, so depending on how the more sophisticated paging would actually work, there may be a lot of wiggle room for where to load modules wrt virtual addresses. In *simple and naive* theory, they could all be assigned a virtual base address at UEFI image construction (which will be free from any physical memory layout constraints due to non-identity mapping) and ASLR could just be a slide value that shifts the entire executable UEFI address space around (randomised). With (virtual) addresses known at build-time, none of that "custom relocation" madness I mentioned before is relevant (gladly). Of course, there needs to be discussion whether fine-grained ASLR would be worth the trouble first.
>
> To get more input on the "ecosystem" of security features mentioned (ASLR, sandboxing, prelinking), we will try to discuss it with Microsoft next week. If you are interested in a prelinking route, I can let you know. This would unlikely be quick to deploy, however, and it would need strong support from Intel. I think the overall pool of ideas is clear now and I'll leave it to you and Nate. Good luck!
>
> Best regards,
> Marvin
>
> On 15.04.22 14:09, Ada Christine wrote:
>> Hi Everybody
>>
>> I've read all the discussion here and condensed my plan into a short 
>> project proposal. It's a little short and light on detail at the 
>> moment because I'm pressed for time for other matters today, but I 
>> wanted to get something in before EOD today as requested. Anybody 
>> else's input or a change in the overall strategy to allow for code 
>> sharing between DXE modules, whether it be prelinking or some kind of 
>> function pointer table is absolutely welcome and I'm not attached to 
>> any particular way of solving the code repetition problem. You can 
>> find my proposal here 
>> https://summerofcode.withgoogle.com/proposals/details/whGX9tXL
>>
>> Looking forward to your commentary!
>>
>> Thanks!
>> - Ada Christine
>
>
>
>
>
>
>
> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Yao, Jiewen]

Exactly. Lacking of doc from different compilers is very painful.
The Asan or UBsan support is just a hack - https://github.com/jyao1/SecurityEx/blob/master/StackCheckPkg/Library/StackCheckLib/ASanStub.c
https://github.com/jyao1/SecurityEx/blob/master/StackCheckPkg/Library/StackCheckLib/UBSanStub.c


Another challenge is: What do we do in case of violation?
If you just want a POC or debug purpose, it is easy to use deadloop.
If you want to a production, then what do we do in RT driver? deadloop in OS is not a good idea.


Thank you
Yao Jiewen

> -----Original Message-----
> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Zimmer,
> Vincent
> Sent: Friday, April 15, 2022 10:54 PM
> To: discuss@edk2.groups.io; mhaeuser@posteo.de; Ada Christine
> <adachristine18@gmail.com>; edk2-devel-groups-io <devel@edk2.groups.io>;
> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Mike Wolan
> <mwolan@caseking.de>; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Andrew Fish <afish@apple.com>; Pedro Falcato
> <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> 
> Historically the challenge we had w/ upstreaming some features was if the
> compiler intrinsic for which a particular feature was dependent didn't have
> public documentation or wasn't supported by all of the compilers that EDKII
> supported. For the former, this lack of info would lead to the patches being
> rejected.
> 
> Vincent
> 
> -----Original Message-----
> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Marvin
> Häuser
> Sent: Friday, April 15, 2022 6:40 AM
> To: discuss@edk2.groups.io; Zimmer, Vincent <vincent.zimmer@intel.com>; Ada
> Christine <adachristine18@gmail.com>; edk2-devel-groups-io
> <devel@edk2.groups.io>; Desimone, Nathaniel L
> <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>;
> Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Andrew Fish <afish@apple.com>; Pedro Falcato
> <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> 
> Hey Vincent,
> 
> In fact I haven't, thanks a lot! Are there any known blockers for these outside
> development resources? Except for C++, they are things we'd want asap
> downstream. I guess rather than OverflowDetectionPkg, ASan has higher priority
> here.
> 
> Best regards,
> Marvin
> 
> On 15.04.22 15:31, Zimmer, Vincent wrote:
> > Fyi
> > There is a running list of some edk2 defense-in-depth work at
> > https://github.com/jyao1/SecurityEx/blob/master/Summary.md, too,
> > including ASLR, if you haven't already seen that material
> >
> > -----Original Message-----
> > From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
> > Marvin Häuser
> > Sent: Friday, April 15, 2022 5:31 AM
> > To: Ada Christine <adachristine18@gmail.com>; edk2-devel-groups-io
> > <devel@edk2.groups.io>; Desimone, Nathaniel L
> > <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>;
> > Kinney, Michael D <michael.d.kinney@intel.com>
> > Cc: Andrew Fish <afish@apple.com>; discuss@edk2.groups.io; Pedro
> > Falcato <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
> > Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> >
> > CC Mike (proposal review as per announcement mail)
> >
> > Hey Ada,
> >
> > I can neither decide on nor even view your proposal (I think that's up to Nate
> and Mike?), but I had a brief conversation with Vitaly about the Amaranth
> downstream. There are other potentially technologically related topics Vitaly's
> team wants to deploy, including driver sandboxing and ASLR (both will probably
> significantly impact paging). The easiest route for these two is likely to let go of
> identity mapping. *If* this is feasible and will be accepted upstream, prelinking
> might become a much simpler matter. For memory protection, all PE/COFF
> image sections must be page-aligned anyway, so depending on how the more
> sophisticated paging would actually work, there may be a lot of wiggle room for
> where to load modules wrt virtual addresses. In *simple and naive* theory, they
> could all be assigned a virtual base address at UEFI image construction (which
> will be free from any physical memory layout constraints due to non-identity
> mapping) and ASLR could just be a slide value that shifts the entire executable
> UEFI address space around (randomised). With (virtual) addresses known at
> build-time, none of that "custom relocation" madness I mentioned before is
> relevant (gladly). Of course, there needs to be discussion whether fine-grained
> ASLR would be worth the trouble first.
> >
> > To get more input on the "ecosystem" of security features mentioned (ASLR,
> sandboxing, prelinking), we will try to discuss it with Microsoft next week. If you
> are interested in a prelinking route, I can let you know. This would unlikely be
> quick to deploy, however, and it would need strong support from Intel. I think
> the overall pool of ideas is clear now and I'll leave it to you and Nate. Good luck!
> >
> > Best regards,
> > Marvin
> >
> > On 15.04.22 14:09, Ada Christine wrote:
> >> Hi Everybody
> >>
> >> I've read all the discussion here and condensed my plan into a short
> >> project proposal. It's a little short and light on detail at the
> >> moment because I'm pressed for time for other matters today, but I
> >> wanted to get something in before EOD today as requested. Anybody
> >> else's input or a change in the overall strategy to allow for code
> >> sharing between DXE modules, whether it be prelinking or some kind of
> >> function pointer table is absolutely welcome and I'm not attached to
> >> any particular way of solving the code repetition problem. You can
> >> find my proposal here
> >> https://summerofcode.withgoogle.com/proposals/details/whGX9tXL
> >>
> >> Looking forward to your commentary!
> >>
> >> Thanks!
> >> - Ada Christine
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

On 15.04.22 17:02, Yao, Jiewen wrote:
> Exactly. Lacking of doc from different compilers is very painful.
> The Asan or UBsan support is just a hack - https://github.com/jyao1/SecurityEx/blob/master/StackCheckPkg/Library/StackCheckLib/ASanStub.c
> https://github.com/jyao1/SecurityEx/blob/master/StackCheckPkg/Library/StackCheckLib/UBSanStub.c
>
>
> Another challenge is: What do we do in case of violation?
> If you just want a POC or debug purpose, it is easy to use deadloop.

I mean, did you plan to deploy ASan in production? I'm not sure a full 
implementation is performant or compatible with space constraints. Being 
a debug feature, I'm surprised this is a reason for rejection. There is 
toolchain-specific debug stuff live already (e.g. symbols support for 
the RVCT debugger).

> If you want to a production, then what do we do in RT driver? deadloop in OS is not a good idea.

As for what to do with RT drivers, I wonder whether Windows can 
cooperate? With Virtualisation-Based Security, there is a (hopefully) 
trusted core that can still intervene kernel-mode drivers from acting 
up. I.e., on fault, can one just fail the RT service call and report to 
the OS for it to apply whatever policy it considers appropriate, like 
force-unload the calling driver (if any)? If you do not consider the 
special-case of SMM-guarded security a malicious OS may want to exploit 
(which is a reason why the Amaranth downstream will outright deprecate 
SMM guarding), a malicious OS could just perform any action it may 
provoke the firmware to do itself, so I feel like it's fair if the OS 
was trusted, it cooperated, and it took handling authority.

I guess the only other alternatives are tolerant failure (continue 
business as usual after returning an error) and intolerant failure (shut 
down RT services till reboot).

Of course this should be configurable, so that high-security platforms 
can indeed deadloop or reset the platform.

Best regards,
Marvin

>
>
> Thank you
> Yao Jiewen
>
>> -----Original Message-----
>> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Zimmer,
>> Vincent
>> Sent: Friday, April 15, 2022 10:54 PM
>> To: discuss@edk2.groups.io; mhaeuser@posteo.de; Ada Christine
>> <adachristine18@gmail.com>; edk2-devel-groups-io <devel@edk2.groups.io>;
>> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Mike Wolan
>> <mwolan@caseking.de>; Kinney, Michael D <michael.d.kinney@intel.com>
>> Cc: Andrew Fish <afish@apple.com>; Pedro Falcato
>> <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>
>> Historically the challenge we had w/ upstreaming some features was if the
>> compiler intrinsic for which a particular feature was dependent didn't have
>> public documentation or wasn't supported by all of the compilers that EDKII
>> supported. For the former, this lack of info would lead to the patches being
>> rejected.
>>
>> Vincent
>>
>> -----Original Message-----
>> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of Marvin
>> Häuser
>> Sent: Friday, April 15, 2022 6:40 AM
>> To: discuss@edk2.groups.io; Zimmer, Vincent <vincent.zimmer@intel.com>; Ada
>> Christine <adachristine18@gmail.com>; edk2-devel-groups-io
>> <devel@edk2.groups.io>; Desimone, Nathaniel L
>> <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>;
>> Kinney, Michael D <michael.d.kinney@intel.com>
>> Cc: Andrew Fish <afish@apple.com>; Pedro Falcato
>> <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>
>> Hey Vincent,
>>
>> In fact I haven't, thanks a lot! Are there any known blockers for these outside
>> development resources? Except for C++, they are things we'd want asap
>> downstream. I guess rather than OverflowDetectionPkg, ASan has higher priority
>> here.
>>
>> Best regards,
>> Marvin
>>
>> On 15.04.22 15:31, Zimmer, Vincent wrote:
>>> Fyi
>>> There is a running list of some edk2 defense-in-depth work at
>>> https://github.com/jyao1/SecurityEx/blob/master/Summary.md, too,
>>> including ASLR, if you haven't already seen that material
>>>
>>> -----Original Message-----
>>> From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
>>> Marvin Häuser
>>> Sent: Friday, April 15, 2022 5:31 AM
>>> To: Ada Christine <adachristine18@gmail.com>; edk2-devel-groups-io
>>> <devel@edk2.groups.io>; Desimone, Nathaniel L
>>> <nathaniel.l.desimone@intel.com>; Mike Wolan <mwolan@caseking.de>;
>>> Kinney, Michael D <michael.d.kinney@intel.com>
>>> Cc: Andrew Fish <afish@apple.com>; discuss@edk2.groups.io; Pedro
>>> Falcato <pedro.falcato@gmail.com>; Shi, Steven <steven.shi@intel.com>
>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>
>>> CC Mike (proposal review as per announcement mail)
>>>
>>> Hey Ada,
>>>
>>> I can neither decide on nor even view your proposal (I think that's up to Nate
>> and Mike?), but I had a brief conversation with Vitaly about the Amaranth
>> downstream. There are other potentially technologically related topics Vitaly's
>> team wants to deploy, including driver sandboxing and ASLR (both will probably
>> significantly impact paging). The easiest route for these two is likely to let go of
>> identity mapping. *If* this is feasible and will be accepted upstream, prelinking
>> might become a much simpler matter. For memory protection, all PE/COFF
>> image sections must be page-aligned anyway, so depending on how the more
>> sophisticated paging would actually work, there may be a lot of wiggle room for
>> where to load modules wrt virtual addresses. In *simple and naive* theory, they
>> could all be assigned a virtual base address at UEFI image construction (which
>> will be free from any physical memory layout constraints due to non-identity
>> mapping) and ASLR could just be a slide value that shifts the entire executable
>> UEFI address space around (randomised). With (virtual) addresses known at
>> build-time, none of that "custom relocation" madness I mentioned before is
>> relevant (gladly). Of course, there needs to be discussion whether fine-grained
>> ASLR would be worth the trouble first.
>>> To get more input on the "ecosystem" of security features mentioned (ASLR,
>> sandboxing, prelinking), we will try to discuss it with Microsoft next week. If you
>> are interested in a prelinking route, I can let you know. This would unlikely be
>> quick to deploy, however, and it would need strong support from Intel. I think
>> the overall pool of ideas is clear now and I'll leave it to you and Nate. Good luck!
>>> Best regards,
>>> Marvin
>>>
>>> On 15.04.22 14:09, Ada Christine wrote:
>>>> Hi Everybody
>>>>
>>>> I've read all the discussion here and condensed my plan into a short
>>>> project proposal. It's a little short and light on detail at the
>>>> moment because I'm pressed for time for other matters today, but I
>>>> wanted to get something in before EOD today as requested. Anybody
>>>> else's input or a change in the overall strategy to allow for code
>>>> sharing between DXE modules, whether it be prelinking or some kind of
>>>> function pointer table is absolutely welcome and I'm not attached to
>>>> any particular way of solving the code repetition problem. You can
>>>> find my proposal here
>>>> https://summerofcode.withgoogle.com/proposals/details/whGX9tXL
>>>>
>>>> Looking forward to your commentary!
>>>>
>>>> Thanks!
>>>> - Ada Christine
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Brian J. Johnson]

Nate, Andrew, Marvin, Pedro, Ada, et al,

This is a great discussion.  I've been debating where to weigh in...

I agree that some sort of library sharing to reduce image size would be 
very helpful.  Although some parts of the BIOS are built separately, 
large parts of it are compiled at the same time, so there should be 
ample opportunity for tool-directed sharing.  Some form of pre-linking 
modules together may be the easiest way to do that.  But however it gets 
implemented, we should try to make it automatic:  requiring library 
writers to manually add a lot of metadata or write thunks or shims would 
be a barrier to adoption.

I also agree that PEI has acquired far too much functionality over time, 
and we badly need to find some way to reverse that.  PEI is pulling in 
more and more of the h/w initialization, leaving DXE mainly to provide 
the boot interface (ACPI, UEFI services, setup screens, etc.)  That puts 
more and more pressure on XIP storage and cache-as-RAM.  How can we 
encourage a change?

Thanks,
Brian J. Johnson


On 4/15/22 03:15, Nate DeSimone wrote:
> Hi Andrew,
> 
> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
>>
>>
>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone <nathaniel.l.desimone@intel.com> wrote:
>>>
>>> Hi Marvin,
>>>
>>>> -----Original Message-----
>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin
>>>> Häuser
>>>> Sent: Thursday, April 14, 2022 12:56 AM
>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>>>> <nathaniel.l.desimone@intel.com>
>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>>>> <steven.shi@intel.com>
>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>
>>>> I feel like there are so many much easier solutions to this problem that are at
>>>> most limited by the clear specification. The UEFI specification with regards to
>>>> booting and all of that obviously is of utmost importance.
>>>
>>> If you have a better idea that retains compatibility with the existing UEFI PI then I would be happy to hear it. Ultimately anything we do needs to be a pure extension that retains compatibility with old code. Given that restriction having the ability to coalesce all the LibraryClasses into a single module and have cross-module linking seems like the best way to handle it to me.
>>>
>>>> The UEFI PI specification parts that deal about internal structure, as far as I know, are
>>>> only in place to make it easy to integrate Intel IP.
>>>
>>> Its not just for Intel. The biggest reason for it to increase the standardization of the boot flow across the PC ecosystem. We have learned from experience that firmware is super critical to get a product out the door but it is also difficult to write. So we try to make it as reusable as humanly possible.
>>>
>>>> In fact, I don’t *know*, but I’m pretty sure the very strict separation between PEI and DXE was
>>>> preserved mostly because MRC was 32-bit-only for a long time. Glad that
>>>> seems to have been resolved, AMD does memory init by PSP nowadays.
>>>
>>> Having less complex early stages chain load more complex later stages is a common design pattern in firmware, not just UEFI. For example, your typical ARM system loads kinda like this:
>>>
>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
>>>
>>> Comparing relative complexity I believe the Intel UEFI PI design is actually pretty simple when you consider how much it gets done:
>>>
>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>>>
>>> My biggest criticism of the design is that the strict separation between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
>>>
>>> There are a few reasons why PEI was 32-bit for quite some time. The biggest one is the code size increase, 64-bit x86 code is 20-30% larger than 32-bit x86 code. Since the only RAM Pre-Memory code has access to is the cache onboard the processor and for security reasons all that code has to fit within that RAM we generally do everything we can to make that image as small as possible. Second, 64-bit requires a page table and since we desired to keep PEI simple we tried to avoid that. Finally, the PI spec didn't allow a 64-bit PEI until recently. MRC is 32-bit code just because that is what PEI happens to be. Porting it to 64-bit is not terribly difficult.
>>>
>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any data structures that get shared between PEI and DXE (HOBs, PCDs, etc.) need to resolve to the same size and packing. LibraryClasses need to be written to compile properly in both modes. In the case of FSP API mode you need to resort to thunking between 32 and 64-bit modes during DXE. More or less we decided that the costs are starting to outweigh the advantages.
>>>
>>
>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
>>
>> In the PC ecosystem a single chipset family can power thousands of unique designs. So the  DRAM memory needs to be external, support lots of different chipset packages(signal integrity...), support the lowest cost through the highest cost DRAM and thousands of different board layouts. So programing DRAM takes a masters degree in antenna design. I’ve seen MRC (Memory Reference Code) with over a MiB of DEBUG prints in it, and it literally is printing histograms of what it is tuning. So all this code has to run before the system has any DRAM, thus it is running using the cache as RAM. I’ve not looked at the x86 architecture specs form the vendors in a while, but back in the day they did not support page tables in ROM or pinned cached. Now it might work, but if it breaks your CPU vendor blames you so you don’t code PEI in X64….
>>
>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a long time ago but I seem to remember the mask ROM used a table in NOR FLASH to init memory and then copied an image from NOR FLASH into DRAM and jumped to it. So PEI is kind of not really needed and we implemented a PrePEI and jumped directly to DXE.
>>
>> Given I was around back in the day when all this stuff was designed I can say SEC was always a place holder for security code, as security code always has to run 1st. PEI (Pre EFI) was designed to get DRAM programmed and then jump to DXE. It kind of also fell in naturally to ACPI S3 flow since that was turning memory back on. When we designed PEI we kind of though of it more like a boot loader stage for the firmware that turned on memory and all the work would happen in DXE. Then reality strikes and the existing BIOS assembly programmers start learning C (lots of  cranky people) and they start having to learn all about PEI to turn on memory. They had to write a big chunk of code for the memory init in PEI. These folks had never written any EFI code, so to them it was easier to move a lot of the chipset init code into PEI as that is the world they had to figure out to get memory turned on. I mean why learn EFI if you don’t have to? So that is how we get so much code in IA32 (i386) on some platforms. This start kind of biased future choices and how to enable non edk2 code bases….
> 
> One of the big reasons a lot of code that should have been written in DXE ended up in PEI is unfortunately due to the FSP and its inability to support DXE code.
> 
>>
>> Thanks,
>>
>> Andrew Fish
>>
>>
>>>>
>>>> For many good reasons, Linux does not provide a stable kernel API. This
>>>> allows to easily deploy breaking changes across the entire codebase.
>>>> Obviously, this is infeasible at a large scale when you need to integrate IP
>>>> blobs, but it would already help to break the expectation that UEFI PI is a
>>>> perfectly forwards- and backwards-compatible system. DXE has SetMem and
>>>> CopyMem as part of gBS. Maybe I don’t like it that much as part of the spec
>>>> itself, but it’s a good idea technically. I’d probably opt to have a quickly
>>>> accessible implementation detail of important function pointers appended to
>>>> PEI and DXE services, used by in-tree modules. This may break both
>>>> forwards- and backwards-compatibility, but as it only applies to in-tree
>>>> modules, that is fine so long as we let go of the compatibility notions. PPIs
>>>> and protocols are an option too of course, but they have a lookup
>>>> performance penalty. Compared to dynamic linking, that should hopefully be
>>>> negligible however.
>>>>
>>>> Absolutely optional to read, I don’t intend to waste anyone’s time much,
>>>> some philosophical stuff about my rationale:
>>>>
>>>> If you started UEFI from scratch right now, would it have strictly separated
>>>> PEI and DXE?
>>>
>>> For sure a clean slate design started today would look a lot different than PEI/DXE... which was a clean slate design circa 1998 😊. In my opinion, if we were to suddenly go back to the drawing board we would build something that is much closer to a full OS now than we did back then. There have been cases where not being able to use interrupt handlers and not having thread scheduling has prevented implementation of desired features. The ARM guys built LittleKernel (https://github.com/littlekernel/lk) for a lot of these reasons. In the data center world some have decided to go to the extreme of putting an entire copy of Linux in SPI so they can do a network boot that downloads the OS image using BitTorrent!
>>>
>>>> The duplication between PEI and DXE core, and by extension
>>>> MM core, would be my most obvious place to start reducing size. I would
>>>> probably opt for a PEI/DXE hybrid where it starts in „minimal mode“ (maybe
>>>> think of PEI more like a microkernel here) and after memory is initialised, the
>>>> rest of DXE is loaded. And MM core would get no loading at all, this
>>>> requirement has gladly been dropped ages ago. Just one prelinked snapshot
>>>> of the address space with a relocation table and a safe self-relocator on entry
>>>> (this is needed at the very least for ARM).
>>>>
>>>> Ironically, with my idea for MM, dynamic loading would be free as everything
>>>> is prelinked anyway. The same is true for PEI XIP, it is prelinked by nature.
>>>
>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that does get used for the PEI GOP driver.
>>>
>>>> What I do not like is the additional dynamic linking code at load-time for non-
>>>> XIP modules. Though, the more I think about it, the more I wonder whether
>>>> not the entirety of UEFI could be composed of prelinked, relocatable
>>>> memory snapshots without traditional image loading entirely (for in-FW
>>>> stuff). macOS has a similar concept with its “Kernel Collections”. Well, way
>>>> too much off-topic now. :)
>>>
>>> If you make the assumption that 100% of the code is compiled all at once then yes that works. UEFI was designed so that assumption does not need to be true. There are good use cases for it: OpROMs, generic OS loaders, network boot, etc.
>>>
>>> Thanks,
>>> Nate
>>>
>>>>
>>>> Why am I explaining all this despite the fact everyone knows this will never
>>>> happen? Because I don’t like the notion of fixing issues of an already
>>>> overcomplicated system by adding even more complicated stuff. Especially
>>>> when the existing overcomplicated stuff is already uncomfortably broken.
>>>>
>>>> Best regards,
>>>> Marvin
>>>>
>>>>> For XIP PEI code… it will really help and would be very timely since the
>>>> transition of PEI from 32-bit to 64-bit is going to increase the size of PEI by
>>>> ~20%.
>>>>>
>>>>> Thanks,
>>>>> Nate
>>>>>
>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>>>> <mhaeuser@posteo.de>
>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>>>> <steven.shi@intel.com>
>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>
>>>>> Hi Marvin, Ada,
>>>>>
>>>>> Some comments:
>>>>>
>>>>> I don't think the purpose of the dynamic linker is to treat EFI as a
>>>>> complete operating system, but to try to eliminate the static linking
>>>>> that may be needlessly duplicating code that could instead be put in a
>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are linked
>>>> into a *lot* of .efi, instead of being just a library. It'd be nice to see some
>>>> numbers on this (something like Google's bloaty could be run on every .efi
>>>> file, in order to understand how much file space we would actually save).
>>>>>
>>>>> Other comments inline.
>>>>>
>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>
>>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>>>> i was replying via the groups.io<http://groups.io > web interface, I'm
>>>>> guessing that messed up the thread? i haven't used mailing lists
>>>>> before and don't know how they work. I'll use my mail client from here on.
>>>>>
>>>>> I'm on board with not treating EFI as an operating system. the more i
>>>>> think about it the more it looks like scope creep.
>>>>>
>>>>> Agreed.
>>>>>
>>>>>
>>>>> I'm not quite as enthusiastic
>>>>> about it as i was at first glance.
>>>>>
>>>>> I'm still keen on doing my gsoc proposal for edk, though, and even if
>>>>> this task and the acpica application are decided to be out of scope
>>>>> unit testing,
>>>>>
>>>>> How about fuzz-testing? This is also something edk2 needs quite badly. At
>>>> Acidanthera, we compile edk2 code in userspace outside the edk2 build
>>>> system and fuzz with dummy applications.
>>>>>
>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>>> https://llvm.org/docs/LibFuzzer.html ) and is something I could happily
>>>> mentor.
>>>>> clang integration
>>>>>
>>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>>> There are working UBSan concepts, but they also need to be mainlined.
>>>>>
>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be me and
>>>> some other, more senior, mentor (possibly Steven Shi, which I included in
>>>> the task).
>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly
>>>>> some other sanitizer it's quite possible that it could be considered a large
>>>> project (which means more hours but a larger stipend too). Fuzzing +
>>>> coverage could be very nice additions to this project idea.
>>>>> Also, is stress-testing a decent idea?
>>>>>
>>>>>
>>>>> and source-level debugging are all relevant to my interests.
>>>>>
>>>>> how about your ideas for security stuff?
>>>>>
>>>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
>>>> only work on MMRAM. Also evaluation of how to best avoid pointers in MM
>>>> communication buffers would be nice.
>>>>>
>>>>> There also is a bunch of other stuff, like working out moving a part of
>>>> CpuDxe into DxeCore to have memory protection live immediately, memory
>>>> protection in PEI, a replacement for the TE format (it’s buggy and most
>>>> platforms mostly abandoned it over various issues), and alternatives to
>>>> guarding critical code with SMM (like allowing NVRAM commits only as part
>>>> of a reboot).
>>>>>
>>>>> I personally find all of those projects very important, but I cannot
>>>>> promise many people agree. Especially those that impose global changes
>>>>> (most notably the TE replacement) may be very tedious to submit.
>>>>> Gladly, I believe you can submit multiple proposals (?)
>>>>>
>>>>> Best regards,
>>>>> Marvin
>>>>>
>>>>>
>>>>> I'm not very knowledgeable about
>>>>> trusted platform or secure boot but I'm willing to learn whatever is
>>>>> necessary to get something spun up for my proposal.
>>>>>
>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>
>>>>>
>>>>> Do you use the “reply all” option in your mail client? Looks like my
>>>>> CCs have been dropped again. Comments inline.
>>>>>
>>>>> On 13. Apr 2022, at 12:54, Ada Christine
>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>>> wrote:
>>>>> Hi, Marvin
>>>>>
>>>>> Its similarity to my own latest experiment is the key to what grabbed
>>>>> my attention. I have no particular use case in mind for it, but I see
>>>>> its potential for anybody developing larger applications in that when
>>>>> a library is changed there's no need to distribute a new version of
>>>>> the whole binary, just the relevant library module.
>>>>>
>>>>> I really do not like the trend of treating UEFI as a full-fledged OS -
>>>>> it is not. The most used UEFI applications, OS loaders, are really not
>>>>> that huge and are distributed as part of the OS image anyway. Even for
>>>>> less used applications, you will always get a full snapshot anyhow.
>>>>> Gladly we don’t have auto-update and package management yet. :)
>>>>>
>>>>>
>>>>> I slept on it and it occurred to me that the whole thing could operate
>>>>> similarly to the shell protocol in that the linker/loader is itself an
>>>>> application that does a LoadImage() on the application needing dynamic
>>>>> linking facilities.
>>>>>
>>>>> That would mean the linker itself is shipped with every application
>>>>> that requires it? Otherwise it doesn’t make much sense for it to be an
>>>>> app and below’s problems apply.
>>>>>
>>>>> If however the whole plan is making the linker as a DXE and including
>>>>> it with the firmware, that I'm not quite as sure about. That would
>>>>> necessarily tie any applications using dynamic linking to TianoCore or
>>>>> any firmware distribution that derives from it.
>>>>>
>>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>>> are slow to update their snapshots or have just given up doing that
>>>>> entirely. Distributing it for external applications like OS loaders
>>>>> would mean this can be leveraged probably no earlier than 10 years
>>>>> from now. And for in-firmware things, I have a hard time thinking about a
>>>> use-case that outweighs the drawbacks.
>>>>>
>>>>>
>>>>> To shift the topic slightly back to GSoC, however, I'm willing to work
>>>>> on other items on the task list. Unit testing and an ACPICA
>>>>> application are the alternative projects I had thought about. I need
>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I know
>>>>> a tiny bit about porting ACPICA as I also have plans to incorporate it into my
>>>> own project.
>>>>>
>>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>>> them as appropriate yet and I’m not here to drive you away from this
>>>>> specific task (or the others). However, I’m still curious and
>>>>> concerned. :)
>>>>>
>>>>> Best regards,
>>>>> Marvin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pedro Falcato
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
> Nate, Andrew, Marvin, Pedro, Ada, et al,
>
> This is a great discussion.  I've been debating where to weigh in...

Personally, I don't think this topic can get enough attention. Thanks! :)

>
> I agree that some sort of library sharing to reduce image size would 
> be very helpful.  Although some parts of the BIOS are built 
> separately, large parts of it are compiled at the same time, so there 
> should be ample opportunity for tool-directed sharing.  Some form of 
> pre-linking modules together may be the easiest way to do that.  But 
> however it gets implemented, we should try to make it automatic:  
> requiring library writers to manually add a lot of metadata or write 
> thunks or shims would be a barrier to adoption.

My suggestion to use a private function array would indeed require 
shims, however this would be the "better than nothing I guess" solution 
where only the most used and most painful to duplicate functions are 
shared, e.g., memory and maybe common string printing functions. This 
would quickly and efficiently achieve the goal of "reduce size". And 
some form of sophisticated prelinking would be my proposal for something 
more systematic, which might come a lot cheaper with the security 
features to be added.

>
> I also agree that PEI has acquired far too much functionality over 
> time, and we badly need to find some way to reverse that.  PEI is 
> pulling in more and more of the h/w initialization, leaving DXE mainly 
> to provide the boot interface (ACPI, UEFI services, setup screens, 
> etc.)  That puts more and more pressure on XIP storage and 
> cache-as-RAM.  How can we encourage a change?

I'm not in the loop enough to comment on the XIP and CAR issues. 
However, HW init moving to PEI (or more generally, into a single stage) 
is not necessarily a bad thing, is it? As I've said before, I'm not sure 
what else to think of PEI than "DXE light". Both "initialise some of the 
hardware" with no clear scope boundary but with similar interfaces. If 
they were sort of fused to provide a minimal "pre-memory PEI" and a more 
DXE-like "post-memory PEI" without having a clear boundary between HW 
init steps, would that be too bad? Now I'm *really* just blindly 
guessing because I barely looked into this project, but isn't that 
somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS. 
Nate earlier said he'd like a clear distinction between DXE and BDS, 
which I didn't really get the point of. But if PEI consumed the HW init 
portions of DXE, a more advanced BDS (which also powers e.g. the HID 
stuff on the way) would be the logical remainder stage to separately 
support UEFI booting as a payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven 
to its extreme, PEI and DXE cores would still share a bunch of code and 
it might make sense to share a foundation technically.

Best regards,
Marvin

>
> Thanks,
> Brian J. Johnson
>
>
> On 4/15/22 03:15, Nate DeSimone wrote:
>> Hi Andrew,
>>
>> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
>>>
>>>
>>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone 
>>>> <nathaniel.l.desimone@intel.com> wrote:
>>>>
>>>> Hi Marvin,
>>>>
>>>>> -----Original Message-----
>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin
>>>>> Häuser
>>>>> Sent: Thursday, April 14, 2022 12:56 AM
>>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>>>>> <nathaniel.l.desimone@intel.com>
>>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>>>>> <steven.shi@intel.com>
>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>
>>>>> I feel like there are so many much easier solutions to this 
>>>>> problem that are at
>>>>> most limited by the clear specification. The UEFI specification 
>>>>> with regards to
>>>>> booting and all of that obviously is of utmost importance.
>>>>
>>>> If you have a better idea that retains compatibility with the 
>>>> existing UEFI PI then I would be happy to hear it. Ultimately 
>>>> anything we do needs to be a pure extension that retains 
>>>> compatibility with old code. Given that restriction having the 
>>>> ability to coalesce all the LibraryClasses into a single module and 
>>>> have cross-module linking seems like the best way to handle it to me.
>>>>
>>>>> The UEFI PI specification parts that deal about internal 
>>>>> structure, as far as I know, are
>>>>> only in place to make it easy to integrate Intel IP.
>>>>
>>>> Its not just for Intel. The biggest reason for it to increase the 
>>>> standardization of the boot flow across the PC ecosystem. We have 
>>>> learned from experience that firmware is super critical to get a 
>>>> product out the door but it is also difficult to write. So we try 
>>>> to make it as reusable as humanly possible.
>>>>
>>>>> In fact, I don’t *know*, but I’m pretty sure the very strict 
>>>>> separation between PEI and DXE was
>>>>> preserved mostly because MRC was 32-bit-only for a long time. Glad 
>>>>> that
>>>>> seems to have been resolved, AMD does memory init by PSP nowadays.
>>>>
>>>> Having less complex early stages chain load more complex later 
>>>> stages is a common design pattern in firmware, not just UEFI. For 
>>>> example, your typical ARM system loads kinda like this:
>>>>
>>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor 
>>>> --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
>>>>
>>>> Comparing relative complexity I believe the Intel UEFI PI design is 
>>>> actually pretty simple when you consider how much it gets done:
>>>>
>>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>>>>
>>>> My biggest criticism of the design is that the strict separation 
>>>> between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
>>>>
>>>> There are a few reasons why PEI was 32-bit for quite some time. The 
>>>> biggest one is the code size increase, 64-bit x86 code is 20-30% 
>>>> larger than 32-bit x86 code. Since the only RAM Pre-Memory code has 
>>>> access to is the cache onboard the processor and for security 
>>>> reasons all that code has to fit within that RAM we generally do 
>>>> everything we can to make that image as small as possible. Second, 
>>>> 64-bit requires a page table and since we desired to keep PEI 
>>>> simple we tried to avoid that. Finally, the PI spec didn't allow a 
>>>> 64-bit PEI until recently. MRC is 32-bit code just because that is 
>>>> what PEI happens to be. Porting it to 64-bit is not terribly 
>>>> difficult.
>>>>
>>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any 
>>>> data structures that get shared between PEI and DXE (HOBs, PCDs, 
>>>> etc.) need to resolve to the same size and packing. LibraryClasses 
>>>> need to be written to compile properly in both modes. In the case 
>>>> of FSP API mode you need to resort to thunking between 32 and 
>>>> 64-bit modes during DXE. More or less we decided that the costs are 
>>>> starting to outweigh the advantages.
>>>>
>>>
>>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it 
>>> works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
>>>
>>> In the PC ecosystem a single chipset family can power thousands of 
>>> unique designs. So the  DRAM memory needs to be external, support 
>>> lots of different chipset packages(signal integrity...), support the 
>>> lowest cost through the highest cost DRAM and thousands of different 
>>> board layouts. So programing DRAM takes a masters degree in antenna 
>>> design. I’ve seen MRC (Memory Reference Code) with over a MiB of 
>>> DEBUG prints in it, and it literally is printing histograms of what 
>>> it is tuning. So all this code has to run before the system has any 
>>> DRAM, thus it is running using the cache as RAM. I’ve not looked at 
>>> the x86 architecture specs form the vendors in a while, but back in 
>>> the day they did not support page tables in ROM or pinned cached. 
>>> Now it might work, but if it breaks your CPU vendor blames you so 
>>> you don’t code PEI in X64….
>>>
>>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a 
>>> long time ago but I seem to remember the mask ROM used a table in 
>>> NOR FLASH to init memory and then copied an image from NOR FLASH 
>>> into DRAM and jumped to it. So PEI is kind of not really needed and 
>>> we implemented a PrePEI and jumped directly to DXE.
>>>
>>> Given I was around back in the day when all this stuff was designed 
>>> I can say SEC was always a place holder for security code, as 
>>> security code always has to run 1st. PEI (Pre EFI) was designed to 
>>> get DRAM programmed and then jump to DXE. It kind of also fell in 
>>> naturally to ACPI S3 flow since that was turning memory back on. 
>>> When we designed PEI we kind of though of it more like a boot loader 
>>> stage for the firmware that turned on memory and all the work would 
>>> happen in DXE. Then reality strikes and the existing BIOS assembly 
>>> programmers start learning C (lots of  cranky people) and they start 
>>> having to learn all about PEI to turn on memory. They had to write a 
>>> big chunk of code for the memory init in PEI. These folks had never 
>>> written any EFI code, so to them it was easier to move a lot of the 
>>> chipset init code into PEI as that is the world they had to figure 
>>> out to get memory turned on. I mean why learn EFI if you don’t have 
>>> to? So that is how we get so much code in IA32 (i386) on some 
>>> platforms. This start kind of biased future choices and how to 
>>> enable non edk2 code bases….
>>
>> One of the big reasons a lot of code that should have been written in 
>> DXE ended up in PEI is unfortunately due to the FSP and its inability 
>> to support DXE code.
>>
>>>
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>>>>
>>>>> For many good reasons, Linux does not provide a stable kernel API. 
>>>>> This
>>>>> allows to easily deploy breaking changes across the entire codebase.
>>>>> Obviously, this is infeasible at a large scale when you need to 
>>>>> integrate IP
>>>>> blobs, but it would already help to break the expectation that 
>>>>> UEFI PI is a
>>>>> perfectly forwards- and backwards-compatible system. DXE has 
>>>>> SetMem and
>>>>> CopyMem as part of gBS. Maybe I don’t like it that much as part of 
>>>>> the spec
>>>>> itself, but it’s a good idea technically. I’d probably opt to have 
>>>>> a quickly
>>>>> accessible implementation detail of important function pointers 
>>>>> appended to
>>>>> PEI and DXE services, used by in-tree modules. This may break both
>>>>> forwards- and backwards-compatibility, but as it only applies to 
>>>>> in-tree
>>>>> modules, that is fine so long as we let go of the compatibility 
>>>>> notions. PPIs
>>>>> and protocols are an option too of course, but they have a lookup
>>>>> performance penalty. Compared to dynamic linking, that should 
>>>>> hopefully be
>>>>> negligible however.
>>>>>
>>>>> Absolutely optional to read, I don’t intend to waste anyone’s time 
>>>>> much,
>>>>> some philosophical stuff about my rationale:
>>>>>
>>>>> If you started UEFI from scratch right now, would it have strictly 
>>>>> separated
>>>>> PEI and DXE?
>>>>
>>>> For sure a clean slate design started today would look a lot 
>>>> different than PEI/DXE... which was a clean slate design circa 1998 
>>>> 😊. In my opinion, if we were to suddenly go back to the drawing 
>>>> board we would build something that is much closer to a full OS now 
>>>> than we did back then. There have been cases where not being able 
>>>> to use interrupt handlers and not having thread scheduling has 
>>>> prevented implementation of desired features. The ARM guys built 
>>>> LittleKernel (https://github.com/littlekernel/lk) for a lot of 
>>>> these reasons. In the data center world some have decided to go to 
>>>> the extreme of putting an entire copy of Linux in SPI so they can 
>>>> do a network boot that downloads the OS image using BitTorrent!
>>>>
>>>>> The duplication between PEI and DXE core, and by extension
>>>>> MM core, would be my most obvious place to start reducing size. I 
>>>>> would
>>>>> probably opt for a PEI/DXE hybrid where it starts in „minimal 
>>>>> mode“ (maybe
>>>>> think of PEI more like a microkernel here) and after memory is 
>>>>> initialised, the
>>>>> rest of DXE is loaded. And MM core would get no loading at all, this
>>>>> requirement has gladly been dropped ages ago. Just one prelinked 
>>>>> snapshot
>>>>> of the address space with a relocation table and a safe 
>>>>> self-relocator on entry
>>>>> (this is needed at the very least for ARM).
>>>>>
>>>>> Ironically, with my idea for MM, dynamic loading would be free as 
>>>>> everything
>>>>> is prelinked anyway. The same is true for PEI XIP, it is prelinked 
>>>>> by nature.
>>>>
>>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that 
>>>> does get used for the PEI GOP driver.
>>>>
>>>>> What I do not like is the additional dynamic linking code at 
>>>>> load-time for non-
>>>>> XIP modules. Though, the more I think about it, the more I wonder 
>>>>> whether
>>>>> not the entirety of UEFI could be composed of prelinked, relocatable
>>>>> memory snapshots without traditional image loading entirely (for 
>>>>> in-FW
>>>>> stuff). macOS has a similar concept with its “Kernel Collections”. 
>>>>> Well, way
>>>>> too much off-topic now. :)
>>>>
>>>> If you make the assumption that 100% of the code is compiled all at 
>>>> once then yes that works. UEFI was designed so that assumption does 
>>>> not need to be true. There are good use cases for it: OpROMs, 
>>>> generic OS loaders, network boot, etc.
>>>>
>>>> Thanks,
>>>> Nate
>>>>
>>>>>
>>>>> Why am I explaining all this despite the fact everyone knows this 
>>>>> will never
>>>>> happen? Because I don’t like the notion of fixing issues of an 
>>>>> already
>>>>> overcomplicated system by adding even more complicated stuff. 
>>>>> Especially
>>>>> when the existing overcomplicated stuff is already uncomfortably 
>>>>> broken.
>>>>>
>>>>> Best regards,
>>>>> Marvin
>>>>>
>>>>>> For XIP PEI code… it will really help and would be very timely 
>>>>>> since the
>>>>> transition of PEI from 32-bit to 64-bit is going to increase the 
>>>>> size of PEI by
>>>>> ~20%.
>>>>>>
>>>>>> Thanks,
>>>>>> Nate
>>>>>>
>>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>>>>> <mhaeuser@posteo.de>
>>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>>>>> <steven.shi@intel.com>
>>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>>
>>>>>> Hi Marvin, Ada,
>>>>>>
>>>>>> Some comments:
>>>>>>
>>>>>> I don't think the purpose of the dynamic linker is to treat EFI as a
>>>>>> complete operating system, but to try to eliminate the static 
>>>>>> linking
>>>>>> that may be needlessly duplicating code that could instead be put 
>>>>>> in a
>>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are 
>>>>> linked
>>>>> into a *lot* of .efi, instead of being just a library. It'd be 
>>>>> nice to see some
>>>>> numbers on this (something like Google's bloaty could be run on 
>>>>> every .efi
>>>>> file, in order to understand how much file space we would actually 
>>>>> save).
>>>>>>
>>>>>> Other comments inline.
>>>>>>
>>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>
>>>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>>>>> i was replying via the groups.io<http://groups.io > web 
>>>>>> interface, I'm
>>>>>> guessing that messed up the thread? i haven't used mailing lists
>>>>>> before and don't know how they work. I'll use my mail client from 
>>>>>> here on.
>>>>>>
>>>>>> I'm on board with not treating EFI as an operating system. the 
>>>>>> more i
>>>>>> think about it the more it looks like scope creep.
>>>>>>
>>>>>> Agreed.
>>>>>>
>>>>>>
>>>>>> I'm not quite as enthusiastic
>>>>>> about it as i was at first glance.
>>>>>>
>>>>>> I'm still keen on doing my gsoc proposal for edk, though, and 
>>>>>> even if
>>>>>> this task and the acpica application are decided to be out of scope
>>>>>> unit testing,
>>>>>>
>>>>>> How about fuzz-testing? This is also something edk2 needs quite 
>>>>>> badly. At
>>>>> Acidanthera, we compile edk2 code in userspace outside the edk2 build
>>>>> system and fuzz with dummy applications.
>>>>>>
>>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>>>> https://llvm.org/docs/LibFuzzer.html ) and is something I could 
>>>>> happily
>>>>> mentor.
>>>>>> clang integration
>>>>>>
>>>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>>>> There are working UBSan concepts, but they also need to be 
>>>>>> mainlined.
>>>>>>
>>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be 
>>>>>> me and
>>>>> some other, more senior, mentor (possibly Steven Shi, which I 
>>>>> included in
>>>>> the task).
>>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly
>>>>>> some other sanitizer it's quite possible that it could be 
>>>>>> considered a large
>>>>> project (which means more hours but a larger stipend too). Fuzzing +
>>>>> coverage could be very nice additions to this project idea.
>>>>>> Also, is stress-testing a decent idea?
>>>>>>
>>>>>>
>>>>>> and source-level debugging are all relevant to my interests.
>>>>>>
>>>>>> how about your ideas for security stuff?
>>>>>>
>>>>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
>>>>> only work on MMRAM. Also evaluation of how to best avoid pointers 
>>>>> in MM
>>>>> communication buffers would be nice.
>>>>>>
>>>>>> There also is a bunch of other stuff, like working out moving a 
>>>>>> part of
>>>>> CpuDxe into DxeCore to have memory protection live immediately, 
>>>>> memory
>>>>> protection in PEI, a replacement for the TE format (it’s buggy and 
>>>>> most
>>>>> platforms mostly abandoned it over various issues), and 
>>>>> alternatives to
>>>>> guarding critical code with SMM (like allowing NVRAM commits only 
>>>>> as part
>>>>> of a reboot).
>>>>>>
>>>>>> I personally find all of those projects very important, but I cannot
>>>>>> promise many people agree. Especially those that impose global 
>>>>>> changes
>>>>>> (most notably the TE replacement) may be very tedious to submit.
>>>>>> Gladly, I believe you can submit multiple proposals (?)
>>>>>>
>>>>>> Best regards,
>>>>>> Marvin
>>>>>>
>>>>>>
>>>>>> I'm not very knowledgeable about
>>>>>> trusted platform or secure boot but I'm willing to learn whatever is
>>>>>> necessary to get something spun up for my proposal.
>>>>>>
>>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>
>>>>>>
>>>>>> Do you use the “reply all” option in your mail client? Looks like my
>>>>>> CCs have been dropped again. Comments inline.
>>>>>>
>>>>>> On 13. Apr 2022, at 12:54, Ada Christine
>>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>>>> wrote:
>>>>>> Hi, Marvin
>>>>>>
>>>>>> Its similarity to my own latest experiment is the key to what 
>>>>>> grabbed
>>>>>> my attention. I have no particular use case in mind for it, but I 
>>>>>> see
>>>>>> its potential for anybody developing larger applications in that 
>>>>>> when
>>>>>> a library is changed there's no need to distribute a new version of
>>>>>> the whole binary, just the relevant library module.
>>>>>>
>>>>>> I really do not like the trend of treating UEFI as a full-fledged 
>>>>>> OS -
>>>>>> it is not. The most used UEFI applications, OS loaders, are 
>>>>>> really not
>>>>>> that huge and are distributed as part of the OS image anyway. 
>>>>>> Even for
>>>>>> less used applications, you will always get a full snapshot anyhow.
>>>>>> Gladly we don’t have auto-update and package management yet. :)
>>>>>>
>>>>>>
>>>>>> I slept on it and it occurred to me that the whole thing could 
>>>>>> operate
>>>>>> similarly to the shell protocol in that the linker/loader is 
>>>>>> itself an
>>>>>> application that does a LoadImage() on the application needing 
>>>>>> dynamic
>>>>>> linking facilities.
>>>>>>
>>>>>> That would mean the linker itself is shipped with every application
>>>>>> that requires it? Otherwise it doesn’t make much sense for it to 
>>>>>> be an
>>>>>> app and below’s problems apply.
>>>>>>
>>>>>> If however the whole plan is making the linker as a DXE and 
>>>>>> including
>>>>>> it with the firmware, that I'm not quite as sure about. That would
>>>>>> necessarily tie any applications using dynamic linking to 
>>>>>> TianoCore or
>>>>>> any firmware distribution that derives from it.
>>>>>>
>>>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>>>> are slow to update their snapshots or have just given up doing that
>>>>>> entirely. Distributing it for external applications like OS loaders
>>>>>> would mean this can be leveraged probably no earlier than 10 years
>>>>>> from now. And for in-firmware things, I have a hard time thinking 
>>>>>> about a
>>>>> use-case that outweighs the drawbacks.
>>>>>>
>>>>>>
>>>>>> To shift the topic slightly back to GSoC, however, I'm willing to 
>>>>>> work
>>>>>> on other items on the task list. Unit testing and an ACPICA
>>>>>> application are the alternative projects I had thought about. I need
>>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I 
>>>>>> know
>>>>>> a tiny bit about porting ACPICA as I also have plans to 
>>>>>> incorporate it into my
>>>>> own project.
>>>>>>
>>>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>>>> them as appropriate yet and I’m not here to drive you away from this
>>>>>> specific task (or the others). However, I’m still curious and
>>>>>> concerned. :)
>>>>>>
>>>>>> Best regards,
>>>>>> Marvin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Pedro Falcato
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>
>
> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Oram, Isaac W]

In practice, it feels like we have evolved far beyond original PEI scope for natural and good reasons.  Initially, it was very resource constrained.  8KB CAR for data only.  That is no longer a real constraint for most HW.  And thus we get things like MP services early, writable variables early, complex PCD databases, and such.

I don't think it is bad.  The PEI environment is much friendlier for developing and debugging ubiquitous silicon initialization than DXE.  Ubiquitous meaning you pretty much need to do it on any boot path to any boot device.  Pre Memory PEI and Post Memory PEI seem to be the right design for many things.  It generally simplifies handshaking between early and later silicon code.

Thus I like:
SEC: 			Basics
PEI Pre-Mem:  		Get memory
PEI Post-Mem:  		Complete all "always required" silicon init.  Can also do silicon init for more optional platform features.  Should not do silicon init for dynamically optional features.
Standalone MM:  	Kicked off from post memory PEI.
DXE:			Setup driver and UEFI environments
BDS:			Initialize boot target specific required HW, provide any required user interface, boot OS

I think that this kind of design division makes a solution very flexible for supporting a wide range of designs.  It allows more generic/reusable DXE and BDS.  It makes SMM a more optional solution.  We have launched Linux as an embedded OS or target OS from DXE IPL, DXE Core, BDS, etc.  It supports that kind of flexibility.  It enables LinuxBoot @ BDS and one day in place of DXE (Universal Payload).
And fundamentally, it can reduce complexity because we can reduce the handshaking across phases.  That tends to introduce complexity that the above helps reduce. 

We tried to incorporate a lot of this in the Minimum Platform Architecture spec and it has been working fairly well.  It is hard to migrate the large mass of existing code, so we don't have a perfect example of this in action.  But the vision is that you can design such that the phases are well bounded and very testable.  And this will help silicon development, board porting, and adding features.  If we succeed, we probably come back to ask for changes to make the pre-memory to post-memory handoff better.  I find few people really know that well and we have repeated performance problems.  But generally, there is plenty of flexibility in the existing architecture and phases to tune this for many answers.

Specific to the discussion of DLL like behavior.  I agree it should be handled by tools and not a unique and complex answer.  It is actually a trivial change to take existing PEIM (for example) and just add the entry point as a library class constructor and then you can link a bunch of PEIM into a single image.  It would be pretty minor to make this easy and supported I would think.  You lose dynamic dispatch, obviously.  But it is a short hack to eliminate early library duplication.

Regards,
Isaac

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
Sent: Friday, April 15, 2022 9:44 AM
To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@hpe.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Andrew Fish <afish@apple.com>; devel@edk2.groups.io
Cc: Pedro Falcato <pedro.falcato@gmail.com>; adachristine18@gmail.com; Shi, Steven <steven.shi@intel.com>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
> Nate, Andrew, Marvin, Pedro, Ada, et al,
>
> This is a great discussion.  I've been debating where to weigh in...

Personally, I don't think this topic can get enough attention. Thanks! :)

>
> I agree that some sort of library sharing to reduce image size would 
> be very helpful.  Although some parts of the BIOS are built 
> separately, large parts of it are compiled at the same time, so there 
> should be ample opportunity for tool-directed sharing.  Some form of 
> pre-linking modules together may be the easiest way to do that.  But 
> however it gets implemented, we should try to make it automatic:
> requiring library writers to manually add a lot of metadata or write 
> thunks or shims would be a barrier to adoption.

My suggestion to use a private function array would indeed require shims, however this would be the "better than nothing I guess" solution where only the most used and most painful to duplicate functions are shared, e.g., memory and maybe common string printing functions. This would quickly and efficiently achieve the goal of "reduce size". And some form of sophisticated prelinking would be my proposal for something more systematic, which might come a lot cheaper with the security features to be added.

>
> I also agree that PEI has acquired far too much functionality over 
> time, and we badly need to find some way to reverse that.  PEI is 
> pulling in more and more of the h/w initialization, leaving DXE mainly 
> to provide the boot interface (ACPI, UEFI services, setup screens,
> etc.)  That puts more and more pressure on XIP storage and 
> cache-as-RAM.  How can we encourage a change?

I'm not in the loop enough to comment on the XIP and CAR issues. 
However, HW init moving to PEI (or more generally, into a single stage) is not necessarily a bad thing, is it? As I've said before, I'm not sure what else to think of PEI than "DXE light". Both "initialise some of the hardware" with no clear scope boundary but with similar interfaces. If they were sort of fused to provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI" without having a clear boundary between HW init steps, would that be too bad? Now I'm *really* just blindly guessing because I barely looked into this project, but isn't that somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS. 
Nate earlier said he'd like a clear distinction between DXE and BDS, which I didn't really get the point of. But if PEI consumed the HW init portions of DXE, a more advanced BDS (which also powers e.g. the HID stuff on the way) would be the logical remainder stage to separately support UEFI booting as a payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven to its extreme, PEI and DXE cores would still share a bunch of code and it might make sense to share a foundation technically.

Best regards,
Marvin

>
> Thanks,
> Brian J. Johnson
>
>
> On 4/15/22 03:15, Nate DeSimone wrote:
>> Hi Andrew,
>>
>> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
>>>
>>>
>>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone 
>>>> <nathaniel.l.desimone@intel.com> wrote:
>>>>
>>>> Hi Marvin,
>>>>
>>>>> -----Original Message-----
>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of 
>>>>> Marvin Häuser
>>>>> Sent: Thursday, April 14, 2022 12:56 AM
>>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L 
>>>>> <nathaniel.l.desimone@intel.com>
>>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io 
>>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven 
>>>>> <steven.shi@intel.com>
>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>
>>>>> I feel like there are so many much easier solutions to this 
>>>>> problem that are at most limited by the clear specification. The 
>>>>> UEFI specification with regards to booting and all of that 
>>>>> obviously is of utmost importance.
>>>>
>>>> If you have a better idea that retains compatibility with the 
>>>> existing UEFI PI then I would be happy to hear it. Ultimately 
>>>> anything we do needs to be a pure extension that retains 
>>>> compatibility with old code. Given that restriction having the 
>>>> ability to coalesce all the LibraryClasses into a single module and 
>>>> have cross-module linking seems like the best way to handle it to me.
>>>>
>>>>> The UEFI PI specification parts that deal about internal 
>>>>> structure, as far as I know, are only in place to make it easy to 
>>>>> integrate Intel IP.
>>>>
>>>> Its not just for Intel. The biggest reason for it to increase the 
>>>> standardization of the boot flow across the PC ecosystem. We have 
>>>> learned from experience that firmware is super critical to get a 
>>>> product out the door but it is also difficult to write. So we try 
>>>> to make it as reusable as humanly possible.
>>>>
>>>>> In fact, I don’t *know*, but I’m pretty sure the very strict 
>>>>> separation between PEI and DXE was preserved mostly because MRC 
>>>>> was 32-bit-only for a long time. Glad that seems to have been 
>>>>> resolved, AMD does memory init by PSP nowadays.
>>>>
>>>> Having less complex early stages chain load more complex later 
>>>> stages is a common design pattern in firmware, not just UEFI. For 
>>>> example, your typical ARM system loads kinda like this:
>>>>
>>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
>>>> --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
>>>>
>>>> Comparing relative complexity I believe the Intel UEFI PI design is 
>>>> actually pretty simple when you consider how much it gets done:
>>>>
>>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>>>>
>>>> My biggest criticism of the design is that the strict separation 
>>>> between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
>>>>
>>>> There are a few reasons why PEI was 32-bit for quite some time. The 
>>>> biggest one is the code size increase, 64-bit x86 code is 20-30% 
>>>> larger than 32-bit x86 code. Since the only RAM Pre-Memory code has 
>>>> access to is the cache onboard the processor and for security 
>>>> reasons all that code has to fit within that RAM we generally do 
>>>> everything we can to make that image as small as possible. Second, 
>>>> 64-bit requires a page table and since we desired to keep PEI 
>>>> simple we tried to avoid that. Finally, the PI spec didn't allow a 
>>>> 64-bit PEI until recently. MRC is 32-bit code just because that is 
>>>> what PEI happens to be. Porting it to 64-bit is not terribly 
>>>> difficult.
>>>>
>>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any 
>>>> data structures that get shared between PEI and DXE (HOBs, PCDs,
>>>> etc.) need to resolve to the same size and packing. LibraryClasses 
>>>> need to be written to compile properly in both modes. In the case 
>>>> of FSP API mode you need to resort to thunking between 32 and 
>>>> 64-bit modes during DXE. More or less we decided that the costs are 
>>>> starting to outweigh the advantages.
>>>>
>>>
>>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it 
>>> works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
>>>
>>> In the PC ecosystem a single chipset family can power thousands of 
>>> unique designs. So the  DRAM memory needs to be external, support 
>>> lots of different chipset packages(signal integrity...), support the 
>>> lowest cost through the highest cost DRAM and thousands of different 
>>> board layouts. So programing DRAM takes a masters degree in antenna 
>>> design. I’ve seen MRC (Memory Reference Code) with over a MiB of 
>>> DEBUG prints in it, and it literally is printing histograms of what 
>>> it is tuning. So all this code has to run before the system has any 
>>> DRAM, thus it is running using the cache as RAM. I’ve not looked at 
>>> the x86 architecture specs form the vendors in a while, but back in 
>>> the day they did not support page tables in ROM or pinned cached.
>>> Now it might work, but if it breaks your CPU vendor blames you so 
>>> you don’t code PEI in X64….
>>>
>>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a 
>>> long time ago but I seem to remember the mask ROM used a table in 
>>> NOR FLASH to init memory and then copied an image from NOR FLASH 
>>> into DRAM and jumped to it. So PEI is kind of not really needed and 
>>> we implemented a PrePEI and jumped directly to DXE.
>>>
>>> Given I was around back in the day when all this stuff was designed 
>>> I can say SEC was always a place holder for security code, as 
>>> security code always has to run 1st. PEI (Pre EFI) was designed to 
>>> get DRAM programmed and then jump to DXE. It kind of also fell in 
>>> naturally to ACPI S3 flow since that was turning memory back on.
>>> When we designed PEI we kind of though of it more like a boot loader 
>>> stage for the firmware that turned on memory and all the work would 
>>> happen in DXE. Then reality strikes and the existing BIOS assembly 
>>> programmers start learning C (lots of  cranky people) and they start 
>>> having to learn all about PEI to turn on memory. They had to write a 
>>> big chunk of code for the memory init in PEI. These folks had never 
>>> written any EFI code, so to them it was easier to move a lot of the 
>>> chipset init code into PEI as that is the world they had to figure 
>>> out to get memory turned on. I mean why learn EFI if you don’t have 
>>> to? So that is how we get so much code in IA32 (i386) on some 
>>> platforms. This start kind of biased future choices and how to 
>>> enable non edk2 code bases….
>>
>> One of the big reasons a lot of code that should have been written in 
>> DXE ended up in PEI is unfortunately due to the FSP and its inability 
>> to support DXE code.
>>
>>>
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>>>>
>>>>> For many good reasons, Linux does not provide a stable kernel API. 
>>>>> This
>>>>> allows to easily deploy breaking changes across the entire codebase.
>>>>> Obviously, this is infeasible at a large scale when you need to 
>>>>> integrate IP blobs, but it would already help to break the 
>>>>> expectation that UEFI PI is a perfectly forwards- and 
>>>>> backwards-compatible system. DXE has SetMem and CopyMem as part of 
>>>>> gBS. Maybe I don’t like it that much as part of the spec itself, 
>>>>> but it’s a good idea technically. I’d probably opt to have a 
>>>>> quickly accessible implementation detail of important function 
>>>>> pointers appended to PEI and DXE services, used by in-tree 
>>>>> modules. This may break both
>>>>> forwards- and backwards-compatibility, but as it only applies to 
>>>>> in-tree modules, that is fine so long as we let go of the 
>>>>> compatibility notions. PPIs and protocols are an option too of 
>>>>> course, but they have a lookup performance penalty. Compared to 
>>>>> dynamic linking, that should hopefully be negligible however.
>>>>>
>>>>> Absolutely optional to read, I don’t intend to waste anyone’s time 
>>>>> much, some philosophical stuff about my rationale:
>>>>>
>>>>> If you started UEFI from scratch right now, would it have strictly 
>>>>> separated PEI and DXE?
>>>>
>>>> For sure a clean slate design started today would look a lot 
>>>> different than PEI/DXE... which was a clean slate design circa 1998
>>>> 😊. In my opinion, if we were to suddenly go back to the drawing 
>>>> board we would build something that is much closer to a full OS now 
>>>> than we did back then. There have been cases where not being able 
>>>> to use interrupt handlers and not having thread scheduling has 
>>>> prevented implementation of desired features. The ARM guys built 
>>>> LittleKernel (https://github.com/littlekernel/lk) for a lot of 
>>>> these reasons. In the data center world some have decided to go to 
>>>> the extreme of putting an entire copy of Linux in SPI so they can 
>>>> do a network boot that downloads the OS image using BitTorrent!
>>>>
>>>>> The duplication between PEI and DXE core, and by extension MM 
>>>>> core, would be my most obvious place to start reducing size. I 
>>>>> would probably opt for a PEI/DXE hybrid where it starts in 
>>>>> „minimal mode“ (maybe think of PEI more like a microkernel here) 
>>>>> and after memory is initialised, the rest of DXE is loaded. And MM 
>>>>> core would get no loading at all, this requirement has gladly been 
>>>>> dropped ages ago. Just one prelinked snapshot of the address space 
>>>>> with a relocation table and a safe self-relocator on entry (this 
>>>>> is needed at the very least for ARM).
>>>>>
>>>>> Ironically, with my idea for MM, dynamic loading would be free as 
>>>>> everything is prelinked anyway. The same is true for PEI XIP, it 
>>>>> is prelinked by nature.
>>>>
>>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that 
>>>> does get used for the PEI GOP driver.
>>>>
>>>>> What I do not like is the additional dynamic linking code at 
>>>>> load-time for non- XIP modules. Though, the more I think about it, 
>>>>> the more I wonder whether not the entirety of UEFI could be 
>>>>> composed of prelinked, relocatable memory snapshots without 
>>>>> traditional image loading entirely (for in-FW stuff). macOS has a 
>>>>> similar concept with its “Kernel Collections”.
>>>>> Well, way
>>>>> too much off-topic now. :)
>>>>
>>>> If you make the assumption that 100% of the code is compiled all at 
>>>> once then yes that works. UEFI was designed so that assumption does 
>>>> not need to be true. There are good use cases for it: OpROMs, 
>>>> generic OS loaders, network boot, etc.
>>>>
>>>> Thanks,
>>>> Nate
>>>>
>>>>>
>>>>> Why am I explaining all this despite the fact everyone knows this 
>>>>> will never happen? Because I don’t like the notion of fixing 
>>>>> issues of an already overcomplicated system by adding even more 
>>>>> complicated stuff.
>>>>> Especially
>>>>> when the existing overcomplicated stuff is already uncomfortably 
>>>>> broken.
>>>>>
>>>>> Best regards,
>>>>> Marvin
>>>>>
>>>>>> For XIP PEI code… it will really help and would be very timely 
>>>>>> since the
>>>>> transition of PEI from 32-bit to 64-bit is going to increase the 
>>>>> size of PEI by ~20%.
>>>>>>
>>>>>> Thanks,
>>>>>> Nate
>>>>>>
>>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser 
>>>>>> <mhaeuser@posteo.de>
>>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone, 
>>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven 
>>>>>> <steven.shi@intel.com>
>>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>>
>>>>>> Hi Marvin, Ada,
>>>>>>
>>>>>> Some comments:
>>>>>>
>>>>>> I don't think the purpose of the dynamic linker is to treat EFI 
>>>>>> as a complete operating system, but to try to eliminate the 
>>>>>> static linking that may be needlessly duplicating code that could 
>>>>>> instead be put in a
>>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are 
>>>>> linked into a *lot* of .efi, instead of being just a library. It'd 
>>>>> be nice to see some numbers on this (something like Google's 
>>>>> bloaty could be run on every .efi file, in order to understand how 
>>>>> much file space we would actually save).
>>>>>>
>>>>>> Other comments inline.
>>>>>>
>>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>
>>>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>>>>> i was replying via the groups.io<http://groups.io > web 
>>>>>> interface, I'm guessing that messed up the thread? i haven't used 
>>>>>> mailing lists before and don't know how they work. I'll use my 
>>>>>> mail client from here on.
>>>>>>
>>>>>> I'm on board with not treating EFI as an operating system. the 
>>>>>> more i think about it the more it looks like scope creep.
>>>>>>
>>>>>> Agreed.
>>>>>>
>>>>>>
>>>>>> I'm not quite as enthusiastic
>>>>>> about it as i was at first glance.
>>>>>>
>>>>>> I'm still keen on doing my gsoc proposal for edk, though, and 
>>>>>> even if this task and the acpica application are decided to be 
>>>>>> out of scope unit testing,
>>>>>>
>>>>>> How about fuzz-testing? This is also something edk2 needs quite 
>>>>>> badly. At
>>>>> Acidanthera, we compile edk2 code in userspace outside the edk2 
>>>>> build system and fuzz with dummy applications.
>>>>>>
>>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>>>> https://llvm.org/docs/LibFuzzer.html ) and is something I could 
>>>>> happily mentor.
>>>>>> clang integration
>>>>>>
>>>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>>>> There are working UBSan concepts, but they also need to be 
>>>>>> mainlined.
>>>>>>
>>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be 
>>>>>> me and
>>>>> some other, more senior, mentor (possibly Steven Shi, which I 
>>>>> included in the task).
>>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and 
>>>>>> possibly some other sanitizer it's quite possible that it could 
>>>>>> be considered a large
>>>>> project (which means more hours but a larger stipend too). Fuzzing 
>>>>> + coverage could be very nice additions to this project idea.
>>>>>> Also, is stress-testing a decent idea?
>>>>>>
>>>>>>
>>>>>> and source-level debugging are all relevant to my interests.
>>>>>>
>>>>>> how about your ideas for security stuff?
>>>>>>
>>>>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would 
>>>>> only work on MMRAM. Also evaluation of how to best avoid pointers 
>>>>> in MM communication buffers would be nice.
>>>>>>
>>>>>> There also is a bunch of other stuff, like working out moving a 
>>>>>> part of
>>>>> CpuDxe into DxeCore to have memory protection live immediately, 
>>>>> memory protection in PEI, a replacement for the TE format (it’s 
>>>>> buggy and most platforms mostly abandoned it over various issues), 
>>>>> and alternatives to guarding critical code with SMM (like allowing 
>>>>> NVRAM commits only as part of a reboot).
>>>>>>
>>>>>> I personally find all of those projects very important, but I 
>>>>>> cannot promise many people agree. Especially those that impose 
>>>>>> global changes (most notably the TE replacement) may be very 
>>>>>> tedious to submit.
>>>>>> Gladly, I believe you can submit multiple proposals (?)
>>>>>>
>>>>>> Best regards,
>>>>>> Marvin
>>>>>>
>>>>>>
>>>>>> I'm not very knowledgeable about
>>>>>> trusted platform or secure boot but I'm willing to learn whatever 
>>>>>> is necessary to get something spun up for my proposal.
>>>>>>
>>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>
>>>>>>
>>>>>> Do you use the “reply all” option in your mail client? Looks like 
>>>>>> my CCs have been dropped again. Comments inline.
>>>>>>
>>>>>> On 13. Apr 2022, at 12:54, Ada Christine 
>>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>>>> wrote:
>>>>>> Hi, Marvin
>>>>>>
>>>>>> Its similarity to my own latest experiment is the key to what 
>>>>>> grabbed my attention. I have no particular use case in mind for 
>>>>>> it, but I see its potential for anybody developing larger 
>>>>>> applications in that when a library is changed there's no need to 
>>>>>> distribute a new version of the whole binary, just the relevant 
>>>>>> library module.
>>>>>>
>>>>>> I really do not like the trend of treating UEFI as a full-fledged 
>>>>>> OS - it is not. The most used UEFI applications, OS loaders, are 
>>>>>> really not that huge and are distributed as part of the OS image 
>>>>>> anyway.
>>>>>> Even for
>>>>>> less used applications, you will always get a full snapshot anyhow.
>>>>>> Gladly we don’t have auto-update and package management yet. :)
>>>>>>
>>>>>>
>>>>>> I slept on it and it occurred to me that the whole thing could 
>>>>>> operate
>>>>>> similarly to the shell protocol in that the linker/loader is 
>>>>>> itself an
>>>>>> application that does a LoadImage() on the application needing 
>>>>>> dynamic
>>>>>> linking facilities.
>>>>>>
>>>>>> That would mean the linker itself is shipped with every application
>>>>>> that requires it? Otherwise it doesn’t make much sense for it to 
>>>>>> be an
>>>>>> app and below’s problems apply.
>>>>>>
>>>>>> If however the whole plan is making the linker as a DXE and 
>>>>>> including
>>>>>> it with the firmware, that I'm not quite as sure about. That would
>>>>>> necessarily tie any applications using dynamic linking to 
>>>>>> TianoCore or
>>>>>> any firmware distribution that derives from it.
>>>>>>
>>>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>>>> are slow to update their snapshots or have just given up doing that
>>>>>> entirely. Distributing it for external applications like OS loaders
>>>>>> would mean this can be leveraged probably no earlier than 10 years
>>>>>> from now. And for in-firmware things, I have a hard time thinking 
>>>>>> about a
>>>>> use-case that outweighs the drawbacks.
>>>>>>
>>>>>>
>>>>>> To shift the topic slightly back to GSoC, however, I'm willing to 
>>>>>> work
>>>>>> on other items on the task list. Unit testing and an ACPICA
>>>>>> application are the alternative projects I had thought about. I need
>>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I 
>>>>>> know
>>>>>> a tiny bit about porting ACPICA as I also have plans to 
>>>>>> incorporate it into my
>>>>> own project.
>>>>>>
>>>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>>>> them as appropriate yet and I’m not here to drive you away from this
>>>>>> specific task (or the others). However, I’m still curious and
>>>>>> concerned. :)
>>>>>>
>>>>>> Best regards,
>>>>>> Marvin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Pedro Falcato
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>
>
> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Marvin Häuser]

Hey Isaac,

Thanks for your input! :)

On 15.04.22 20:47, Oram, Isaac W wrote:
> In practice, it feels like we have evolved far beyond original PEI scope for natural and good reasons.  Initially, it was very resource constrained.  8KB CAR for data only.  That is no longer a real constraint for most HW.  And thus we get things like MP services early, writable variables early, complex PCD databases, and such.

Looking at the amount of things that are duplicated from DXE to PEI (MP, 
variable services, PCD, GOP, ...), this sounds like all the more reason 
to me to have some unified technological foundation for PEI and DXE 
core. With mixed-architecture execution decaying (at least for the time 
being), what'd be the point of duplicating this code in both stages? Of 
course this hypothetical shared foundation should be flexible enough to 
be able to build distinct PEI and DXE cores to allow for UEFI PI and 
UEFI to be implemented separately from each other.

>
> I don't think it is bad.  The PEI environment is much friendlier for developing and debugging ubiquitous silicon initialization than DXE.  Ubiquitous meaning you pretty much need to do it on any boot path to any boot device.  Pre Memory PEI and Post Memory PEI seem to be the right design for many things.  It generally simplifies handshaking between early and later silicon code.
>
> Thus I like:
> SEC: 			Basics
> PEI Pre-Mem:  		Get memory
> PEI Post-Mem:  		Complete all "always required" silicon init.  Can also do silicon init for more optional platform features.  Should not do silicon init for dynamically optional features.
> Standalone MM:  	Kicked off from post memory PEI.
> DXE:			Setup driver and UEFI environments
> BDS:			Initialize boot target specific required HW, provide any required user interface, boot OS

Yes! :)

>
> I think that this kind of design division makes a solution very flexible for supporting a wide range of designs.  It allows more generic/reusable DXE and BDS.  It makes SMM a more optional solution.  We have launched Linux as an embedded OS or target OS from DXE IPL, DXE Core, BDS, etc.  It supports that kind of flexibility.  It enables LinuxBoot @ BDS and one day in place of DXE (Universal Payload).

In that hypothetical model, yeah, I think DXE and BDS are *mostly* the 
same thing. BDS of course is a distinctly defined sub-process of 
determining the boot option, but the whole DXE+BDS stage combination 
would just be the generic "user experience" portion of UEFI. And I 
totally see things being interchangeable both ways, be that e.g. 
Slimbootloader + UEFI DXE/BDS or UEFI PEI + LinuxBoot.

> And fundamentally, it can reduce complexity because we can reduce the handshaking across phases.  That tends to introduce complexity that the above helps reduce.
>
> We tried to incorporate a lot of this in the Minimum Platform Architecture spec and it has been working fairly well.  It is hard to migrate the large mass of existing code, so we don't have a perfect example of this in action.  But the vision is that you can design such that the phases are well bounded and very testable.  And this will help silicon development, board porting, and adding features.  If we succeed, we probably come back to ask for changes to make the pre-memory to post-memory handoff better.  I find few people really know that well and we have repeated performance problems.  But generally, there is plenty of flexibility in the existing architecture and phases to tune this for many answers.
>
> Specific to the discussion of DLL like behavior.  I agree it should be handled by tools and not a unique and complex answer.  It is actually a trivial change to take existing PEIM (for example) and just add the entry point as a library class constructor and then you can link a bunch of PEIM into a single image.  It would be pretty minor to make this easy and supported I would think.  You lose dynamic dispatch, obviously.  But it is a short hack to eliminate early library duplication.

This would actually be a great thing to have upstream (as in, something 
more straight-forward that'd allow modules to be merged without 
modifications, especially taking Depex into account somehow)! Amaranth 
will purge any sort of linking (at the very least downstream) and will 
mostly be just one monolith. I do not see that as an alternative to the 
previously discussed implementations of and alternatives to dynamic 
linking however, but as something complementary. Has there been any 
prior work on this?

Best regards,
Marvin

>
> Regards,
> Isaac
>
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
> Sent: Friday, April 15, 2022 9:44 AM
> To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@hpe.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Andrew Fish <afish@apple.com>; devel@edk2.groups.io
> Cc: Pedro Falcato <pedro.falcato@gmail.com>; adachristine18@gmail.com; Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>
> Hey Brian,
>
> On 15.04.22 18:22, Brian J. Johnson wrote:
>> Nate, Andrew, Marvin, Pedro, Ada, et al,
>>
>> This is a great discussion.  I've been debating where to weigh in...
> Personally, I don't think this topic can get enough attention. Thanks! :)
>
>> I agree that some sort of library sharing to reduce image size would
>> be very helpful.  Although some parts of the BIOS are built
>> separately, large parts of it are compiled at the same time, so there
>> should be ample opportunity for tool-directed sharing.  Some form of
>> pre-linking modules together may be the easiest way to do that.  But
>> however it gets implemented, we should try to make it automatic:
>> requiring library writers to manually add a lot of metadata or write
>> thunks or shims would be a barrier to adoption.
> My suggestion to use a private function array would indeed require shims, however this would be the "better than nothing I guess" solution where only the most used and most painful to duplicate functions are shared, e.g., memory and maybe common string printing functions. This would quickly and efficiently achieve the goal of "reduce size". And some form of sophisticated prelinking would be my proposal for something more systematic, which might come a lot cheaper with the security features to be added.
>
>> I also agree that PEI has acquired far too much functionality over
>> time, and we badly need to find some way to reverse that.  PEI is
>> pulling in more and more of the h/w initialization, leaving DXE mainly
>> to provide the boot interface (ACPI, UEFI services, setup screens,
>> etc.)  That puts more and more pressure on XIP storage and
>> cache-as-RAM.  How can we encourage a change?
> I'm not in the loop enough to comment on the XIP and CAR issues.
> However, HW init moving to PEI (or more generally, into a single stage) is not necessarily a bad thing, is it? As I've said before, I'm not sure what else to think of PEI than "DXE light". Both "initialise some of the hardware" with no clear scope boundary but with similar interfaces. If they were sort of fused to provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI" without having a clear boundary between HW init steps, would that be too bad? Now I'm *really* just blindly guessing because I barely looked into this project, but isn't that somewhat the idea of SlimBootloader?
>
> Your description of DXE sort of is it collapsing into an advanced BDS.
> Nate earlier said he'd like a clear distinction between DXE and BDS, which I didn't really get the point of. But if PEI consumed the HW init portions of DXE, a more advanced BDS (which also powers e.g. the HID stuff on the way) would be the logical remainder stage to separately support UEFI booting as a payload (for SlimBootloader, coreboot, ...).
>
> All of that was logically speaking. Even if the development was driven to its extreme, PEI and DXE cores would still share a bunch of code and it might make sense to share a foundation technically.
>
> Best regards,
> Marvin
>
>> Thanks,
>> Brian J. Johnson
>>
>>
>> On 4/15/22 03:15, Nate DeSimone wrote:
>>> Hi Andrew,
>>>
>>> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
>>>>
>>>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone
>>>>> <nathaniel.l.desimone@intel.com> wrote:
>>>>>
>>>>> Hi Marvin,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
>>>>>> Marvin Häuser
>>>>>> Sent: Thursday, April 14, 2022 12:56 AM
>>>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>>>>>> <nathaniel.l.desimone@intel.com>
>>>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>>>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>>>>>> <steven.shi@intel.com>
>>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>>
>>>>>> I feel like there are so many much easier solutions to this
>>>>>> problem that are at most limited by the clear specification. The
>>>>>> UEFI specification with regards to booting and all of that
>>>>>> obviously is of utmost importance.
>>>>> If you have a better idea that retains compatibility with the
>>>>> existing UEFI PI then I would be happy to hear it. Ultimately
>>>>> anything we do needs to be a pure extension that retains
>>>>> compatibility with old code. Given that restriction having the
>>>>> ability to coalesce all the LibraryClasses into a single module and
>>>>> have cross-module linking seems like the best way to handle it to me.
>>>>>
>>>>>> The UEFI PI specification parts that deal about internal
>>>>>> structure, as far as I know, are only in place to make it easy to
>>>>>> integrate Intel IP.
>>>>> Its not just for Intel. The biggest reason for it to increase the
>>>>> standardization of the boot flow across the PC ecosystem. We have
>>>>> learned from experience that firmware is super critical to get a
>>>>> product out the door but it is also difficult to write. So we try
>>>>> to make it as reusable as humanly possible.
>>>>>
>>>>>> In fact, I don’t *know*, but I’m pretty sure the very strict
>>>>>> separation between PEI and DXE was preserved mostly because MRC
>>>>>> was 32-bit-only for a long time. Glad that seems to have been
>>>>>> resolved, AMD does memory init by PSP nowadays.
>>>>> Having less complex early stages chain load more complex later
>>>>> stages is a common design pattern in firmware, not just UEFI. For
>>>>> example, your typical ARM system loads kinda like this:
>>>>>
>>>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
>>>>> --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
>>>>>
>>>>> Comparing relative complexity I believe the Intel UEFI PI design is
>>>>> actually pretty simple when you consider how much it gets done:
>>>>>
>>>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>>>>>
>>>>> My biggest criticism of the design is that the strict separation
>>>>> between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
>>>>>
>>>>> There are a few reasons why PEI was 32-bit for quite some time. The
>>>>> biggest one is the code size increase, 64-bit x86 code is 20-30%
>>>>> larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
>>>>> access to is the cache onboard the processor and for security
>>>>> reasons all that code has to fit within that RAM we generally do
>>>>> everything we can to make that image as small as possible. Second,
>>>>> 64-bit requires a page table and since we desired to keep PEI
>>>>> simple we tried to avoid that. Finally, the PI spec didn't allow a
>>>>> 64-bit PEI until recently. MRC is 32-bit code just because that is
>>>>> what PEI happens to be. Porting it to 64-bit is not terribly
>>>>> difficult.
>>>>>
>>>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any
>>>>> data structures that get shared between PEI and DXE (HOBs, PCDs,
>>>>> etc.) need to resolve to the same size and packing. LibraryClasses
>>>>> need to be written to compile properly in both modes. In the case
>>>>> of FSP API mode you need to resort to thunking between 32 and
>>>>> 64-bit modes during DXE. More or less we decided that the costs are
>>>>> starting to outweigh the advantages.
>>>>>
>>>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
>>>> works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
>>>>
>>>> In the PC ecosystem a single chipset family can power thousands of
>>>> unique designs. So the  DRAM memory needs to be external, support
>>>> lots of different chipset packages(signal integrity...), support the
>>>> lowest cost through the highest cost DRAM and thousands of different
>>>> board layouts. So programing DRAM takes a masters degree in antenna
>>>> design. I’ve seen MRC (Memory Reference Code) with over a MiB of
>>>> DEBUG prints in it, and it literally is printing histograms of what
>>>> it is tuning. So all this code has to run before the system has any
>>>> DRAM, thus it is running using the cache as RAM. I’ve not looked at
>>>> the x86 architecture specs form the vendors in a while, but back in
>>>> the day they did not support page tables in ROM or pinned cached.
>>>> Now it might work, but if it breaks your CPU vendor blames you so
>>>> you don’t code PEI in X64….
>>>>
>>>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
>>>> long time ago but I seem to remember the mask ROM used a table in
>>>> NOR FLASH to init memory and then copied an image from NOR FLASH
>>>> into DRAM and jumped to it. So PEI is kind of not really needed and
>>>> we implemented a PrePEI and jumped directly to DXE.
>>>>
>>>> Given I was around back in the day when all this stuff was designed
>>>> I can say SEC was always a place holder for security code, as
>>>> security code always has to run 1st. PEI (Pre EFI) was designed to
>>>> get DRAM programmed and then jump to DXE. It kind of also fell in
>>>> naturally to ACPI S3 flow since that was turning memory back on.
>>>> When we designed PEI we kind of though of it more like a boot loader
>>>> stage for the firmware that turned on memory and all the work would
>>>> happen in DXE. Then reality strikes and the existing BIOS assembly
>>>> programmers start learning C (lots of  cranky people) and they start
>>>> having to learn all about PEI to turn on memory. They had to write a
>>>> big chunk of code for the memory init in PEI. These folks had never
>>>> written any EFI code, so to them it was easier to move a lot of the
>>>> chipset init code into PEI as that is the world they had to figure
>>>> out to get memory turned on. I mean why learn EFI if you don’t have
>>>> to? So that is how we get so much code in IA32 (i386) on some
>>>> platforms. This start kind of biased future choices and how to
>>>> enable non edk2 code bases….
>>> One of the big reasons a lot of code that should have been written in
>>> DXE ended up in PEI is unfortunately due to the FSP and its inability
>>> to support DXE code.
>>>
>>>> Thanks,
>>>>
>>>> Andrew Fish
>>>>
>>>>
>>>>>> For many good reasons, Linux does not provide a stable kernel API.
>>>>>> This
>>>>>> allows to easily deploy breaking changes across the entire codebase.
>>>>>> Obviously, this is infeasible at a large scale when you need to
>>>>>> integrate IP blobs, but it would already help to break the
>>>>>> expectation that UEFI PI is a perfectly forwards- and
>>>>>> backwards-compatible system. DXE has SetMem and CopyMem as part of
>>>>>> gBS. Maybe I don’t like it that much as part of the spec itself,
>>>>>> but it’s a good idea technically. I’d probably opt to have a
>>>>>> quickly accessible implementation detail of important function
>>>>>> pointers appended to PEI and DXE services, used by in-tree
>>>>>> modules. This may break both
>>>>>> forwards- and backwards-compatibility, but as it only applies to
>>>>>> in-tree modules, that is fine so long as we let go of the
>>>>>> compatibility notions. PPIs and protocols are an option too of
>>>>>> course, but they have a lookup performance penalty. Compared to
>>>>>> dynamic linking, that should hopefully be negligible however.
>>>>>>
>>>>>> Absolutely optional to read, I don’t intend to waste anyone’s time
>>>>>> much, some philosophical stuff about my rationale:
>>>>>>
>>>>>> If you started UEFI from scratch right now, would it have strictly
>>>>>> separated PEI and DXE?
>>>>> For sure a clean slate design started today would look a lot
>>>>> different than PEI/DXE... which was a clean slate design circa 1998
>>>>> 😊. In my opinion, if we were to suddenly go back to the drawing
>>>>> board we would build something that is much closer to a full OS now
>>>>> than we did back then. There have been cases where not being able
>>>>> to use interrupt handlers and not having thread scheduling has
>>>>> prevented implementation of desired features. The ARM guys built
>>>>> LittleKernel (https://github.com/littlekernel/lk) for a lot of
>>>>> these reasons. In the data center world some have decided to go to
>>>>> the extreme of putting an entire copy of Linux in SPI so they can
>>>>> do a network boot that downloads the OS image using BitTorrent!
>>>>>
>>>>>> The duplication between PEI and DXE core, and by extension MM
>>>>>> core, would be my most obvious place to start reducing size. I
>>>>>> would probably opt for a PEI/DXE hybrid where it starts in
>>>>>> „minimal mode“ (maybe think of PEI more like a microkernel here)
>>>>>> and after memory is initialised, the rest of DXE is loaded. And MM
>>>>>> core would get no loading at all, this requirement has gladly been
>>>>>> dropped ages ago. Just one prelinked snapshot of the address space
>>>>>> with a relocation table and a safe self-relocator on entry (this
>>>>>> is needed at the very least for ARM).
>>>>>>
>>>>>> Ironically, with my idea for MM, dynamic loading would be free as
>>>>>> everything is prelinked anyway. The same is true for PEI XIP, it
>>>>>> is prelinked by nature.
>>>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that
>>>>> does get used for the PEI GOP driver.
>>>>>
>>>>>> What I do not like is the additional dynamic linking code at
>>>>>> load-time for non- XIP modules. Though, the more I think about it,
>>>>>> the more I wonder whether not the entirety of UEFI could be
>>>>>> composed of prelinked, relocatable memory snapshots without
>>>>>> traditional image loading entirely (for in-FW stuff). macOS has a
>>>>>> similar concept with its “Kernel Collections”.
>>>>>> Well, way
>>>>>> too much off-topic now. :)
>>>>> If you make the assumption that 100% of the code is compiled all at
>>>>> once then yes that works. UEFI was designed so that assumption does
>>>>> not need to be true. There are good use cases for it: OpROMs,
>>>>> generic OS loaders, network boot, etc.
>>>>>
>>>>> Thanks,
>>>>> Nate
>>>>>
>>>>>> Why am I explaining all this despite the fact everyone knows this
>>>>>> will never happen? Because I don’t like the notion of fixing
>>>>>> issues of an already overcomplicated system by adding even more
>>>>>> complicated stuff.
>>>>>> Especially
>>>>>> when the existing overcomplicated stuff is already uncomfortably
>>>>>> broken.
>>>>>>
>>>>>> Best regards,
>>>>>> Marvin
>>>>>>
>>>>>>> For XIP PEI code… it will really help and would be very timely
>>>>>>> since the
>>>>>> transition of PEI from 32-bit to 64-bit is going to increase the
>>>>>> size of PEI by ~20%.
>>>>>>> Thanks,
>>>>>>> Nate
>>>>>>>
>>>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>>>>>> <mhaeuser@posteo.de>
>>>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>>>>>> <steven.shi@intel.com>
>>>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>>>
>>>>>>> Hi Marvin, Ada,
>>>>>>>
>>>>>>> Some comments:
>>>>>>>
>>>>>>> I don't think the purpose of the dynamic linker is to treat EFI
>>>>>>> as a complete operating system, but to try to eliminate the
>>>>>>> static linking that may be needlessly duplicating code that could
>>>>>>> instead be put in a
>>>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are
>>>>>> linked into a *lot* of .efi, instead of being just a library. It'd
>>>>>> be nice to see some numbers on this (something like Google's
>>>>>> bloaty could be run on every .efi file, in order to understand how
>>>>>> much file space we would actually save).
>>>>>>> Other comments inline.
>>>>>>>
>>>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>> wrote:
>>>>>>> i was replying via the groups.io<http://groups.io > web
>>>>>>> interface, I'm guessing that messed up the thread? i haven't used
>>>>>>> mailing lists before and don't know how they work. I'll use my
>>>>>>> mail client from here on.
>>>>>>>
>>>>>>> I'm on board with not treating EFI as an operating system. the
>>>>>>> more i think about it the more it looks like scope creep.
>>>>>>>
>>>>>>> Agreed.
>>>>>>>
>>>>>>>
>>>>>>> I'm not quite as enthusiastic
>>>>>>> about it as i was at first glance.
>>>>>>>
>>>>>>> I'm still keen on doing my gsoc proposal for edk, though, and
>>>>>>> even if this task and the acpica application are decided to be
>>>>>>> out of scope unit testing,
>>>>>>>
>>>>>>> How about fuzz-testing? This is also something edk2 needs quite
>>>>>>> badly. At
>>>>>> Acidanthera, we compile edk2 code in userspace outside the edk2
>>>>>> build system and fuzz with dummy applications.
>>>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>>>>> https://llvm.org/docs/LibFuzzer.html ) and is something I could
>>>>>> happily mentor.
>>>>>>> clang integration
>>>>>>>
>>>>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>>>>> There are working UBSan concepts, but they also need to be
>>>>>>> mainlined.
>>>>>>>
>>>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be
>>>>>>> me and
>>>>>> some other, more senior, mentor (possibly Steven Shi, which I
>>>>>> included in the task).
>>>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and
>>>>>>> possibly some other sanitizer it's quite possible that it could
>>>>>>> be considered a large
>>>>>> project (which means more hours but a larger stipend too). Fuzzing
>>>>>> + coverage could be very nice additions to this project idea.
>>>>>>> Also, is stress-testing a decent idea?
>>>>>>>
>>>>>>>
>>>>>>> and source-level debugging are all relevant to my interests.
>>>>>>>
>>>>>>> how about your ideas for security stuff?
>>>>>>>
>>>>>>> I want the entirety of MM to leverage SmmMemLib and to support SMAP.
>>>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
>>>>>> only work on MMRAM. Also evaluation of how to best avoid pointers
>>>>>> in MM communication buffers would be nice.
>>>>>>> There also is a bunch of other stuff, like working out moving a
>>>>>>> part of
>>>>>> CpuDxe into DxeCore to have memory protection live immediately,
>>>>>> memory protection in PEI, a replacement for the TE format (it’s
>>>>>> buggy and most platforms mostly abandoned it over various issues),
>>>>>> and alternatives to guarding critical code with SMM (like allowing
>>>>>> NVRAM commits only as part of a reboot).
>>>>>>> I personally find all of those projects very important, but I
>>>>>>> cannot promise many people agree. Especially those that impose
>>>>>>> global changes (most notably the TE replacement) may be very
>>>>>>> tedious to submit.
>>>>>>> Gladly, I believe you can submit multiple proposals (?)
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Marvin
>>>>>>>
>>>>>>>
>>>>>>> I'm not very knowledgeable about
>>>>>>> trusted platform or secure boot but I'm willing to learn whatever
>>>>>>> is necessary to get something spun up for my proposal.
>>>>>>>
>>>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>>
>>>>>>> Do you use the “reply all” option in your mail client? Looks like
>>>>>>> my CCs have been dropped again. Comments inline.
>>>>>>>
>>>>>>> On 13. Apr 2022, at 12:54, Ada Christine
>>>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>>>>> wrote:
>>>>>>> Hi, Marvin
>>>>>>>
>>>>>>> Its similarity to my own latest experiment is the key to what
>>>>>>> grabbed my attention. I have no particular use case in mind for
>>>>>>> it, but I see its potential for anybody developing larger
>>>>>>> applications in that when a library is changed there's no need to
>>>>>>> distribute a new version of the whole binary, just the relevant
>>>>>>> library module.
>>>>>>>
>>>>>>> I really do not like the trend of treating UEFI as a full-fledged
>>>>>>> OS - it is not. The most used UEFI applications, OS loaders, are
>>>>>>> really not that huge and are distributed as part of the OS image
>>>>>>> anyway.
>>>>>>> Even for
>>>>>>> less used applications, you will always get a full snapshot anyhow.
>>>>>>> Gladly we don’t have auto-update and package management yet. :)
>>>>>>>
>>>>>>>
>>>>>>> I slept on it and it occurred to me that the whole thing could
>>>>>>> operate
>>>>>>> similarly to the shell protocol in that the linker/loader is
>>>>>>> itself an
>>>>>>> application that does a LoadImage() on the application needing
>>>>>>> dynamic
>>>>>>> linking facilities.
>>>>>>>
>>>>>>> That would mean the linker itself is shipped with every application
>>>>>>> that requires it? Otherwise it doesn’t make much sense for it to
>>>>>>> be an
>>>>>>> app and below’s problems apply.
>>>>>>>
>>>>>>> If however the whole plan is making the linker as a DXE and
>>>>>>> including
>>>>>>> it with the firmware, that I'm not quite as sure about. That would
>>>>>>> necessarily tie any applications using dynamic linking to
>>>>>>> TianoCore or
>>>>>>> any firmware distribution that derives from it.
>>>>>>>
>>>>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>>>>> are slow to update their snapshots or have just given up doing that
>>>>>>> entirely. Distributing it for external applications like OS loaders
>>>>>>> would mean this can be leveraged probably no earlier than 10 years
>>>>>>> from now. And for in-firmware things, I have a hard time thinking
>>>>>>> about a
>>>>>> use-case that outweighs the drawbacks.
>>>>>>>
>>>>>>> To shift the topic slightly back to GSoC, however, I'm willing to
>>>>>>> work
>>>>>>> on other items on the task list. Unit testing and an ACPICA
>>>>>>> application are the alternative projects I had thought about. I need
>>>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I
>>>>>>> know
>>>>>>> a tiny bit about porting ACPICA as I also have plans to
>>>>>>> incorporate it into my
>>>>>> own project.
>>>>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>>>>> them as appropriate yet and I’m not here to drive you away from this
>>>>>>> specific task (or the others). However, I’m still curious and
>>>>>>> concerned. :)
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Marvin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Pedro Falcato
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
>
>
>
>
>
> 
>
>

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Nate DeSimone]

Hi Marvin,

> -----Original Message-----
> From: Marvin Häuser <mhaeuser@posteo.de>
> Sent: Friday, April 15, 2022 9:44 AM
> To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@hpe.com>;
> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Andrew Fish
> <afish@apple.com>; devel@edk2.groups.io
> Cc: Pedro Falcato <pedro.falcato@gmail.com>; adachristine18@gmail.com;
> Shi, Steven <steven.shi@intel.com>
> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> 
> Hey Brian,
> 
> On 15.04.22 18:22, Brian J. Johnson wrote:
> > Nate, Andrew, Marvin, Pedro, Ada, et al,
> >
> > This is a great discussion.  I've been debating where to weigh in...
> 
> Personally, I don't think this topic can get enough attention. Thanks! :)
> 
> >
> > I agree that some sort of library sharing to reduce image size would
> > be very helpful.  Although some parts of the BIOS are built
> > separately, large parts of it are compiled at the same time, so there
> > should be ample opportunity for tool-directed sharing.  Some form of
> > pre-linking modules together may be the easiest way to do that.  But
> > however it gets implemented, we should try to make it automatic:
> > requiring library writers to manually add a lot of metadata or write
> > thunks or shims would be a barrier to adoption.
> 
> My suggestion to use a private function array would indeed require shims,
> however this would be the "better than nothing I guess" solution where only
> the most used and most painful to duplicate functions are shared, e.g.,
> memory and maybe common string printing functions. This would quickly and
> efficiently achieve the goal of "reduce size". And some form of sophisticated
> prelinking would be my proposal for something more systematic, which
> might come a lot cheaper with the security features to be added.
> 
> >
> > I also agree that PEI has acquired far too much functionality over
> > time, and we badly need to find some way to reverse that.  PEI is
> > pulling in more and more of the h/w initialization, leaving DXE mainly
> > to provide the boot interface (ACPI, UEFI services, setup screens,
> > etc.)  That puts more and more pressure on XIP storage and
> > cache-as-RAM.  How can we encourage a change?
> 
> I'm not in the loop enough to comment on the XIP and CAR issues.
> However, HW init moving to PEI (or more generally, into a single stage) is not
> necessarily a bad thing, is it? As I've said before, I'm not sure what else to
> think of PEI than "DXE light". Both "initialise some of the hardware" with no
> clear scope boundary but with similar interfaces. If they were sort of fused to
> provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI"
> without having a clear boundary between HW init steps, would that be too
> bad? Now I'm *really* just blindly guessing because I barely looked into this
> project, but isn't that somewhat the idea of SlimBootloader?
> 
> Your description of DXE sort of is it collapsing into an advanced BDS.
> Nate earlier said he'd like a clear distinction between DXE and BDS, which I
> didn't really get the point of.

I guess what I'm trying to express is where does DXE end and where does BDS begin? A lot of drivers that are only used in BDS get dispatched early in DXE. Conversely, it is entirely possible for some DXE drivers to only run after we have technically entered the BDS phase. The line where DXE ends and BDS begins is blurry and not really a line but more like a rectangle.

To Isaac's point, there is a good argument to be made for beefing up PEI's capabilities to the point that DXE and BDS more or less become one thing. But what does that mean for PEI? Do we change from the 1-D PPI database to the 2-D Handle database in order to express device topologies better for example? That would be useful for stuff like the PEI UFS drivers as they run into difficulties expressing LUN IDs that the DXE version does not for example. Not sure if I would actually formally propose that in the PI WG but interesting stuff to think about regardless.

> But if PEI consumed the HW init portions of DXE,
> a more advanced BDS (which also powers e.g. the HID stuff on the way)
> would be the logical remainder stage to separately support UEFI booting as a
> payload (for SlimBootloader, coreboot, ...).
> 
> All of that was logically speaking. Even if the development was driven to its
> extreme, PEI and DXE cores would still share a bunch of code and it might
> make sense to share a foundation technically.
> 
> Best regards,
> Marvin
> 
> >
> > Thanks,
> > Brian J. Johnson
> >
> >
> > On 4/15/22 03:15, Nate DeSimone wrote:
> >> Hi Andrew,
> >>
> >> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
> >>>
> >>>
> >>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone
> >>>> <nathaniel.l.desimone@intel.com> wrote:
> >>>>
> >>>> Hi Marvin,
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> >>>>> Marvin Häuser
> >>>>> Sent: Thursday, April 14, 2022 12:56 AM
> >>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
> >>>>> <nathaniel.l.desimone@intel.com>
> >>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
> >>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
> >>>>> <steven.shi@intel.com>
> >>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> >>>>>
> >>>>> I feel like there are so many much easier solutions to this
> >>>>> problem that are at most limited by the clear specification. The
> >>>>> UEFI specification with regards to booting and all of that
> >>>>> obviously is of utmost importance.
> >>>>
> >>>> If you have a better idea that retains compatibility with the
> >>>> existing UEFI PI then I would be happy to hear it. Ultimately
> >>>> anything we do needs to be a pure extension that retains
> >>>> compatibility with old code. Given that restriction having the
> >>>> ability to coalesce all the LibraryClasses into a single module and
> >>>> have cross-module linking seems like the best way to handle it to me.
> >>>>
> >>>>> The UEFI PI specification parts that deal about internal
> >>>>> structure, as far as I know, are only in place to make it easy to
> >>>>> integrate Intel IP.
> >>>>
> >>>> Its not just for Intel. The biggest reason for it to increase the
> >>>> standardization of the boot flow across the PC ecosystem. We have
> >>>> learned from experience that firmware is super critical to get a
> >>>> product out the door but it is also difficult to write. So we try
> >>>> to make it as reusable as humanly possible.
> >>>>
> >>>>> In fact, I don’t *know*, but I’m pretty sure the very strict
> >>>>> separation between PEI and DXE was preserved mostly because MRC
> >>>>> was 32-bit-only for a long time. Glad that seems to have been
> >>>>> resolved, AMD does memory init by PSP nowadays.
> >>>>
> >>>> Having less complex early stages chain load more complex later
> >>>> stages is a common design pattern in firmware, not just UEFI. For
> >>>> example, your typical ARM system loads kinda like this:
> >>>>
> >>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
> >>>> --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
> >>>>
> >>>> Comparing relative complexity I believe the Intel UEFI PI design is
> >>>> actually pretty simple when you consider how much it gets done:
> >>>>
> >>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
> >>>>
> >>>> My biggest criticism of the design is that the strict separation
> >>>> between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
> >>>>
> >>>> There are a few reasons why PEI was 32-bit for quite some time. The
> >>>> biggest one is the code size increase, 64-bit x86 code is 20-30%
> >>>> larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
> >>>> access to is the cache onboard the processor and for security
> >>>> reasons all that code has to fit within that RAM we generally do
> >>>> everything we can to make that image as small as possible. Second,
> >>>> 64-bit requires a page table and since we desired to keep PEI
> >>>> simple we tried to avoid that. Finally, the PI spec didn't allow a
> >>>> 64-bit PEI until recently. MRC is 32-bit code just because that is
> >>>> what PEI happens to be. Porting it to 64-bit is not terribly
> >>>> difficult.
> >>>>
> >>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any
> >>>> data structures that get shared between PEI and DXE (HOBs, PCDs,
> >>>> etc.) need to resolve to the same size and packing. LibraryClasses
> >>>> need to be written to compile properly in both modes. In the case
> >>>> of FSP API mode you need to resort to thunking between 32 and
> >>>> 64-bit modes during DXE. More or less we decided that the costs are
> >>>> starting to outweigh the advantages.
> >>>>
> >>>
> >>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
> >>> works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
> >>>
> >>> In the PC ecosystem a single chipset family can power thousands of
> >>> unique designs. So the  DRAM memory needs to be external, support
> >>> lots of different chipset packages(signal integrity...), support the
> >>> lowest cost through the highest cost DRAM and thousands of different
> >>> board layouts. So programing DRAM takes a masters degree in antenna
> >>> design. I’ve seen MRC (Memory Reference Code) with over a MiB of
> >>> DEBUG prints in it, and it literally is printing histograms of what
> >>> it is tuning. So all this code has to run before the system has any
> >>> DRAM, thus it is running using the cache as RAM. I’ve not looked at
> >>> the x86 architecture specs form the vendors in a while, but back in
> >>> the day they did not support page tables in ROM or pinned cached.
> >>> Now it might work, but if it breaks your CPU vendor blames you so
> >>> you don’t code PEI in X64….
> >>>
> >>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
> >>> long time ago but I seem to remember the mask ROM used a table in
> >>> NOR FLASH to init memory and then copied an image from NOR FLASH
> >>> into DRAM and jumped to it. So PEI is kind of not really needed and
> >>> we implemented a PrePEI and jumped directly to DXE.
> >>>
> >>> Given I was around back in the day when all this stuff was designed
> >>> I can say SEC was always a place holder for security code, as
> >>> security code always has to run 1st. PEI (Pre EFI) was designed to
> >>> get DRAM programmed and then jump to DXE. It kind of also fell in
> >>> naturally to ACPI S3 flow since that was turning memory back on.
> >>> When we designed PEI we kind of though of it more like a boot loader
> >>> stage for the firmware that turned on memory and all the work would
> >>> happen in DXE. Then reality strikes and the existing BIOS assembly
> >>> programmers start learning C (lots of  cranky people) and they start
> >>> having to learn all about PEI to turn on memory. They had to write a
> >>> big chunk of code for the memory init in PEI. These folks had never
> >>> written any EFI code, so to them it was easier to move a lot of the
> >>> chipset init code into PEI as that is the world they had to figure
> >>> out to get memory turned on. I mean why learn EFI if you don’t have
> >>> to? So that is how we get so much code in IA32 (i386) on some
> >>> platforms. This start kind of biased future choices and how to
> >>> enable non edk2 code bases….
> >>
> >> One of the big reasons a lot of code that should have been written in
> >> DXE ended up in PEI is unfortunately due to the FSP and its inability
> >> to support DXE code.
> >>
> >>>
> >>> Thanks,
> >>>
> >>> Andrew Fish
> >>>
> >>>
> >>>>>
> >>>>> For many good reasons, Linux does not provide a stable kernel API.
> >>>>> This
> >>>>> allows to easily deploy breaking changes across the entire codebase.
> >>>>> Obviously, this is infeasible at a large scale when you need to
> >>>>> integrate IP blobs, but it would already help to break the
> >>>>> expectation that UEFI PI is a perfectly forwards- and
> >>>>> backwards-compatible system. DXE has SetMem and CopyMem as
> part of
> >>>>> gBS. Maybe I don’t like it that much as part of the spec itself,
> >>>>> but it’s a good idea technically. I’d probably opt to have a
> >>>>> quickly accessible implementation detail of important function
> >>>>> pointers appended to PEI and DXE services, used by in-tree
> >>>>> modules. This may break both
> >>>>> forwards- and backwards-compatibility, but as it only applies to
> >>>>> in-tree modules, that is fine so long as we let go of the
> >>>>> compatibility notions. PPIs and protocols are an option too of
> >>>>> course, but they have a lookup performance penalty. Compared to
> >>>>> dynamic linking, that should hopefully be negligible however.
> >>>>>
> >>>>> Absolutely optional to read, I don’t intend to waste anyone’s time
> >>>>> much, some philosophical stuff about my rationale:
> >>>>>
> >>>>> If you started UEFI from scratch right now, would it have strictly
> >>>>> separated PEI and DXE?
> >>>>
> >>>> For sure a clean slate design started today would look a lot
> >>>> different than PEI/DXE... which was a clean slate design circa 1998
> >>>> 😊. In my opinion, if we were to suddenly go back to the drawing
> >>>> board we would build something that is much closer to a full OS now
> >>>> than we did back then. There have been cases where not being able
> >>>> to use interrupt handlers and not having thread scheduling has
> >>>> prevented implementation of desired features. The ARM guys built
> >>>> LittleKernel (https://github.com/littlekernel/lk) for a lot of
> >>>> these reasons. In the data center world some have decided to go to
> >>>> the extreme of putting an entire copy of Linux in SPI so they can
> >>>> do a network boot that downloads the OS image using BitTorrent!
> >>>>
> >>>>> The duplication between PEI and DXE core, and by extension MM
> >>>>> core, would be my most obvious place to start reducing size. I
> >>>>> would probably opt for a PEI/DXE hybrid where it starts in
> >>>>> „minimal mode“ (maybe think of PEI more like a microkernel here)
> >>>>> and after memory is initialised, the rest of DXE is loaded. And MM
> >>>>> core would get no loading at all, this requirement has gladly been
> >>>>> dropped ages ago. Just one prelinked snapshot of the address space
> >>>>> with a relocation table and a safe self-relocator on entry (this
> >>>>> is needed at the very least for ARM).
> >>>>>
> >>>>> Ironically, with my idea for MM, dynamic loading would be free as
> >>>>> everything is prelinked anyway. The same is true for PEI XIP, it
> >>>>> is prelinked by nature.
> >>>>
> >>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that
> >>>> does get used for the PEI GOP driver.
> >>>>
> >>>>> What I do not like is the additional dynamic linking code at
> >>>>> load-time for non- XIP modules. Though, the more I think about it,
> >>>>> the more I wonder whether not the entirety of UEFI could be
> >>>>> composed of prelinked, relocatable memory snapshots without
> >>>>> traditional image loading entirely (for in-FW stuff). macOS has a
> >>>>> similar concept with its “Kernel Collections”.
> >>>>> Well, way
> >>>>> too much off-topic now. :)
> >>>>
> >>>> If you make the assumption that 100% of the code is compiled all at
> >>>> once then yes that works. UEFI was designed so that assumption does
> >>>> not need to be true. There are good use cases for it: OpROMs,
> >>>> generic OS loaders, network boot, etc.
> >>>>
> >>>> Thanks,
> >>>> Nate
> >>>>
> >>>>>
> >>>>> Why am I explaining all this despite the fact everyone knows this
> >>>>> will never happen? Because I don’t like the notion of fixing
> >>>>> issues of an already overcomplicated system by adding even more
> >>>>> complicated stuff.
> >>>>> Especially
> >>>>> when the existing overcomplicated stuff is already uncomfortably
> >>>>> broken.
> >>>>>
> >>>>> Best regards,
> >>>>> Marvin
> >>>>>
> >>>>>> For XIP PEI code… it will really help and would be very timely
> >>>>>> since the
> >>>>> transition of PEI from 32-bit to 64-bit is going to increase the
> >>>>> size of PEI by ~20%.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Nate
> >>>>>>
> >>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
> >>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
> >>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
> >>>>>> <mhaeuser@posteo.de>
> >>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
> >>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
> >>>>>> <steven.shi@intel.com>
> >>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> >>>>>>
> >>>>>> Hi Marvin, Ada,
> >>>>>>
> >>>>>> Some comments:
> >>>>>>
> >>>>>> I don't think the purpose of the dynamic linker is to treat EFI
> >>>>>> as a complete operating system, but to try to eliminate the
> >>>>>> static linking that may be needlessly duplicating code that could
> >>>>>> instead be put in a
> >>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are
> >>>>> linked into a *lot* of .efi, instead of being just a library. It'd
> >>>>> be nice to see some numbers on this (something like Google's
> >>>>> bloaty could be run on every .efi file, in order to understand how
> >>>>> much file space we would actually save).
> >>>>>>
> >>>>>> Other comments inline.
> >>>>>>
> >>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
> >>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> >>>>>>
> >>>>>> On 13. Apr 2022, at 16:38, Ada Christine
> >>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
> wrote:
> >>>>>> i was replying via the groups.io<http://groups.io > web
> >>>>>> interface, I'm guessing that messed up the thread? i haven't used
> >>>>>> mailing lists before and don't know how they work. I'll use my
> >>>>>> mail client from here on.
> >>>>>>
> >>>>>> I'm on board with not treating EFI as an operating system. the
> >>>>>> more i think about it the more it looks like scope creep.
> >>>>>>
> >>>>>> Agreed.
> >>>>>>
> >>>>>>
> >>>>>> I'm not quite as enthusiastic
> >>>>>> about it as i was at first glance.
> >>>>>>
> >>>>>> I'm still keen on doing my gsoc proposal for edk, though, and
> >>>>>> even if this task and the acpica application are decided to be
> >>>>>> out of scope unit testing,
> >>>>>>
> >>>>>> How about fuzz-testing? This is also something edk2 needs quite
> >>>>>> badly. At
> >>>>> Acidanthera, we compile edk2 code in userspace outside the edk2
> >>>>> build system and fuzz with dummy applications.
> >>>>>>
> >>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
> >>>>> https://llvm.org/docs/LibFuzzer.html ) and is something I could
> >>>>> happily mentor.
> >>>>>> clang integration
> >>>>>>
> >>>>>> Pedro and Vitaly are looking for someone to finish ASan:
> >>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
> >>>>>> There are working UBSan concepts, but they also need to be
> >>>>>> mainlined.
> >>>>>>
> >>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be
> >>>>>> me and
> >>>>> some other, more senior, mentor (possibly Steven Shi, which I
> >>>>> included in the task).
> >>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and
> >>>>>> possibly some other sanitizer it's quite possible that it could
> >>>>>> be considered a large
> >>>>> project (which means more hours but a larger stipend too). Fuzzing
> >>>>> + coverage could be very nice additions to this project idea.
> >>>>>> Also, is stress-testing a decent idea?
> >>>>>>
> >>>>>>
> >>>>>> and source-level debugging are all relevant to my interests.
> >>>>>>
> >>>>>> how about your ideas for security stuff?
> >>>>>>
> >>>>>> I want the entirety of MM to leverage SmmMemLib and to support
> SMAP.
> >>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib
> would
> >>>>> only work on MMRAM. Also evaluation of how to best avoid pointers
> >>>>> in MM communication buffers would be nice.
> >>>>>>
> >>>>>> There also is a bunch of other stuff, like working out moving a
> >>>>>> part of
> >>>>> CpuDxe into DxeCore to have memory protection live immediately,
> >>>>> memory protection in PEI, a replacement for the TE format (it’s
> >>>>> buggy and most platforms mostly abandoned it over various issues),
> >>>>> and alternatives to guarding critical code with SMM (like allowing
> >>>>> NVRAM commits only as part of a reboot).
> >>>>>>
> >>>>>> I personally find all of those projects very important, but I
> >>>>>> cannot promise many people agree. Especially those that impose
> >>>>>> global changes (most notably the TE replacement) may be very
> >>>>>> tedious to submit.
> >>>>>> Gladly, I believe you can submit multiple proposals (?)
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Marvin
> >>>>>>
> >>>>>>
> >>>>>> I'm not very knowledgeable about
> >>>>>> trusted platform or secure boot but I'm willing to learn whatever
> >>>>>> is necessary to get something spun up for my proposal.
> >>>>>>
> >>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
> >>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Do you use the “reply all” option in your mail client? Looks like
> >>>>>> my CCs have been dropped again. Comments inline.
> >>>>>>
> >>>>>> On 13. Apr 2022, at 12:54, Ada Christine
> >>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
> >>>>>> wrote:
> >>>>>> Hi, Marvin
> >>>>>>
> >>>>>> Its similarity to my own latest experiment is the key to what
> >>>>>> grabbed my attention. I have no particular use case in mind for
> >>>>>> it, but I see its potential for anybody developing larger
> >>>>>> applications in that when a library is changed there's no need to
> >>>>>> distribute a new version of the whole binary, just the relevant
> >>>>>> library module.
> >>>>>>
> >>>>>> I really do not like the trend of treating UEFI as a full-fledged
> >>>>>> OS - it is not. The most used UEFI applications, OS loaders, are
> >>>>>> really not that huge and are distributed as part of the OS image
> >>>>>> anyway.
> >>>>>> Even for
> >>>>>> less used applications, you will always get a full snapshot anyhow.
> >>>>>> Gladly we don’t have auto-update and package management yet. :)
> >>>>>>
> >>>>>>
> >>>>>> I slept on it and it occurred to me that the whole thing could
> >>>>>> operate
> >>>>>> similarly to the shell protocol in that the linker/loader is
> >>>>>> itself an
> >>>>>> application that does a LoadImage() on the application needing
> >>>>>> dynamic
> >>>>>> linking facilities.
> >>>>>>
> >>>>>> That would mean the linker itself is shipped with every application
> >>>>>> that requires it? Otherwise it doesn’t make much sense for it to
> >>>>>> be an
> >>>>>> app and below’s problems apply.
> >>>>>>
> >>>>>> If however the whole plan is making the linker as a DXE and
> >>>>>> including
> >>>>>> it with the firmware, that I'm not quite as sure about. That would
> >>>>>> necessarily tie any applications using dynamic linking to
> >>>>>> TianoCore or
> >>>>>> any firmware distribution that derives from it.
> >>>>>>
> >>>>>> I think that was the idea referred to as “edk2 core” by Steven, but
> >>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
> >>>>>> edk2, so that itself is not the problem, but versioning is. Vendors
> >>>>>> are slow to update their snapshots or have just given up doing that
> >>>>>> entirely. Distributing it for external applications like OS loaders
> >>>>>> would mean this can be leveraged probably no earlier than 10 years
> >>>>>> from now. And for in-firmware things, I have a hard time thinking
> >>>>>> about a
> >>>>> use-case that outweighs the drawbacks.
> >>>>>>
> >>>>>>
> >>>>>> To shift the topic slightly back to GSoC, however, I'm willing to
> >>>>>> work
> >>>>>> on other items on the task list. Unit testing and an ACPICA
> >>>>>> application are the alternative projects I had thought about. I need
> >>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I
> >>>>>> know
> >>>>>> a tiny bit about porting ACPICA as I also have plans to
> >>>>>> incorporate it into my
> >>>>> own project.
> >>>>>>
> >>>>>> I have a few more ideas for security stuff, but Nate did not confirm
> >>>>>> them as appropriate yet and I’m not here to drive you away from
> this
> >>>>>> specific task (or the others). However, I’m still curious and
> >>>>>> concerned. :)
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Marvin
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Pedro Falcato
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> > 
> >
> >

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Ada Christine]

[-- Attachment #1: Type: text/plain, Size: 27190 bytes --]

I've submitted an updated version of my proposal document. There's a little
more detail and I've shifted from using the term "dynamic linking" to "code
sharing" to reflect the fact that the question of which solution will fit
best is not yet answered. If there are any more important matters not
addressed in the proposal please do let me know and I will fit them in. For
those who can view it, the proposal is here:
https://summerofcode.withgoogle.com/proposals/details/whGX9tXL

Bless!

- Ada Christine

On Sat, Apr 16, 2022 at 4:24 AM Nate DeSimone <
nathaniel.l.desimone@intel.com> wrote:

> Hi Marvin,
>
> > -----Original Message-----
> > From: Marvin Häuser <mhaeuser@posteo.de>
> > Sent: Friday, April 15, 2022 9:44 AM
> > To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@hpe.com>;
> > Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Andrew Fish
> > <afish@apple.com>; devel@edk2.groups.io
> > Cc: Pedro Falcato <pedro.falcato@gmail.com>; adachristine18@gmail.com;
> > Shi, Steven <steven.shi@intel.com>
> > Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> >
> > Hey Brian,
> >
> > On 15.04.22 18:22, Brian J. Johnson wrote:
> > > Nate, Andrew, Marvin, Pedro, Ada, et al,
> > >
> > > This is a great discussion.  I've been debating where to weigh in...
> >
> > Personally, I don't think this topic can get enough attention. Thanks! :)
> >
> > >
> > > I agree that some sort of library sharing to reduce image size would
> > > be very helpful.  Although some parts of the BIOS are built
> > > separately, large parts of it are compiled at the same time, so there
> > > should be ample opportunity for tool-directed sharing.  Some form of
> > > pre-linking modules together may be the easiest way to do that.  But
> > > however it gets implemented, we should try to make it automatic:
> > > requiring library writers to manually add a lot of metadata or write
> > > thunks or shims would be a barrier to adoption.
> >
> > My suggestion to use a private function array would indeed require shims,
> > however this would be the "better than nothing I guess" solution where
> only
> > the most used and most painful to duplicate functions are shared, e.g.,
> > memory and maybe common string printing functions. This would quickly and
> > efficiently achieve the goal of "reduce size". And some form of
> sophisticated
> > prelinking would be my proposal for something more systematic, which
> > might come a lot cheaper with the security features to be added.
> >
> > >
> > > I also agree that PEI has acquired far too much functionality over
> > > time, and we badly need to find some way to reverse that.  PEI is
> > > pulling in more and more of the h/w initialization, leaving DXE mainly
> > > to provide the boot interface (ACPI, UEFI services, setup screens,
> > > etc.)  That puts more and more pressure on XIP storage and
> > > cache-as-RAM.  How can we encourage a change?
> >
> > I'm not in the loop enough to comment on the XIP and CAR issues.
> > However, HW init moving to PEI (or more generally, into a single stage)
> is not
> > necessarily a bad thing, is it? As I've said before, I'm not sure what
> else to
> > think of PEI than "DXE light". Both "initialise some of the hardware"
> with no
> > clear scope boundary but with similar interfaces. If they were sort of
> fused to
> > provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI"
> > without having a clear boundary between HW init steps, would that be too
> > bad? Now I'm *really* just blindly guessing because I barely looked into
> this
> > project, but isn't that somewhat the idea of SlimBootloader?
> >
> > Your description of DXE sort of is it collapsing into an advanced BDS.
> > Nate earlier said he'd like a clear distinction between DXE and BDS,
> which I
> > didn't really get the point of.
>
> I guess what I'm trying to express is where does DXE end and where does
> BDS begin? A lot of drivers that are only used in BDS get dispatched early
> in DXE. Conversely, it is entirely possible for some DXE drivers to only
> run after we have technically entered the BDS phase. The line where DXE
> ends and BDS begins is blurry and not really a line but more like a
> rectangle.
>
> To Isaac's point, there is a good argument to be made for beefing up PEI's
> capabilities to the point that DXE and BDS more or less become one thing.
> But what does that mean for PEI? Do we change from the 1-D PPI database to
> the 2-D Handle database in order to express device topologies better for
> example? That would be useful for stuff like the PEI UFS drivers as they
> run into difficulties expressing LUN IDs that the DXE version does not for
> example. Not sure if I would actually formally propose that in the PI WG
> but interesting stuff to think about regardless.
>
> > But if PEI consumed the HW init portions of DXE,
> > a more advanced BDS (which also powers e.g. the HID stuff on the way)
> > would be the logical remainder stage to separately support UEFI booting
> as a
> > payload (for SlimBootloader, coreboot, ...).
> >
> > All of that was logically speaking. Even if the development was driven
> to its
> > extreme, PEI and DXE cores would still share a bunch of code and it might
> > make sense to share a foundation technically.
> >
> > Best regards,
> > Marvin
> >
> > >
> > > Thanks,
> > > Brian J. Johnson
> > >
> > >
> > > On 4/15/22 03:15, Nate DeSimone wrote:
> > >> Hi Andrew,
> > >>
> > >> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
> > >>>
> > >>>
> > >>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone
> > >>>> <nathaniel.l.desimone@intel.com> wrote:
> > >>>>
> > >>>> Hi Marvin,
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> > >>>>> Marvin Häuser
> > >>>>> Sent: Thursday, April 14, 2022 12:56 AM
> > >>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
> > >>>>> <nathaniel.l.desimone@intel.com>
> > >>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
> > >>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
> > >>>>> <steven.shi@intel.com>
> > >>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> > >>>>>
> > >>>>> I feel like there are so many much easier solutions to this
> > >>>>> problem that are at most limited by the clear specification. The
> > >>>>> UEFI specification with regards to booting and all of that
> > >>>>> obviously is of utmost importance.
> > >>>>
> > >>>> If you have a better idea that retains compatibility with the
> > >>>> existing UEFI PI then I would be happy to hear it. Ultimately
> > >>>> anything we do needs to be a pure extension that retains
> > >>>> compatibility with old code. Given that restriction having the
> > >>>> ability to coalesce all the LibraryClasses into a single module and
> > >>>> have cross-module linking seems like the best way to handle it to
> me.
> > >>>>
> > >>>>> The UEFI PI specification parts that deal about internal
> > >>>>> structure, as far as I know, are only in place to make it easy to
> > >>>>> integrate Intel IP.
> > >>>>
> > >>>> Its not just for Intel. The biggest reason for it to increase the
> > >>>> standardization of the boot flow across the PC ecosystem. We have
> > >>>> learned from experience that firmware is super critical to get a
> > >>>> product out the door but it is also difficult to write. So we try
> > >>>> to make it as reusable as humanly possible.
> > >>>>
> > >>>>> In fact, I don’t *know*, but I’m pretty sure the very strict
> > >>>>> separation between PEI and DXE was preserved mostly because MRC
> > >>>>> was 32-bit-only for a long time. Glad that seems to have been
> > >>>>> resolved, AMD does memory init by PSP nowadays.
> > >>>>
> > >>>> Having less complex early stages chain load more complex later
> > >>>> stages is a common design pattern in firmware, not just UEFI. For
> > >>>> example, your typical ARM system loads kinda like this:
> > >>>>
> > >>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
> > >>>> --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
> > >>>>
> > >>>> Comparing relative complexity I believe the Intel UEFI PI design is
> > >>>> actually pretty simple when you consider how much it gets done:
> > >>>>
> > >>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
> > >>>>
> > >>>> My biggest criticism of the design is that the strict separation
> > >>>> between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
> > >>>>
> > >>>> There are a few reasons why PEI was 32-bit for quite some time. The
> > >>>> biggest one is the code size increase, 64-bit x86 code is 20-30%
> > >>>> larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
> > >>>> access to is the cache onboard the processor and for security
> > >>>> reasons all that code has to fit within that RAM we generally do
> > >>>> everything we can to make that image as small as possible. Second,
> > >>>> 64-bit requires a page table and since we desired to keep PEI
> > >>>> simple we tried to avoid that. Finally, the PI spec didn't allow a
> > >>>> 64-bit PEI until recently. MRC is 32-bit code just because that is
> > >>>> what PEI happens to be. Porting it to 64-bit is not terribly
> > >>>> difficult.
> > >>>>
> > >>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any
> > >>>> data structures that get shared between PEI and DXE (HOBs, PCDs,
> > >>>> etc.) need to resolve to the same size and packing. LibraryClasses
> > >>>> need to be written to compile properly in both modes. In the case
> > >>>> of FSP API mode you need to resort to thunking between 32 and
> > >>>> 64-bit modes during DXE. More or less we decided that the costs are
> > >>>> starting to outweigh the advantages.
> > >>>>
> > >>>
> > >>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
> > >>> works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
> > >>>
> > >>> In the PC ecosystem a single chipset family can power thousands of
> > >>> unique designs. So the  DRAM memory needs to be external, support
> > >>> lots of different chipset packages(signal integrity...), support the
> > >>> lowest cost through the highest cost DRAM and thousands of different
> > >>> board layouts. So programing DRAM takes a masters degree in antenna
> > >>> design. I’ve seen MRC (Memory Reference Code) with over a MiB of
> > >>> DEBUG prints in it, and it literally is printing histograms of what
> > >>> it is tuning. So all this code has to run before the system has any
> > >>> DRAM, thus it is running using the cache as RAM. I’ve not looked at
> > >>> the x86 architecture specs form the vendors in a while, but back in
> > >>> the day they did not support page tables in ROM or pinned cached.
> > >>> Now it might work, but if it breaks your CPU vendor blames you so
> > >>> you don’t code PEI in X64….
> > >>>
> > >>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
> > >>> long time ago but I seem to remember the mask ROM used a table in
> > >>> NOR FLASH to init memory and then copied an image from NOR FLASH
> > >>> into DRAM and jumped to it. So PEI is kind of not really needed and
> > >>> we implemented a PrePEI and jumped directly to DXE.
> > >>>
> > >>> Given I was around back in the day when all this stuff was designed
> > >>> I can say SEC was always a place holder for security code, as
> > >>> security code always has to run 1st. PEI (Pre EFI) was designed to
> > >>> get DRAM programmed and then jump to DXE. It kind of also fell in
> > >>> naturally to ACPI S3 flow since that was turning memory back on.
> > >>> When we designed PEI we kind of though of it more like a boot loader
> > >>> stage for the firmware that turned on memory and all the work would
> > >>> happen in DXE. Then reality strikes and the existing BIOS assembly
> > >>> programmers start learning C (lots of  cranky people) and they start
> > >>> having to learn all about PEI to turn on memory. They had to write a
> > >>> big chunk of code for the memory init in PEI. These folks had never
> > >>> written any EFI code, so to them it was easier to move a lot of the
> > >>> chipset init code into PEI as that is the world they had to figure
> > >>> out to get memory turned on. I mean why learn EFI if you don’t have
> > >>> to? So that is how we get so much code in IA32 (i386) on some
> > >>> platforms. This start kind of biased future choices and how to
> > >>> enable non edk2 code bases….
> > >>
> > >> One of the big reasons a lot of code that should have been written in
> > >> DXE ended up in PEI is unfortunately due to the FSP and its inability
> > >> to support DXE code.
> > >>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Andrew Fish
> > >>>
> > >>>
> > >>>>>
> > >>>>> For many good reasons, Linux does not provide a stable kernel API.
> > >>>>> This
> > >>>>> allows to easily deploy breaking changes across the entire
> codebase.
> > >>>>> Obviously, this is infeasible at a large scale when you need to
> > >>>>> integrate IP blobs, but it would already help to break the
> > >>>>> expectation that UEFI PI is a perfectly forwards- and
> > >>>>> backwards-compatible system. DXE has SetMem and CopyMem as
> > part of
> > >>>>> gBS. Maybe I don’t like it that much as part of the spec itself,
> > >>>>> but it’s a good idea technically. I’d probably opt to have a
> > >>>>> quickly accessible implementation detail of important function
> > >>>>> pointers appended to PEI and DXE services, used by in-tree
> > >>>>> modules. This may break both
> > >>>>> forwards- and backwards-compatibility, but as it only applies to
> > >>>>> in-tree modules, that is fine so long as we let go of the
> > >>>>> compatibility notions. PPIs and protocols are an option too of
> > >>>>> course, but they have a lookup performance penalty. Compared to
> > >>>>> dynamic linking, that should hopefully be negligible however.
> > >>>>>
> > >>>>> Absolutely optional to read, I don’t intend to waste anyone’s time
> > >>>>> much, some philosophical stuff about my rationale:
> > >>>>>
> > >>>>> If you started UEFI from scratch right now, would it have strictly
> > >>>>> separated PEI and DXE?
> > >>>>
> > >>>> For sure a clean slate design started today would look a lot
> > >>>> different than PEI/DXE... which was a clean slate design circa 1998
> > >>>> 😊. In my opinion, if we were to suddenly go back to the drawing
> > >>>> board we would build something that is much closer to a full OS now
> > >>>> than we did back then. There have been cases where not being able
> > >>>> to use interrupt handlers and not having thread scheduling has
> > >>>> prevented implementation of desired features. The ARM guys built
> > >>>> LittleKernel (https://github.com/littlekernel/lk) for a lot of
> > >>>> these reasons. In the data center world some have decided to go to
> > >>>> the extreme of putting an entire copy of Linux in SPI so they can
> > >>>> do a network boot that downloads the OS image using BitTorrent!
> > >>>>
> > >>>>> The duplication between PEI and DXE core, and by extension MM
> > >>>>> core, would be my most obvious place to start reducing size. I
> > >>>>> would probably opt for a PEI/DXE hybrid where it starts in
> > >>>>> „minimal mode“ (maybe think of PEI more like a microkernel here)
> > >>>>> and after memory is initialised, the rest of DXE is loaded. And MM
> > >>>>> core would get no loading at all, this requirement has gladly been
> > >>>>> dropped ages ago. Just one prelinked snapshot of the address space
> > >>>>> with a relocation table and a safe self-relocator on entry (this
> > >>>>> is needed at the very least for ARM).
> > >>>>>
> > >>>>> Ironically, with my idea for MM, dynamic loading would be free as
> > >>>>> everything is prelinked anyway. The same is true for PEI XIP, it
> > >>>>> is prelinked by nature.
> > >>>>
> > >>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that
> > >>>> does get used for the PEI GOP driver.
> > >>>>
> > >>>>> What I do not like is the additional dynamic linking code at
> > >>>>> load-time for non- XIP modules. Though, the more I think about it,
> > >>>>> the more I wonder whether not the entirety of UEFI could be
> > >>>>> composed of prelinked, relocatable memory snapshots without
> > >>>>> traditional image loading entirely (for in-FW stuff). macOS has a
> > >>>>> similar concept with its “Kernel Collections”.
> > >>>>> Well, way
> > >>>>> too much off-topic now. :)
> > >>>>
> > >>>> If you make the assumption that 100% of the code is compiled all at
> > >>>> once then yes that works. UEFI was designed so that assumption does
> > >>>> not need to be true. There are good use cases for it: OpROMs,
> > >>>> generic OS loaders, network boot, etc.
> > >>>>
> > >>>> Thanks,
> > >>>> Nate
> > >>>>
> > >>>>>
> > >>>>> Why am I explaining all this despite the fact everyone knows this
> > >>>>> will never happen? Because I don’t like the notion of fixing
> > >>>>> issues of an already overcomplicated system by adding even more
> > >>>>> complicated stuff.
> > >>>>> Especially
> > >>>>> when the existing overcomplicated stuff is already uncomfortably
> > >>>>> broken.
> > >>>>>
> > >>>>> Best regards,
> > >>>>> Marvin
> > >>>>>
> > >>>>>> For XIP PEI code… it will really help and would be very timely
> > >>>>>> since the
> > >>>>> transition of PEI from 32-bit to 64-bit is going to increase the
> > >>>>> size of PEI by ~20%.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Nate
> > >>>>>>
> > >>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
> > >>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
> > >>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
> > >>>>>> <mhaeuser@posteo.de>
> > >>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
> > >>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
> > >>>>>> <steven.shi@intel.com>
> > >>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
> > >>>>>>
> > >>>>>> Hi Marvin, Ada,
> > >>>>>>
> > >>>>>> Some comments:
> > >>>>>>
> > >>>>>> I don't think the purpose of the dynamic linker is to treat EFI
> > >>>>>> as a complete operating system, but to try to eliminate the
> > >>>>>> static linking that may be needlessly duplicating code that could
> > >>>>>> instead be put in a
> > >>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are
> > >>>>> linked into a *lot* of .efi, instead of being just a library. It'd
> > >>>>> be nice to see some numbers on this (something like Google's
> > >>>>> bloaty could be run on every .efi file, in order to understand how
> > >>>>> much file space we would actually save).
> > >>>>>>
> > >>>>>> Other comments inline.
> > >>>>>>
> > >>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
> > >>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> > >>>>>>
> > >>>>>> On 13. Apr 2022, at 16:38, Ada Christine
> > >>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
> > wrote:
> > >>>>>> i was replying via the groups.io<http://groups.io > web
> > >>>>>> interface, I'm guessing that messed up the thread? i haven't used
> > >>>>>> mailing lists before and don't know how they work. I'll use my
> > >>>>>> mail client from here on.
> > >>>>>>
> > >>>>>> I'm on board with not treating EFI as an operating system. the
> > >>>>>> more i think about it the more it looks like scope creep.
> > >>>>>>
> > >>>>>> Agreed.
> > >>>>>>
> > >>>>>>
> > >>>>>> I'm not quite as enthusiastic
> > >>>>>> about it as i was at first glance.
> > >>>>>>
> > >>>>>> I'm still keen on doing my gsoc proposal for edk, though, and
> > >>>>>> even if this task and the acpica application are decided to be
> > >>>>>> out of scope unit testing,
> > >>>>>>
> > >>>>>> How about fuzz-testing? This is also something edk2 needs quite
> > >>>>>> badly. At
> > >>>>> Acidanthera, we compile edk2 code in userspace outside the edk2
> > >>>>> build system and fuzz with dummy applications.
> > >>>>>>
> > >>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
> > >>>>> https://llvm.org/docs/LibFuzzer.html ) and is something I could
> > >>>>> happily mentor.
> > >>>>>> clang integration
> > >>>>>>
> > >>>>>> Pedro and Vitaly are looking for someone to finish ASan:
> > >>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
> > >>>>>> There are working UBSan concepts, but they also need to be
> > >>>>>> mainlined.
> > >>>>>>
> > >>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be
> > >>>>>> me and
> > >>>>> some other, more senior, mentor (possibly Steven Shi, which I
> > >>>>> included in the task).
> > >>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and
> > >>>>>> possibly some other sanitizer it's quite possible that it could
> > >>>>>> be considered a large
> > >>>>> project (which means more hours but a larger stipend too). Fuzzing
> > >>>>> + coverage could be very nice additions to this project idea.
> > >>>>>> Also, is stress-testing a decent idea?
> > >>>>>>
> > >>>>>>
> > >>>>>> and source-level debugging are all relevant to my interests.
> > >>>>>>
> > >>>>>> how about your ideas for security stuff?
> > >>>>>>
> > >>>>>> I want the entirety of MM to leverage SmmMemLib and to support
> > SMAP.
> > >>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib
> > would
> > >>>>> only work on MMRAM. Also evaluation of how to best avoid pointers
> > >>>>> in MM communication buffers would be nice.
> > >>>>>>
> > >>>>>> There also is a bunch of other stuff, like working out moving a
> > >>>>>> part of
> > >>>>> CpuDxe into DxeCore to have memory protection live immediately,
> > >>>>> memory protection in PEI, a replacement for the TE format (it’s
> > >>>>> buggy and most platforms mostly abandoned it over various issues),
> > >>>>> and alternatives to guarding critical code with SMM (like allowing
> > >>>>> NVRAM commits only as part of a reboot).
> > >>>>>>
> > >>>>>> I personally find all of those projects very important, but I
> > >>>>>> cannot promise many people agree. Especially those that impose
> > >>>>>> global changes (most notably the TE replacement) may be very
> > >>>>>> tedious to submit.
> > >>>>>> Gladly, I believe you can submit multiple proposals (?)
> > >>>>>>
> > >>>>>> Best regards,
> > >>>>>> Marvin
> > >>>>>>
> > >>>>>>
> > >>>>>> I'm not very knowledgeable about
> > >>>>>> trusted platform or secure boot but I'm willing to learn whatever
> > >>>>>> is necessary to get something spun up for my proposal.
> > >>>>>>
> > >>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
> > >>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>> Do you use the “reply all” option in your mail client? Looks like
> > >>>>>> my CCs have been dropped again. Comments inline.
> > >>>>>>
> > >>>>>> On 13. Apr 2022, at 12:54, Ada Christine
> > >>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
> > >>>>>> wrote:
> > >>>>>> Hi, Marvin
> > >>>>>>
> > >>>>>> Its similarity to my own latest experiment is the key to what
> > >>>>>> grabbed my attention. I have no particular use case in mind for
> > >>>>>> it, but I see its potential for anybody developing larger
> > >>>>>> applications in that when a library is changed there's no need to
> > >>>>>> distribute a new version of the whole binary, just the relevant
> > >>>>>> library module.
> > >>>>>>
> > >>>>>> I really do not like the trend of treating UEFI as a full-fledged
> > >>>>>> OS - it is not. The most used UEFI applications, OS loaders, are
> > >>>>>> really not that huge and are distributed as part of the OS image
> > >>>>>> anyway.
> > >>>>>> Even for
> > >>>>>> less used applications, you will always get a full snapshot
> anyhow.
> > >>>>>> Gladly we don’t have auto-update and package management yet. :)
> > >>>>>>
> > >>>>>>
> > >>>>>> I slept on it and it occurred to me that the whole thing could
> > >>>>>> operate
> > >>>>>> similarly to the shell protocol in that the linker/loader is
> > >>>>>> itself an
> > >>>>>> application that does a LoadImage() on the application needing
> > >>>>>> dynamic
> > >>>>>> linking facilities.
> > >>>>>>
> > >>>>>> That would mean the linker itself is shipped with every
> application
> > >>>>>> that requires it? Otherwise it doesn’t make much sense for it to
> > >>>>>> be an
> > >>>>>> app and below’s problems apply.
> > >>>>>>
> > >>>>>> If however the whole plan is making the linker as a DXE and
> > >>>>>> including
> > >>>>>> it with the firmware, that I'm not quite as sure about. That would
> > >>>>>> necessarily tie any applications using dynamic linking to
> > >>>>>> TianoCore or
> > >>>>>> any firmware distribution that derives from it.
> > >>>>>>
> > >>>>>> I think that was the idea referred to as “edk2 core” by Steven,
> but
> > >>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
> > >>>>>> edk2, so that itself is not the problem, but versioning is.
> Vendors
> > >>>>>> are slow to update their snapshots or have just given up doing
> that
> > >>>>>> entirely. Distributing it for external applications like OS
> loaders
> > >>>>>> would mean this can be leveraged probably no earlier than 10 years
> > >>>>>> from now. And for in-firmware things, I have a hard time thinking
> > >>>>>> about a
> > >>>>> use-case that outweighs the drawbacks.
> > >>>>>>
> > >>>>>>
> > >>>>>> To shift the topic slightly back to GSoC, however, I'm willing to
> > >>>>>> work
> > >>>>>> on other items on the task list. Unit testing and an ACPICA
> > >>>>>> application are the alternative projects I had thought about. I
> need
> > >>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I
> > >>>>>> know
> > >>>>>> a tiny bit about porting ACPICA as I also have plans to
> > >>>>>> incorporate it into my
> > >>>>> own project.
> > >>>>>>
> > >>>>>> I have a few more ideas for security stuff, but Nate did not
> confirm
> > >>>>>> them as appropriate yet and I’m not here to drive you away from
> > this
> > >>>>>> specific task (or the others). However, I’m still curious and
> > >>>>>> concerned. :)
> > >>>>>>
> > >>>>>> Best regards,
> > >>>>>> Marvin
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Pedro Falcato
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > >
> > >
> > >
> > >
>
>
>
> 
>
>
>

[-- Attachment #2: Type: text/html, Size: 39571 bytes --]

Re: [edk2-devel] [edk2-discuss] GSoC Proposal

[Brian J. Johnson]

On 4/15/22 23:23, Desimone, Nathaniel L wrote:
> Hi Marvin,
> 
>> -----Original Message-----
>> From: Marvin Häuser <mhaeuser@posteo.de>
>> Sent: Friday, April 15, 2022 9:44 AM
>> To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@hpe.com>;
>> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Andrew Fish
>> <afish@apple.com>; devel@edk2.groups.io
>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; adachristine18@gmail.com;
>> Shi, Steven <steven.shi@intel.com>
>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>
>> Hey Brian,
>>
>> On 15.04.22 18:22, Brian J. Johnson wrote:
>>> Nate, Andrew, Marvin, Pedro, Ada, et al,
>>>
>>> This is a great discussion.  I've been debating where to weigh in...
>>
>> Personally, I don't think this topic can get enough attention. Thanks! :)
>>
>>>
>>> I agree that some sort of library sharing to reduce image size would
>>> be very helpful.  Although some parts of the BIOS are built
>>> separately, large parts of it are compiled at the same time, so there
>>> should be ample opportunity for tool-directed sharing.  Some form of
>>> pre-linking modules together may be the easiest way to do that.  But
>>> however it gets implemented, we should try to make it automatic:
>>> requiring library writers to manually add a lot of metadata or write
>>> thunks or shims would be a barrier to adoption.
>>
>> My suggestion to use a private function array would indeed require shims,
>> however this would be the "better than nothing I guess" solution where only
>> the most used and most painful to duplicate functions are shared, e.g.,
>> memory and maybe common string printing functions. This would quickly and
>> efficiently achieve the goal of "reduce size". And some form of sophisticated
>> prelinking would be my proposal for something more systematic, which
>> might come a lot cheaper with the security features to be added.
>>
>>>
>>> I also agree that PEI has acquired far too much functionality over
>>> time, and we badly need to find some way to reverse that.  PEI is
>>> pulling in more and more of the h/w initialization, leaving DXE mainly
>>> to provide the boot interface (ACPI, UEFI services, setup screens,
>>> etc.)  That puts more and more pressure on XIP storage and
>>> cache-as-RAM.  How can we encourage a change?
>>
>> I'm not in the loop enough to comment on the XIP and CAR issues.
>> However, HW init moving to PEI (or more generally, into a single stage) is not
>> necessarily a bad thing, is it? As I've said before, I'm not sure what else to
>> think of PEI than "DXE light". Both "initialise some of the hardware" with no
>> clear scope boundary but with similar interfaces. If they were sort of fused to
>> provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI"
>> without having a clear boundary between HW init steps, would that be too
>> bad? Now I'm *really* just blindly guessing because I barely looked into this
>> project, but isn't that somewhat the idea of SlimBootloader?
>>
>> Your description of DXE sort of is it collapsing into an advanced BDS.
>> Nate earlier said he'd like a clear distinction between DXE and BDS, which I
>> didn't really get the point of.
> 
> I guess what I'm trying to express is where does DXE end and where
> does BDS begin? A lot of drivers that are only used in BDS get
> dispatched early in DXE. Conversely, it is entirely possible for some
> DXE drivers to only run after we have technically entered the BDS
> phase. The line where DXE ends and BDS begins is blurry and not
> really a line but more like a rectangle.
> 
> To Isaac's point, there is a good argument to be made for beefing up
> PEI's capabilities to the point that DXE and BDS more or less become
> one thing. But what does that mean for PEI? Do we change from the
> 1-D PPI database to the 2-D Handle database in order to express
> device topologies better for example? That would be useful for stuff
> like the PEI UFS drivers as they run into difficulties expressing
> LUN IDs that the DXE version does not for example. Not sure if I
> would actually formally propose that in the PI WG but interesting
> stuff to think about regardless.
 >

Andrew pointed out that PEI was originally intended for memory init, and 
DXE for the rest.  One nice aspect of that is that there's a simple, 
architected, consolidated handoff of state between them:  the HOB list. 
That makes it easier to do "unusual" things in SEC+PEI (special 
security, address map changes, etc.) but have it all hidden from DXE, 
where the majority of "standard" code lives.  The trend of moving more 
and more into PEI is eroding that advantage.

Maybe a similar handoff of state is needed between pre-memory and 
post-memory PEI?  Or between DXE and BDS?

One could argue that we already have that for BDS:  the UEFI boot services.

>> But if PEI consumed the HW init portions of DXE,
>> a more advanced BDS (which also powers e.g. the HID stuff on the way)
>> would be the logical remainder stage to separately support UEFI booting as a
>> payload (for SlimBootloader, coreboot, ...).
>>
>> All of that was logically speaking. Even if the development was driven to its
>> extreme, PEI and DXE cores would still share a bunch of code and it might
>> make sense to share a foundation technically.
>>
>> Best regards,
>> Marvin
>>
>>>
>>> Thanks,
>>> Brian J. Johnson
>>>
>>>
>>> On 4/15/22 03:15, Nate DeSimone wrote:
>>>> Hi Andrew,
>>>>
>>>> On 4/14/22, 7:43 PM, "Andrew Fish" <afish@apple.com> wrote:
>>>>>
>>>>>
>>>>>> On Apr 14, 2022, at 6:06 PM, Nate DeSimone
>>>>>> <nathaniel.l.desimone@intel.com> wrote:
>>>>>>
>>>>>> Hi Marvin,
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
>>>>>>> Marvin Häuser
>>>>>>> Sent: Thursday, April 14, 2022 12:56 AM
>>>>>>> To: discuss@edk2.groups.io; Desimone, Nathaniel L
>>>>>>> <nathaniel.l.desimone@intel.com>
>>>>>>> Cc: Pedro Falcato <pedro.falcato@gmail.com>; edk2-devel-groups-io
>>>>>>> <devel@edk2.groups.io>; adachristine18@gmail.com; Shi, Steven
>>>>>>> <steven.shi@intel.com>
>>>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>>>
>>>>>>> I feel like there are so many much easier solutions to this
>>>>>>> problem that are at most limited by the clear specification. The
>>>>>>> UEFI specification with regards to booting and all of that
>>>>>>> obviously is of utmost importance.
>>>>>>
>>>>>> If you have a better idea that retains compatibility with the
>>>>>> existing UEFI PI then I would be happy to hear it. Ultimately
>>>>>> anything we do needs to be a pure extension that retains
>>>>>> compatibility with old code. Given that restriction having the
>>>>>> ability to coalesce all the LibraryClasses into a single module and
>>>>>> have cross-module linking seems like the best way to handle it to me.
>>>>>>
>>>>>>> The UEFI PI specification parts that deal about internal
>>>>>>> structure, as far as I know, are only in place to make it easy to
>>>>>>> integrate Intel IP.
>>>>>>
>>>>>> Its not just for Intel. The biggest reason for it to increase the
>>>>>> standardization of the boot flow across the PC ecosystem. We have
>>>>>> learned from experience that firmware is super critical to get a
>>>>>> product out the door but it is also difficult to write. So we try
>>>>>> to make it as reusable as humanly possible.
>>>>>>
>>>>>>> In fact, I don’t *know*, but I’m pretty sure the very strict
>>>>>>> separation between PEI and DXE was preserved mostly because MRC
>>>>>>> was 32-bit-only for a long time. Glad that seems to have been
>>>>>>> resolved, AMD does memory init by PSP nowadays.
>>>>>>
>>>>>> Having less complex early stages chain load more complex later
>>>>>> stages is a common design pattern in firmware, not just UEFI. For
>>>>>> example, your typical ARM system loads kinda like this:
>>>>>>
>>>>>> PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
>>>>>> --> EDK II or U-Boot or LittleKernel (which runs android fastboot)
>>>>>>
>>>>>> Comparing relative complexity I believe the Intel UEFI PI design is
>>>>>> actually pretty simple when you consider how much it gets done:
>>>>>>
>>>>>> Ucode ROM --> SEC + PEI --> DXE + SMM + BDS
>>>>>>
>>>>>> My biggest criticism of the design is that the strict separation
>>>>>> between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊
>>>>>>
>>>>>> There are a few reasons why PEI was 32-bit for quite some time. The
>>>>>> biggest one is the code size increase, 64-bit x86 code is 20-30%
>>>>>> larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
>>>>>> access to is the cache onboard the processor and for security
>>>>>> reasons all that code has to fit within that RAM we generally do
>>>>>> everything we can to make that image as small as possible. Second,
>>>>>> 64-bit requires a page table and since we desired to keep PEI
>>>>>> simple we tried to avoid that. Finally, the PI spec didn't allow a
>>>>>> 64-bit PEI until recently. MRC is 32-bit code just because that is
>>>>>> what PEI happens to be. Porting it to 64-bit is not terribly
>>>>>> difficult.
>>>>>>
>>>>>> Ultimately the mix of 32/64-bit does cause some difficulties. Any
>>>>>> data structures that get shared between PEI and DXE (HOBs, PCDs,
>>>>>> etc.) need to resolve to the same size and packing. LibraryClasses
>>>>>> need to be written to compile properly in both modes. In the case
>>>>>> of FSP API mode you need to resort to thunking between 32 and
>>>>>> 64-bit modes during DXE. More or less we decided that the costs are
>>>>>> starting to outweigh the advantages.
>>>>>>
>>>>>
>>>>> I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
>>>>> works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.
>>>>>
>>>>> In the PC ecosystem a single chipset family can power thousands of
>>>>> unique designs. So the  DRAM memory needs to be external, support
>>>>> lots of different chipset packages(signal integrity...), support the
>>>>> lowest cost through the highest cost DRAM and thousands of different
>>>>> board layouts. So programing DRAM takes a masters degree in antenna
>>>>> design. I’ve seen MRC (Memory Reference Code) with over a MiB of
>>>>> DEBUG prints in it, and it literally is printing histograms of what
>>>>> it is tuning. So all this code has to run before the system has any
>>>>> DRAM, thus it is running using the cache as RAM. I’ve not looked at
>>>>> the x86 architecture specs form the vendors in a while, but back in
>>>>> the day they did not support page tables in ROM or pinned cached.
>>>>> Now it might work, but if it breaks your CPU vendor blames you so
>>>>> you don’t code PEI in X64….
>>>>>
>>>>> We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
>>>>> long time ago but I seem to remember the mask ROM used a table in
>>>>> NOR FLASH to init memory and then copied an image from NOR FLASH
>>>>> into DRAM and jumped to it. So PEI is kind of not really needed and
>>>>> we implemented a PrePEI and jumped directly to DXE.
>>>>>
>>>>> Given I was around back in the day when all this stuff was designed
>>>>> I can say SEC was always a place holder for security code, as
>>>>> security code always has to run 1st. PEI (Pre EFI) was designed to
>>>>> get DRAM programmed and then jump to DXE. It kind of also fell in
>>>>> naturally to ACPI S3 flow since that was turning memory back on.
>>>>> When we designed PEI we kind of though of it more like a boot loader
>>>>> stage for the firmware that turned on memory and all the work would
>>>>> happen in DXE. Then reality strikes and the existing BIOS assembly
>>>>> programmers start learning C (lots of  cranky people) and they start
>>>>> having to learn all about PEI to turn on memory. They had to write a
>>>>> big chunk of code for the memory init in PEI. These folks had never
>>>>> written any EFI code, so to them it was easier to move a lot of the
>>>>> chipset init code into PEI as that is the world they had to figure
>>>>> out to get memory turned on. I mean why learn EFI if you don’t have
>>>>> to? So that is how we get so much code in IA32 (i386) on some
>>>>> platforms. This start kind of biased future choices and how to
>>>>> enable non edk2 code bases….
>>>>
>>>> One of the big reasons a lot of code that should have been written in
>>>> DXE ended up in PEI is unfortunately due to the FSP and its inability
>>>> to support DXE code.
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Andrew Fish
>>>>>
>>>>>
>>>>>>>
>>>>>>> For many good reasons, Linux does not provide a stable kernel API.
>>>>>>> This
>>>>>>> allows to easily deploy breaking changes across the entire codebase.
>>>>>>> Obviously, this is infeasible at a large scale when you need to
>>>>>>> integrate IP blobs, but it would already help to break the
>>>>>>> expectation that UEFI PI is a perfectly forwards- and
>>>>>>> backwards-compatible system. DXE has SetMem and CopyMem as
>> part of
>>>>>>> gBS. Maybe I don’t like it that much as part of the spec itself,
>>>>>>> but it’s a good idea technically. I’d probably opt to have a
>>>>>>> quickly accessible implementation detail of important function
>>>>>>> pointers appended to PEI and DXE services, used by in-tree
>>>>>>> modules. This may break both
>>>>>>> forwards- and backwards-compatibility, but as it only applies to
>>>>>>> in-tree modules, that is fine so long as we let go of the
>>>>>>> compatibility notions. PPIs and protocols are an option too of
>>>>>>> course, but they have a lookup performance penalty. Compared to
>>>>>>> dynamic linking, that should hopefully be negligible however.
>>>>>>>
>>>>>>> Absolutely optional to read, I don’t intend to waste anyone’s time
>>>>>>> much, some philosophical stuff about my rationale:
>>>>>>>
>>>>>>> If you started UEFI from scratch right now, would it have strictly
>>>>>>> separated PEI and DXE?
>>>>>>
>>>>>> For sure a clean slate design started today would look a lot
>>>>>> different than PEI/DXE... which was a clean slate design circa 1998
>>>>>> 😊. In my opinion, if we were to suddenly go back to the drawing
>>>>>> board we would build something that is much closer to a full OS now
>>>>>> than we did back then. There have been cases where not being able
>>>>>> to use interrupt handlers and not having thread scheduling has
>>>>>> prevented implementation of desired features. The ARM guys built
>>>>>> LittleKernel (https://github.com/littlekernel/lk) for a lot of
>>>>>> these reasons. In the data center world some have decided to go to
>>>>>> the extreme of putting an entire copy of Linux in SPI so they can
>>>>>> do a network boot that downloads the OS image using BitTorrent!
>>>>>>
>>>>>>> The duplication between PEI and DXE core, and by extension MM
>>>>>>> core, would be my most obvious place to start reducing size. I
>>>>>>> would probably opt for a PEI/DXE hybrid where it starts in
>>>>>>> „minimal mode“ (maybe think of PEI more like a microkernel here)
>>>>>>> and after memory is initialised, the rest of DXE is loaded. And MM
>>>>>>> core would get no loading at all, this requirement has gladly been
>>>>>>> dropped ages ago. Just one prelinked snapshot of the address space
>>>>>>> with a relocation table and a safe self-relocator on entry (this
>>>>>>> is needed at the very least for ARM).
>>>>>>>
>>>>>>> Ironically, with my idea for MM, dynamic loading would be free as
>>>>>>> everything is prelinked anyway. The same is true for PEI XIP, it
>>>>>>> is prelinked by nature.
>>>>>>
>>>>>> Actually Post-Memory PEI can have non-prelinked PEIMs. And that
>>>>>> does get used for the PEI GOP driver.
>>>>>>
>>>>>>> What I do not like is the additional dynamic linking code at
>>>>>>> load-time for non- XIP modules. Though, the more I think about it,
>>>>>>> the more I wonder whether not the entirety of UEFI could be
>>>>>>> composed of prelinked, relocatable memory snapshots without
>>>>>>> traditional image loading entirely (for in-FW stuff). macOS has a
>>>>>>> similar concept with its “Kernel Collections”.
>>>>>>> Well, way
>>>>>>> too much off-topic now. :)
>>>>>>
>>>>>> If you make the assumption that 100% of the code is compiled all at
>>>>>> once then yes that works. UEFI was designed so that assumption does
>>>>>> not need to be true. There are good use cases for it: OpROMs,
>>>>>> generic OS loaders, network boot, etc.
>>>>>>
>>>>>> Thanks,
>>>>>> Nate
>>>>>>
>>>>>>>
>>>>>>> Why am I explaining all this despite the fact everyone knows this
>>>>>>> will never happen? Because I don’t like the notion of fixing
>>>>>>> issues of an already overcomplicated system by adding even more
>>>>>>> complicated stuff.
>>>>>>> Especially
>>>>>>> when the existing overcomplicated stuff is already uncomfortably
>>>>>>> broken.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Marvin
>>>>>>>
>>>>>>>> For XIP PEI code… it will really help and would be very timely
>>>>>>>> since the
>>>>>>> transition of PEI from 32-bit to 64-bit is going to increase the
>>>>>>> size of PEI by ~20%.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Nate
>>>>>>>>
>>>>>>>> From: Pedro Falcato <pedro.falcato@gmail.com>
>>>>>>>> Sent: Wednesday, April 13, 2022 11:43 AM
>>>>>>>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
>>>>>>>> <mhaeuser@posteo.de>
>>>>>>>> Cc: discuss@edk2.groups.io; adachristine18@gmail.com; Desimone,
>>>>>>>> Nathaniel L <nathaniel.l.desimone@intel.com>; Shi, Steven
>>>>>>>> <steven.shi@intel.com>
>>>>>>>> Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal
>>>>>>>>
>>>>>>>> Hi Marvin, Ada,
>>>>>>>>
>>>>>>>> Some comments:
>>>>>>>>
>>>>>>>> I don't think the purpose of the dynamic linker is to treat EFI
>>>>>>>> as a complete operating system, but to try to eliminate the
>>>>>>>> static linking that may be needlessly duplicating code that could
>>>>>>>> instead be put in a
>>>>>>> single dynamic library. For instance, MdePkg and MdeModulePkg are
>>>>>>> linked into a *lot* of .efi, instead of being just a library. It'd
>>>>>>> be nice to see some numbers on this (something like Google's
>>>>>>> bloaty could be run on every .efi file, in order to understand how
>>>>>>> much file space we would actually save).
>>>>>>>>
>>>>>>>> Other comments inline.
>>>>>>>>
>>>>>>>> On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
>>>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>>>
>>>>>>>> On 13. Apr 2022, at 16:38, Ada Christine
>>>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>> wrote:
>>>>>>>> i was replying via the groups.io<http://groups.io  > web
>>>>>>>> interface, I'm guessing that messed up the thread? i haven't used
>>>>>>>> mailing lists before and don't know how they work. I'll use my
>>>>>>>> mail client from here on.
>>>>>>>>
>>>>>>>> I'm on board with not treating EFI as an operating system. the
>>>>>>>> more i think about it the more it looks like scope creep.
>>>>>>>>
>>>>>>>> Agreed.
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not quite as enthusiastic
>>>>>>>> about it as i was at first glance.
>>>>>>>>
>>>>>>>> I'm still keen on doing my gsoc proposal for edk, though, and
>>>>>>>> even if this task and the acpica application are decided to be
>>>>>>>> out of scope unit testing,
>>>>>>>>
>>>>>>>> How about fuzz-testing? This is also something edk2 needs quite
>>>>>>>> badly. At
>>>>>>> Acidanthera, we compile edk2 code in userspace outside the edk2
>>>>>>> build system and fuzz with dummy applications.
>>>>>>>>
>>>>>>>> Note: fuzzing is also part of the LLVM instrumentation suite (see
>>>>>>> https://llvm.org/docs/LibFuzzer.html  ) and is something I could
>>>>>>> happily mentor.
>>>>>>>> clang integration
>>>>>>>>
>>>>>>>> Pedro and Vitaly are looking for someone to finish ASan:
>>>>>>>> https://edk2.groups.io/g/devel/topic/90010978#87991
>>>>>>>> There are working UBSan concepts, but they also need to be
>>>>>>>> mainlined.
>>>>>>>>
>>>>>>>> Is Vitaly going to be a mentor? I was assuming it was going to be
>>>>>>>> me and
>>>>>>> some other, more senior, mentor (possibly Steven Shi, which I
>>>>>>> included in the task).
>>>>>>>> Anyway, re: ASAN, if the project includes ASAN, UBSAN and
>>>>>>>> possibly some other sanitizer it's quite possible that it could
>>>>>>>> be considered a large
>>>>>>> project (which means more hours but a larger stipend too). Fuzzing
>>>>>>> + coverage could be very nice additions to this project idea.
>>>>>>>> Also, is stress-testing a decent idea?
>>>>>>>>
>>>>>>>>
>>>>>>>> and source-level debugging are all relevant to my interests.
>>>>>>>>
>>>>>>>> how about your ideas for security stuff?
>>>>>>>>
>>>>>>>> I want the entirety of MM to leverage SmmMemLib and to support
>> SMAP.
>>>>>>> SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib
>> would
>>>>>>> only work on MMRAM. Also evaluation of how to best avoid pointers
>>>>>>> in MM communication buffers would be nice.
>>>>>>>>
>>>>>>>> There also is a bunch of other stuff, like working out moving a
>>>>>>>> part of
>>>>>>> CpuDxe into DxeCore to have memory protection live immediately,
>>>>>>> memory protection in PEI, a replacement for the TE format (it’s
>>>>>>> buggy and most platforms mostly abandoned it over various issues),
>>>>>>> and alternatives to guarding critical code with SMM (like allowing
>>>>>>> NVRAM commits only as part of a reboot).
>>>>>>>>
>>>>>>>> I personally find all of those projects very important, but I
>>>>>>>> cannot promise many people agree. Especially those that impose
>>>>>>>> global changes (most notably the TE replacement) may be very
>>>>>>>> tedious to submit.
>>>>>>>> Gladly, I believe you can submit multiple proposals (?)
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Marvin
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not very knowledgeable about
>>>>>>>> trusted platform or secure boot but I'm willing to learn whatever
>>>>>>>> is necessary to get something spun up for my proposal.
>>>>>>>>
>>>>>>>> On Wed, Apr 13, 2022, 12:05 Marvin Häuser
>>>>>>> <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Do you use the “reply all” option in your mail client? Looks like
>>>>>>>> my CCs have been dropped again. Comments inline.
>>>>>>>>
>>>>>>>> On 13. Apr 2022, at 12:54, Ada Christine
>>>>>>>> <adachristine18@gmail.com<mailto:adachristine18@gmail.com>>
>>>>>>>> wrote:
>>>>>>>> Hi, Marvin
>>>>>>>>
>>>>>>>> Its similarity to my own latest experiment is the key to what
>>>>>>>> grabbed my attention. I have no particular use case in mind for
>>>>>>>> it, but I see its potential for anybody developing larger
>>>>>>>> applications in that when a library is changed there's no need to
>>>>>>>> distribute a new version of the whole binary, just the relevant
>>>>>>>> library module.
>>>>>>>>
>>>>>>>> I really do not like the trend of treating UEFI as a full-fledged
>>>>>>>> OS - it is not. The most used UEFI applications, OS loaders, are
>>>>>>>> really not that huge and are distributed as part of the OS image
>>>>>>>> anyway.
>>>>>>>> Even for
>>>>>>>> less used applications, you will always get a full snapshot anyhow.
>>>>>>>> Gladly we don’t have auto-update and package management yet. :)
>>>>>>>>
>>>>>>>>
>>>>>>>> I slept on it and it occurred to me that the whole thing could
>>>>>>>> operate
>>>>>>>> similarly to the shell protocol in that the linker/loader is
>>>>>>>> itself an
>>>>>>>> application that does a LoadImage() on the application needing
>>>>>>>> dynamic
>>>>>>>> linking facilities.
>>>>>>>>
>>>>>>>> That would mean the linker itself is shipped with every application
>>>>>>>> that requires it? Otherwise it doesn’t make much sense for it to
>>>>>>>> be an
>>>>>>>> app and below’s problems apply.
>>>>>>>>
>>>>>>>> If however the whole plan is making the linker as a DXE and
>>>>>>>> including
>>>>>>>> it with the firmware, that I'm not quite as sure about. That would
>>>>>>>> necessarily tie any applications using dynamic linking to
>>>>>>>> TianoCore or
>>>>>>>> any firmware distribution that derives from it.
>>>>>>>>
>>>>>>>> I think that was the idea referred to as “edk2 core” by Steven, but
>>>>>>>> I’d like to hear his proposal to be sure. Virtually everyone uses
>>>>>>>> edk2, so that itself is not the problem, but versioning is. Vendors
>>>>>>>> are slow to update their snapshots or have just given up doing that
>>>>>>>> entirely. Distributing it for external applications like OS loaders
>>>>>>>> would mean this can be leveraged probably no earlier than 10 years
>>>>>>>> from now. And for in-firmware things, I have a hard time thinking
>>>>>>>> about a
>>>>>>> use-case that outweighs the drawbacks.
>>>>>>>>
>>>>>>>>
>>>>>>>> To shift the topic slightly back to GSoC, however, I'm willing to
>>>>>>>> work
>>>>>>>> on other items on the task list. Unit testing and an ACPICA
>>>>>>>> application are the alternative projects I had thought about. I need
>>>>>>>> to choose fairly soon as the proposal deadline is next Tuesday. I
>>>>>>>> know
>>>>>>>> a tiny bit about porting ACPICA as I also have plans to
>>>>>>>> incorporate it into my
>>>>>>> own project.
>>>>>>>>
>>>>>>>> I have a few more ideas for security stuff, but Nate did not confirm
>>>>>>>> them as appropriate yet and I’m not here to drive you away from
>> this
>>>>>>>> specific task (or the others). However, I’m still curious and
>>>>>>>> concerned. :)
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Marvin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Pedro Falcato
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> 
>>>
>>>
>