RFC: Universal Payload Interface

RFC: Universal Payload Interface

[Ni, Ray]

[-- Attachment #1: Type: text/plain, Size: 1925 bytes --]

With the fact that there are many different firmware implementations, we tried to decouple today's monolithic UEFI firmware binary to two independent components: bootloader and payload.

Basically, bootloader initializes the silicon hardware and payload prepares the OS required data and services. Bootloader passes control to payload.

https://universalpayload.github.io/documentation/spec/spec.html defines the universal interface between bootloader and payload. So that different bootloaders can work with different payloads, initializing different hardware and booting different OSes.
The interface document is in very draft phase. Any feedback is welcome.

We also developed the POC code to demonstrate the idea. Please use below steps to get the code that uses SBL as the bootloader and EDKII UEFI Payload as the payload. This POC is being developed for QEMU Q35 virtual machine.

  1.  Run "git clone https://github.com/universalpayload/tools.git payload"

This step downloads the initial tools that will setup the dev environment.

  1.  Run "py -3 clone_and_build_sbl_with_uefipayload.py" in the "payload" directory

This script downloads branched SBL (https://github.com/universalpayload/slimbootloader.git) and edk2 (https://github.com/universalpayload/edk2.git).

Then it builds the firmware binary "SlimBootloader.bin" in "codeworkspace" directory.

  1.  Boot QEMU by running "qemu-system-x86_64.exe -machine q35 -pflash  codeworkspace\SlimBootloader.bin  -serial file:test.log"

Because the code is under active development, please contact us when you cannot build or boot successfully.

Besides the SBL, we modified coreboot (https://github.com/universalpayload/coreboot.git) to let it conform to the universal interface as a bootloader.
Besides the EDK2 UEFI Payload, we created a payload (https://github.com/universalpayload/linuxpayload.git) that can boot Linux.

Thanks,
Ray


[-- Attachment #2: Type: text/html, Size: 6912 bytes --]

Re: [edk2-devel] RFC: Universal Payload Interface

[Laszlo Ersek]

On 10/23/20 03:18, Ni, Ray wrote:
> With the fact that there are many different firmware implementations, we tried to decouple today's monolithic UEFI firmware binary to two independent components: bootloader and payload.

(1) "Bootloader" is an extremely loaded word. Regardless of everything
else in this topic, I strongly suggest picking a different name.

We already have "Platform Init" or "PI"; maybe use "Silicon Init" or "SI"?


(2) What is the *exact* use case (or workflow) that the proposed
interface enables, or improves?

What groups of people (what roles) are supposed to benefit from the
proposed interface?

Thanks
Laszlo

> 
> Basically, bootloader initializes the silicon hardware and payload prepares the OS required data and services. Bootloader passes control to payload.
> 
> https://universalpayload.github.io/documentation/spec/spec.html defines the universal interface between bootloader and payload. So that different bootloaders can work with different payloads, initializing different hardware and booting different OSes.
> The interface document is in very draft phase. Any feedback is welcome.
> 
> We also developed the POC code to demonstrate the idea. Please use below steps to get the code that uses SBL as the bootloader and EDKII UEFI Payload as the payload. This POC is being developed for QEMU Q35 virtual machine.
> 
>   1.  Run "git clone https://github.com/universalpayload/tools.git payload"
> 
> This step downloads the initial tools that will setup the dev environment.
> 
>   1.  Run "py -3 clone_and_build_sbl_with_uefipayload.py" in the "payload" directory
> 
> This script downloads branched SBL (https://github.com/universalpayload/slimbootloader.git) and edk2 (https://github.com/universalpayload/edk2.git).
> 
> Then it builds the firmware binary "SlimBootloader.bin" in "codeworkspace" directory.
> 
>   1.  Boot QEMU by running "qemu-system-x86_64.exe -machine q35 -pflash  codeworkspace\SlimBootloader.bin  -serial file:test.log"
> 
> Because the code is under active development, please contact us when you cannot build or boot successfully.
> 
> Besides the SBL, we modified coreboot (https://github.com/universalpayload/coreboot.git) to let it conform to the universal interface as a bootloader.
> Besides the EDK2 UEFI Payload, we created a payload (https://github.com/universalpayload/linuxpayload.git) that can boot Linux.
> 
> Thanks,
> Ray
> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] RFC: Universal Payload Interface

[Ni, Ray]

Laszlo,
Thank you for the comments.
Reply inline.

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo
> Ersek
> Sent: Tuesday, October 27, 2020 10:21 PM
> To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>
> Cc: Zimmer, Vincent <vincent.zimmer@intel.com>; Ma, Maurice
> <maurice.ma@intel.com>; Rangarajan, Ravi P <ravi.p.rangarajan@intel.com>;
> Dong, Guo <guo.dong@intel.com>; Hau, Tze-ming <tze-ming.hau@intel.com>
> Subject: Re: [edk2-devel] RFC: Universal Payload Interface
> 
> On 10/23/20 03:18, Ni, Ray wrote:
> > With the fact that there are many different firmware implementations, we
> tried to decouple today's monolithic UEFI firmware binary to two independent
> components: bootloader and payload.
> 
> (1) "Bootloader" is an extremely loaded word. Regardless of everything
> else in this topic, I strongly suggest picking a different name.
> 
> We already have "Platform Init" or "PI"; maybe use "Silicon Init" or "SI"?
I am a UEFI guy for many years. So your suggestion "PI" looks very friendly to me.
But I am not sure if the audiences include broader people, like coreboot, SBL developers,
do they like the names.

Personally I am open to any name as long as the concept is not changed: the binary blob
is responsible to initialize the silicon.


> 
> 
> (2) What is the *exact* use case (or workflow) that the proposed
> interface enables, or improves?
> What groups of people (what roles) are supposed to benefit from the
> proposed interface?

1. Unified UEFI Payload Binary.
By standardizing the bootloader->payload interface, we keep in mind to move all
platform/silicon specific implementation to bootloader and all the specific info is
passed to payload through the standard interface such that the payload doesn't need
to deal with concrete hardware but just the abstracted info. It gives possibility to create
the unified UEFI payload binary that can run in any platform/silicon (off course, one binary
per one CPU arch). Just like today's UEFI Shell.
It's a huge save on validation side to every company that uses UEFI as boot solution for
hardware.
People may argue maintaining such a binary causes additional overhead. I agree.
But I am optimistic on such direction. 
(The code will be still in open source.)

2. Bootloaders
It shows an attitude of EDKII community that it doesn't restrict to use EDKII PEI as the
only acknowledged silicon code execution environment. The standard interface as a promise
allows any compliant bootloader to work with EDKII UEFI Payload.

3. Payloads
I saw different tries to change EDKII DXE environment for different hardware/OS.
It may cause defragmentation of UEFI spec. The standard interface also allows any
compliant payload to be created.

> 
> Thanks
> Laszlo
> 
> >
> > Basically, bootloader initializes the silicon hardware and payload prepares
> the OS required data and services. Bootloader passes control to payload.
> >
> > https://universalpayload.github.io/documentation/spec/spec.html defines the
> universal interface between bootloader and payload. So that different
> bootloaders can work with different payloads, initializing different hardware
> and booting different OSes.
> > The interface document is in very draft phase. Any feedback is welcome.
> >
> > We also developed the POC code to demonstrate the idea. Please use below
> steps to get the code that uses SBL as the bootloader and EDKII UEFI Payload as
> the payload. This POC is being developed for QEMU Q35 virtual machine.
> >
> >   1.  Run "git clone https://github.com/universalpayload/tools.git payload"
> >
> > This step downloads the initial tools that will setup the dev environment.
> >
> >   1.  Run "py -3 clone_and_build_sbl_with_uefipayload.py" in the "payload"
> directory
> >
> > This script downloads branched SBL
> (https://github.com/universalpayload/slimbootloader.git) and edk2
> (https://github.com/universalpayload/edk2.git).
> >
> > Then it builds the firmware binary "SlimBootloader.bin" in "codeworkspace"
> directory.
> >
> >   1.  Boot QEMU by running "qemu-system-x86_64.exe -machine q35 -pflash
> codeworkspace\SlimBootloader.bin  -serial file:test.log"
> >
> > Because the code is under active development, please contact us when you
> cannot build or boot successfully.
> >
> > Besides the SBL, we modified coreboot
> (https://github.com/universalpayload/coreboot.git) to let it conform to the
> universal interface as a bootloader.
> > Besides the EDK2 UEFI Payload, we created a payload
> (https://github.com/universalpayload/linuxpayload.git) that can boot Linux.
> >
> > Thanks,
> > Ray
> >
> >
> >
> >
> >
> >
> >
> 
> 
> 
> 
> 

Re: RFC: Universal Payload Interface

[Carsey, Jaben]

[-- Attachment #1: Type: text/plain, Size: 2500 bytes --]

Ray

Is there any more information on "Payload Image Extend Header"? This is mentioned, but I don't see detailed information.

-Jaben

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ni, Ray
Sent: Thursday, October 22, 2020 6:18 PM
To: devel@edk2.groups.io
Cc: Zimmer, Vincent <vincent.zimmer@intel.com>; Ma, Maurice <maurice.ma@intel.com>; Rangarajan, Ravi P <ravi.p.rangarajan@intel.com>; Dong, Guo <guo.dong@intel.com>; Hau, Tze-ming <tze-ming.hau@intel.com>; Ni, Ray <ray.ni@intel.com>
Subject: [edk2-devel] RFC: Universal Payload Interface

With the fact that there are many different firmware implementations, we tried to decouple today's monolithic UEFI firmware binary to two independent components: bootloader and payload.

Basically, bootloader initializes the silicon hardware and payload prepares the OS required data and services. Bootloader passes control to payload.

https://universalpayload.github.io/documentation/spec/spec.html defines the universal interface between bootloader and payload. So that different bootloaders can work with different payloads, initializing different hardware and booting different OSes.
The interface document is in very draft phase. Any feedback is welcome.

We also developed the POC code to demonstrate the idea. Please use below steps to get the code that uses SBL as the bootloader and EDKII UEFI Payload as the payload. This POC is being developed for QEMU Q35 virtual machine.

  1.  Run "git clone https://github.com/universalpayload/tools.git payload"

This step downloads the initial tools that will setup the dev environment.

  1.  Run "py -3 clone_and_build_sbl_with_uefipayload.py" in the "payload" directory

This script downloads branched SBL (https://github.com/universalpayload/slimbootloader.git) and edk2 (https://github.com/universalpayload/edk2.git).

Then it builds the firmware binary "SlimBootloader.bin" in "codeworkspace" directory.

  1.  Boot QEMU by running "qemu-system-x86_64.exe -machine q35 -pflash  codeworkspace\SlimBootloader.bin  -serial file:test.log"

Because the code is under active development, please contact us when you cannot build or boot successfully.

Besides the SBL, we modified coreboot (https://github.com/universalpayload/coreboot.git) to let it conform to the universal interface as a bootloader.
Besides the EDK2 UEFI Payload, we created a payload (https://github.com/universalpayload/linuxpayload.git) that can boot Linux.

Thanks,
Ray



[-- Attachment #2: Type: text/html, Size: 8751 bytes --]

Re: RFC: Universal Payload Interface

[Guo Dong]

[-- Attachment #1: Type: text/plain, Size: 3543 bytes --]


Hi Jaben,

I think the "Payload Image Extend Header" means the optional tables below in Payload Image Format<https://universalpayload.github.io/documentation/spec/spec.html#payload-image-format>.
These tables exist only when corresponding bit is set in Capability field.

  *   Payload Image Relocation Table
  *   Payload Image Authentication Table

Thanks,
Guo

From: Carsey, Jaben <jaben.carsey@intel.com>
Sent: Wednesday, October 28, 2020 1:41 PM
To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>
Cc: Zimmer, Vincent <vincent.zimmer@intel.com>; Ma, Maurice <maurice.ma@intel.com>; Rangarajan, Ravi P <ravi.p.rangarajan@intel.com>; Dong, Guo <guo.dong@intel.com>; Hau, Tze-ming <tze-ming.hau@intel.com>
Subject: RE: RFC: Universal Payload Interface

Ray

Is there any more information on "Payload Image Extend Header"? This is mentioned, but I don't see detailed information.

-Jaben

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray
Sent: Thursday, October 22, 2020 6:18 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Zimmer, Vincent <vincent.zimmer@intel.com<mailto:vincent.zimmer@intel.com>>; Ma, Maurice <maurice.ma@intel.com<mailto:maurice.ma@intel.com>>; Rangarajan, Ravi P <ravi.p.rangarajan@intel.com<mailto:ravi.p.rangarajan@intel.com>>; Dong, Guo <guo.dong@intel.com<mailto:guo.dong@intel.com>>; Hau, Tze-ming <tze-ming.hau@intel.com<mailto:tze-ming.hau@intel.com>>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>
Subject: [edk2-devel] RFC: Universal Payload Interface

With the fact that there are many different firmware implementations, we tried to decouple today's monolithic UEFI firmware binary to two independent components: bootloader and payload.

Basically, bootloader initializes the silicon hardware and payload prepares the OS required data and services. Bootloader passes control to payload.

https://universalpayload.github.io/documentation/spec/spec.html defines the universal interface between bootloader and payload. So that different bootloaders can work with different payloads, initializing different hardware and booting different OSes.
The interface document is in very draft phase. Any feedback is welcome.

We also developed the POC code to demonstrate the idea. Please use below steps to get the code that uses SBL as the bootloader and EDKII UEFI Payload as the payload. This POC is being developed for QEMU Q35 virtual machine.

  1.  Run "git clone https://github.com/universalpayload/tools.git payload"

This step downloads the initial tools that will setup the dev environment.

  1.  Run "py -3 clone_and_build_sbl_with_uefipayload.py" in the "payload" directory

This script downloads branched SBL (https://github.com/universalpayload/slimbootloader.git) and edk2 (https://github.com/universalpayload/edk2.git).

Then it builds the firmware binary "SlimBootloader.bin" in "codeworkspace" directory.

  1.  Boot QEMU by running "qemu-system-x86_64.exe -machine q35 -pflash  codeworkspace\SlimBootloader.bin  -serial file:test.log"

Because the code is under active development, please contact us when you cannot build or boot successfully.

Besides the SBL, we modified coreboot (https://github.com/universalpayload/coreboot.git) to let it conform to the universal interface as a bootloader.
Besides the EDK2 UEFI Payload, we created a payload (https://github.com/universalpayload/linuxpayload.git) that can boot Linux.

Thanks,
Ray



[-- Attachment #2: Type: text/html, Size: 14244 bytes --]

Re: [edk2-devel] RFC: Universal Payload Interface

[Laszlo Ersek]

On 10/28/20 04:26, Ni, Ray wrote:
> Laszlo,
> Thank you for the comments.
> Reply inline.
> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo
>> Ersek
>> Sent: Tuesday, October 27, 2020 10:21 PM
>> To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>
>> Cc: Zimmer, Vincent <vincent.zimmer@intel.com>; Ma, Maurice
>> <maurice.ma@intel.com>; Rangarajan, Ravi P <ravi.p.rangarajan@intel.com>;
>> Dong, Guo <guo.dong@intel.com>; Hau, Tze-ming <tze-ming.hau@intel.com>
>> Subject: Re: [edk2-devel] RFC: Universal Payload Interface
>>
>> On 10/23/20 03:18, Ni, Ray wrote:
>>> With the fact that there are many different firmware implementations, we
>> tried to decouple today's monolithic UEFI firmware binary to two independent
>> components: bootloader and payload.
>>
>> (1) "Bootloader" is an extremely loaded word. Regardless of everything
>> else in this topic, I strongly suggest picking a different name.
>>
>> We already have "Platform Init" or "PI"; maybe use "Silicon Init" or "SI"?
> I am a UEFI guy for many years. So your suggestion "PI" looks very friendly to me.
> But I am not sure if the audiences include broader people, like coreboot, SBL developers,
> do they like the names.
> 
> Personally I am open to any name as long as the concept is not changed: the binary blob
> is responsible to initialize the silicon.
> 
> 
>>
>>
>> (2) What is the *exact* use case (or workflow) that the proposed
>> interface enables, or improves?
>> What groups of people (what roles) are supposed to benefit from the
>> proposed interface?
> 
> 1. Unified UEFI Payload Binary.
> By standardizing the bootloader->payload interface, we keep in mind to move all
> platform/silicon specific implementation to bootloader and all the specific info is
> passed to payload through the standard interface such that the payload doesn't need
> to deal with concrete hardware but just the abstracted info. It gives possibility to create
> the unified UEFI payload binary that can run in any platform/silicon (off course, one binary
> per one CPU arch). Just like today's UEFI Shell.
> It's a huge save on validation side to every company that uses UEFI as boot solution for
> hardware.
> People may argue maintaining such a binary causes additional overhead. I agree.
> But I am optimistic on such direction. 
> (The code will be still in open source.)
> 
> 2. Bootloaders
> It shows an attitude of EDKII community that it doesn't restrict to use EDKII PEI as the
> only acknowledged silicon code execution environment. The standard interface as a promise
> allows any compliant bootloader to work with EDKII UEFI Payload.
> 
> 3. Payloads
> I saw different tries to change EDKII DXE environment for different hardware/OS.
> It may cause defragmentation of UEFI spec. The standard interface also allows any
> compliant payload to be created.

- I don't have anything against this, as long as existent platforms are
not required to adopt the new scheme.

- The above description helps, but it is *still* too generic for me to
understand:

(a) Intel already distributes Firmware Support Packages, which are
supposed to deal with RAM / chipset initialization, AIUI,

(b) the PI spec is not tied to edk2 PEIMs, and I don't see where EDKII
PEI modules are currently the only "acknowledged" silicon init
environment. The edk2 tree itself seems to contain platforms that don't
use the edk2 PEI module set at all, but (IIRC) jump from SEC to DXE. I
believe "ArmPlatformPkg/PrePi" and "ArmVirtPkg/PrePi" are related to this.

(c) Replacing edk2 DXE should already be possible without replacing the
edk2 DXE IPL PEIM -- isn't it enough to change the contents of the boot
firmware volume?

I haven't looked at the above interaces in detail for a very long time
now, but I feel we already have the necessary abstractions in place. So
clearly they are insufficient for *some* workflows / use cases. That's
what I'd like to understand more closely. What are the specific problems
with the edk2 offerings? Can you describe example activities that
vendors or other stakeholders would like to perform today, but they
can't, or at high cost only?

I'm quite "out of the loop" on how firmware is composed *in practice*
for proprietary / binary-only / multi-vendor platforms. With my
background in OVMF, I could be missing your point-of-view entirely.

Which is why I want to say very clearly that I'm not attempting to
"block" this proposal at all -- as long as it causes neither
regressions, nor new requirements, for existent platforms --; I just
think the limitations of the current interfaces / implementation, and
the desired improvements, should be written up precisely. Case studies,
actual projects etc would help.

(I understand that you referenced actual code already, but I don't have
capacity for reviewing code, without an actual use case for OVMF. Hence
my request for a natural language description.)

Thanks
Laszlo