Re: [EXTERNAL] [edk2-devel] building the shell for edk2-stable202102

["Laszlo Ersek" <lersek@redhat.com>]

+Mike

On 01/19/21 20:34, Bret Barkelew via groups.io wrote:
> We've definitely build release pipelines for binaries internally and
> would be happy to help with this.

Thanks!

>
> I like the idea of kicking a new release after a release is tagged. Do
> you have any thoughts about how we're possibly re-release if there
> were changes to one of the new stable/* branches, or would it be
> easier to just cross that bridge when we get there?

If I remember correctly, any given stable branch also carries tags, so
the tagging action could be used to kick off the build process there as
well.

Please refer to section (4) in the following RFC -- "Release
requirements for supported branches":

  [edk2-rfc] [RFC V3] Create supported branch from edk2-stable* tag
  (Required to address critical bug BZ3111)

  https://edk2.groups.io/g/rfc/message/470

> Do you have opinions about Nuget vs other binary release mechanisms
> (our experience is with Nuget, but I know there are other feed types
> that we could publish to)?

I think we should primarily release via the github.com Assets drop-downs
<https://github.com/tianocore/edk2/releases/> -- as long as we publish
our sources there, anyway.

Additional release channels could be enabled (if they are convenient for
some users).

... I do think it's quite important to offer bare https URLs (at least
in the primary location), which nuget.org doesn't seem to do (correct me
if I'm wrong please).

> Ideally it would be something that could be authenticated or compared
> somehow.

Yep, that's messy.

I vaguely recall discussing this, namely that github.com generates
*some* of the Assets (namely the edk2 source zip files?) on-demand, and
that leads to different results e.g. whenever the central github.com
compression settings change.

Hm... this was a topic from Leif and Mike, in the December 2020
stewards' meeting. References (from the minutes):

  https://github.com/abseil/federation-hello/issues/3
  https://github.com/spack/spack/issues/5411

FWIW, the submodule source tarballs are not affected, because those are
prepared by Liming manually, according to the method originally proposed
here: <https://bugzilla.tianocore.org/show_bug.cgi?id=2558#c8>.

In the end, Mike removed said re-generation manually, for past releases:

  https://bugzilla.tianocore.org/show_bug.cgi?id=3099

But I think we'd benefit from an automatism that doesn't leave anything
to github.com or to manual processes, and generates all the artifacts /
assets internally. Perhaps the final upload could be left to the release
manager (for final human supervision).

Summary (of my opinion):

- Tags look suitable for kicking off release builds, both on the master
  and the stable branches.

- I'd prefer releasing all assets through
  <https://github.com/tianocore/edk2/releases/> primarily, for now;
  additional channels are welcome.

- Reproducible builds are important; they appear available on github.com
  too (only every tarball etc has to be uploaded as a custom binary
  file, disabling on-demand generation).

- Cryptographic signing for assets: probably not (needs a whole separate
  organization and infrastructure for handling private keys, AFAICT).

Anyway, let's not allow "perfect" to get in the way of "good" -- this
deserves its own RFC thread probably; for now I'd just like to add a
permanent reminder "somewhere" about the need to rebuild the shell, for
edk2-stable202102.

Thanks!
Laszlo


>
> - Bret
>
> From: Laszlo Ersek via groups.io<mailto:lersek=redhat.com@groups.io>
> Sent: Tuesday, January 19, 2021 11:30 AM
> To: Liming Gao (Byosoft address)<mailto:gaoliming@byosoft.com.cn>;
> Leif Lindholm (Nuvia address)<mailto:leif@nuviainc.com>; Ard
> Biesheuvel (TianoCore)<mailto:ardb+tianocore@kernel.org>; Ni,
> Ray<mailto:ray.ni@intel.com>; Zhichao
> Gao<mailto:zhichao.gao@intel.com>; Abner
> Chang<mailto:abner.chang@hpe.com>
> Cc: edk2-devel-groups-io<mailto:devel@edk2.groups.io>
> Subject: [EXTERNAL] [edk2-devel] building the shell for
> edk2-stable202102
>
> Ouch, I totally forgot to add the mailing list to the address list!
> Doing that now. Apologies.
>
> --o--
>
> Hi All,
>
> we've last built the UEFI shell binary for edk2-stable202002 (i.e.,
> almost 1 year ago):
>
>   https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Freleases%2Ftag%2Fedk2-stable202002&amp;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C8e0e9a9a97be429c1b6c08d8bcb0bcb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637466814561378976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ObFHM4vr8SuNSLwSwBD95qIvZjt7wRCA54xinkWZeLY%3D&amp;reserved=0
>
> Note "ShellBinPkg.zip" under Assets -- there is no stable tag that is
> (a) more recent and (b) whose Assets contain "ShellBinPkg.zip".
>
> Contents:
>
>> Archive:  ShellBinPkg.zip
>>   Length      Date    Time    Name
>> ---------  ---------- -----   ----
>>         0  03-06-2020 22:43   ShellBinPkg/MinUefiShell/
>>         0  03-06-2020 22:41   ShellBinPkg/MinUefiShell/AArch64/
>>    380928  03-06-2020 17:39   ShellBinPkg/MinUefiShell/AArch64/Shell.efi
>>         0  03-06-2020 22:41   ShellBinPkg/MinUefiShell/Arm/
>>    321568  03-06-2020 17:38   ShellBinPkg/MinUefiShell/Arm/Shell.efi
>>         0  03-05-2020 09:01   ShellBinPkg/MinUefiShell/Ia32/
>>    339424  03-05-2020 09:01   ShellBinPkg/MinUefiShell/Ia32/Shell.efi
>>       643  03-06-2020 22:43   ShellBinPkg/MinUefiShell/MinUefiShell.inf
>>         0  03-05-2020 09:01   ShellBinPkg/MinUefiShell/X64/
>>    392352  03-05-2020 09:01   ShellBinPkg/MinUefiShell/X64/Shell.efi
>>         0  03-06-2020 22:43   ShellBinPkg/UefiShell/
>>         0  03-06-2020 22:41   ShellBinPkg/UefiShell/AArch64/
>>    892928  03-06-2020 17:40   ShellBinPkg/UefiShell/AArch64/Shell.efi
>>         0  03-06-2020 22:41   ShellBinPkg/UefiShell/Arm/
>>    791360  03-06-2020 17:39   ShellBinPkg/UefiShell/Arm/Shell.efi
>>         0  03-05-2020 09:01   ShellBinPkg/UefiShell/Ia32/
>>    825184  03-05-2020 09:00   ShellBinPkg/UefiShell/Ia32/Shell.efi
>>       643  03-06-2020 22:43   ShellBinPkg/UefiShell/UefiShell.inf
>>         0  03-05-2020 09:01   ShellBinPkg/UefiShell/X64/
>>    939648  03-05-2020 09:01   ShellBinPkg/UefiShell/X64/Shell.efi
>>         0  03-06-2020 22:40   ShellBinPkg/
>> ---------                     -------
>>   4884678                     21 files
>
> I propose that we rebuild the shell for edk2-stable202102. Reasons:
>
> (1) There are two small shell features minimally in the latest
>     development cycle:
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Release-Planning&amp;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C8e0e9a9a97be429c1b6c08d8bcb0bcb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637466814561378976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=sH5nHKgFwcykHgNKm6Nu2esq05F1dKt3t%2BEncNyRap8%3D&amp;reserved=0
>
>     * add file buffering to the UEFI shell's COMP command
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3123&amp;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C8e0e9a9a97be429c1b6c08d8bcb0bcb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637466814561378976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Po1VBWyzo7YPioV1FUJZ2t1rB%2FhwgoofX8EH9YKJPBc%3D&amp;reserved=0
>
>     * Shell: pathname / filename sorting
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3151&amp;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C8e0e9a9a97be429c1b6c08d8bcb0bcb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637466814561378976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Qz58BSZ7iwNpyD%2BvOayA0VaA0IlDWNz14zwCjxsS6aU%3D&amp;reserved=0
>
> (2) The zip file listed above does not contain a RISC-V binary, and
>     RISC-V has been an official UEFI and edk2 platform minimally since
>     edk2-stable202005 /
>     <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2672&amp;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C8e0e9a9a97be429c1b6c08d8bcb0bcb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637466814561378976%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=B6E6J8HHERxZyqgV87n0vbJgp4NWUSMVN%2FoyaW%2FcF0U%3D&amp;reserved=0>.
>
>     In particular, the following two platforms in edk2-platforms include
>     the shell (SUPPORTED_ARCHITECTURES = RISCV64):
>
>       Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
>       Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.dsc
>
>     However, as of this writing (@ 6e5586863148), we only have the
>     following list in "Maintainers.txt":
>
>> UEFI Shell Binaries (ShellBinPkg.zip) from EDK II Releases:
>> -----------------------------------------------------------
>> W: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Freleases%2F&amp;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C8e0e9a9a97be429c1b6c08d8bcb0bcb0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637466814561388970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ikDhWcETk90pMfREpPuvAWzobu41W2vUQT9wrRRBhzM%3D&amp;reserved=0
>> M: Ray Ni <ray.ni@intel.com>                  (Ia32/X64)
>> M: Zhichao Gao <zhichao.gao@intel.com>        (Ia32/X64)
>> M: Leif Lindholm <leif@nuviainc.com>          (ARM/AArch64)
>> M: Ard Biesheuvel <ardb+tianocore@kernel.org> (ARM/AArch64)
>
>     I think that (a) Abner should be added to this list, and (b) we
>     should include a RISC-V shell binary in the upcoming assets.
>
>     Abner, can you send a patch for "Maintainers.txt" please?
>
> Questions:
>
> - I'm not clear on how we intend to build the shell binaries -- will we
>   retrieve them from CI / Azure somehow, or is it a manual process?
>
> - Given that this is a release activity, I'm unsure where I could file a
>   reminder about it -- clearly, the binaries should be built right after
>   the tag has been made.
>
>   Should I perhaps file a new reminder BZ for the "N/A" Package, and
>   maybe assign it to Liming (our release manager)?
>
> Thanks,
> Laszlo
>
>
>
>
>
>
>
>
> 
>
>
>