arm64 support for stuart

arm64 support for stuart

[Ard Biesheuvel]

Hello all,

Given some recent issues with OVMF and ArmVirtPkg, where regressions
were not caught, or resulted in different behavior between TCG
(emulation) and KVM (virtualized executon) when running under KVM.

So I have started with increasing the test coverage for ArmVIrtPkg,
even for configurations that are not necessarily used in production.
(E.g., ArmVirtQemu can be built with secure boot support, even though
the virtual platform is not constructed in a way where using it makes
a lot of sense). This should increase the likelihood that we will
catch CryptoPkg or SecurityPkg build time regressions that could
potentially affect other ARM platforms as well.

Another thing I would like to implement is support for running QEMU in
KVM mode from stuart_build.py. One aspect of that is deciding whether
or not KVM is available and accessible (suggestions welcome,
especially with the containerized build env now in the mix)

On arm64, the architecture is much more finicky when it comes to
memory ordering, alignment and mismatched memory attributes and
coherency, and so enabling boot testing under KVM would be a
meaningful addition there as well, given that TCG emulation does not
model any of these behaviors However, the stuart stack fails to locate
some of its dependencies (notably mu_nasm and iasl), and so I cannot
even build the firmware on a arm64 Linux system, let alone execute it
under virtualization.

Given that nasm is x86 specific, we should be able to work around this
by moving the nasm_ext_dep.yaml file into the right place. Then,
if/when mu_nasm for arm64 becomes available, we will also be able to
build OVMF from arm64 (although I am probably the only person in the
world who does that regularly.)

iasl is a different matter, as we need it to build for arm64 as well.
iasl is already available in the arm64 distros, so as I see it, there
are 3 options here:
- build iasl for Linux/arm64 and add it to the nuget repo
- allow a fallback to system-wide iasl (how?)
- install the x86 version and run it under qemu user mode emulation
(which can be enabled transparently on most distros using binfmt-misc)

I have no idea whether or not the latter is feasible in terms of
dependencies (i.e., if the binaries are statically built, it should be
straightforward, but otherwise it is going to be tricky)

With this out of the way, I can happily run stuart_setup/update/build
and build and boot test ArmVirtQemu under KVM virtualization.

Re: arm64 support for stuart

[Gerd Hoffmann]

  Hi,

> Given that nasm is x86 specific, we should be able to work around this
> by moving the nasm_ext_dep.yaml file into the right place.

Overall stuart feels kind of alien to linux.  It just goes download
stuff from the internet, even in case the tools are already available
locally.  Oliver fixed some (but not all) of these when moving CI over
to using containers.  IIRC at least the cross compilers are just the
standard fedora cross compiler packages now.

While being at it:  edk2-pytool-library fails to build with network
access turned off[1] because it tries to download vswhere.exe from the
internet.  Even when building on linux.

> Then, if/when mu_nasm for arm64 becomes available, we will also be
> able to build OVMF from arm64 (although I am probably the only person
> in the world who does that regularly.)

Fedora build system does that too.  We have a patch to make x86 cross
builds work like arm cross builds, by just setting GCC5_${ARCH}_PREFIX
environment variable:
  https://github.com/kraxel/edk2/commit/6b2ca6f01bb76a3b9632e902b4bf0ef9e912ce40

Guess I should submit that one for upstream inclusion ;)

> iasl is a different matter, as we need it to build for arm64 as well.
> iasl is already available in the arm64 distros, so as I see it, there
> are 3 options here:
> - build iasl for Linux/arm64 and add it to the nuget repo
> - allow a fallback to system-wide iasl (how?)

Just use the system-wide tools is the best option IMHO.  The packages
are available in Fedora (other distros should be have them too), on both
x86_64 and aarch64, we only need to add them to the CI container image.
So why bother adding nuget builds?

That is also less fragile than downloading them on each CI run and have
checks fail now and then due to network problems.

Is it possible to run github actions (used to build containers) and
azure pipelines on aarch64 systems?  So we could move ArmVirt CI from
x86 cross builds to native arm builds?

take care,
  Gerd

[1] Offline builds are standard for linux distro builds to make sure
    all sources needed are to produce the binary package are actually
    included in the source package.

Re: [edk2-devel] arm64 support for stuart

[Michael Brown]

On 25/01/2023 16:55, Gerd Hoffmann wrote:
>> iasl is a different matter, as we need it to build for arm64 as well.
>> iasl is already available in the arm64 distros, so as I see it, there
>> are 3 options here:
>> - build iasl for Linux/arm64 and add it to the nuget repo
>> - allow a fallback to system-wide iasl (how?)
> 
> Just use the system-wide tools is the best option IMHO.  The packages
> are available in Fedora (other distros should be have them too), on both
> x86_64 and aarch64, we only need to add them to the CI container image.
> So why bother adding nuget builds?

I totally agree with this.  Life is much easier when everything just 
uses the existing system-wide tools.

Deliberately imposing a restriction that CI uses only system-wide tools 
also helps to prevent a project slipping down into the abyss of obscure 
dependencies and multi-GB custom SDK container images.

Thanks,

Michael

Re: arm64 support for stuart

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 2632 bytes --]

On Wed, Jan 25, 2023 at 5:56 PM Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
>
> > Given that nasm is x86 specific, we should be able to work around this
> > by moving the nasm_ext_dep.yaml file into the right place.
>
> Overall stuart feels kind of alien to linux.  It just goes download
> stuff from the internet, even in case the tools are already available
> locally.  Oliver fixed some (but not all) of these when moving CI over
> to using containers.  IIRC at least the cross compilers are just the
> standard fedora cross compiler packages now.
>

I would like an option to just turn off all the toolchain dependency magic
stuart does.  I had a look at stuart and how to implement this but then
did not want to attempt anything without feedback from the maintainers.
For the CI we ended up just plainly deleting the ext_dep files for the
compilers.


>
> While being at it:  edk2-pytool-library fails to build with network
> access turned off[1] because it tries to download vswhere.exe from the
> internet.  Even when building on linux.
>
> > Then, if/when mu_nasm for arm64 becomes available, we will also be
> > able to build OVMF from arm64 (although I am probably the only person
> > in the world who does that regularly.)
>
> Fedora build system does that too.  We have a patch to make x86 cross
> builds work like arm cross builds, by just setting GCC5_${ARCH}_PREFIX
> environment variable:
>
> https://github.com/kraxel/edk2/commit/6b2ca6f01bb76a3b9632e902b4bf0ef9e912ce40
>
> Guess I should submit that one for upstream inclusion ;)
>
> > iasl is a different matter, as we need it to build for arm64 as well.
> > iasl is already available in the arm64 distros, so as I see it, there
> > are 3 options here:
> > - build iasl for Linux/arm64 and add it to the nuget repo
> > - allow a fallback to system-wide iasl (how?)
>
> Just use the system-wide tools is the best option IMHO.  The packages
> are available in Fedora (other distros should be have them too), on both
> x86_64 and aarch64, we only need to add them to the CI container image.
> So why bother adding nuget builds?
>
> That is also less fragile than downloading them on each CI run and have
> checks fail now and then due to network problems.
>
> Is it possible to run github actions (used to build containers) and
> azure pipelines on aarch64 systems?  So we could move ArmVirt CI from
> x86 cross builds to native arm builds?
>
> take care,
>   Gerd
>
> [1] Offline builds are standard for linux distro builds to make sure
>     all sources needed are to produce the binary package are actually
>     included in the source package.
>

-Oliver

[-- Attachment #2: Type: text/html, Size: 3963 bytes --]

Re: arm64 support for stuart

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

On Wed, Jan 25, 2023 at 5:56 PM Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
>
> > Given that nasm is x86 specific, we should be able to work around this
> > by moving the nasm_ext_dep.yaml file into the right place.
>
> Overall stuart feels kind of alien to linux.  It just goes download
> stuff from the internet, even in case the tools are already available
> locally.  Oliver fixed some (but not all) of these when moving CI over
> to using containers.  IIRC at least the cross compilers are just the
> standard fedora cross compiler packages now.
>
> While being at it:  edk2-pytool-library fails to build with network
> access turned off[1] because it tries to download vswhere.exe from the
> internet.  Even when building on linux.
>
> > Then, if/when mu_nasm for arm64 becomes available, we will also be
> > able to build OVMF from arm64 (although I am probably the only person
> > in the world who does that regularly.)
>
> Fedora build system does that too.  We have a patch to make x86 cross
> builds work like arm cross builds, by just setting GCC5_${ARCH}_PREFIX
> environment variable:
>
> https://github.com/kraxel/edk2/commit/6b2ca6f01bb76a3b9632e902b4bf0ef9e912ce40
>
> Guess I should submit that one for upstream inclusion ;)
>
> > iasl is a different matter, as we need it to build for arm64 as well.
> > iasl is already available in the arm64 distros, so as I see it, there
> > are 3 options here:
> > - build iasl for Linux/arm64 and add it to the nuget repo
> > - allow a fallback to system-wide iasl (how?)
>
> Just use the system-wide tools is the best option IMHO.  The packages
> are available in Fedora (other distros should be have them too), on both
> x86_64 and aarch64, we only need to add them to the CI container image.
> So why bother adding nuget builds?
>
> That is also less fragile than downloading them on each CI run and have
> checks fail now and then due to network problems.
>
> Is it possible to run github actions (used to build containers) and
> azure pipelines on aarch64 systems?  So we could move ArmVirt CI from
> x86 cross builds to native arm builds?


Not natively, AFAIK, at least not without self-hosting a runner or using
3rd-party
services.
Most suggestions I came across suggest running in Qemu (tcg)...


>
> take care,
>   Gerd
>
> [1] Offline builds are standard for linux distro builds to make sure
>     all sources needed are to produce the binary package are actually
>     included in the source package.
>
>

[-- Attachment #2: Type: text/html, Size: 3710 bytes --]

Re: [edk2-devel] arm64 support for stuart

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 3482 bytes --]

Sounds like a reasonable feature request to disable install of all external tools.  Pytools uses GitHub issues, so a feature request like this can be entered there.

One caution on this approach is that you make see passing conditions using your local versions of dependent tools,
but when you submit a PR and the CI tools run with potentially different tools, there may be failures.

If a failure it detected, then you will likely need to install the same version of tools the CI uses to reproduce,
debug, and resolve.

Mike

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Oliver Steffen
Sent: Wednesday, January 25, 2023 10:28 AM
To: Gerd Hoffmann <kraxel@redhat.com>
Cc: Ard Biesheuvel <ardb@kernel.org>; Michael Kubacki <mikuback@linux.microsoft.com>; Sean Brogan <sean.brogan@microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>; edk2-devel-groups-io <devel@edk2.groups.io>; Yao, Jiewen <jiewen.yao@intel.com>
Subject: Re: [edk2-devel] arm64 support for stuart



On Wed, Jan 25, 2023 at 5:56 PM Gerd Hoffmann <kraxel@redhat.com<mailto:kraxel@redhat.com>> wrote:
  Hi,

> Given that nasm is x86 specific, we should be able to work around this
> by moving the nasm_ext_dep.yaml file into the right place.

Overall stuart feels kind of alien to linux.  It just goes download
stuff from the internet, even in case the tools are already available
locally.  Oliver fixed some (but not all) of these when moving CI over
to using containers.  IIRC at least the cross compilers are just the
standard fedora cross compiler packages now.

While being at it:  edk2-pytool-library fails to build with network
access turned off[1] because it tries to download vswhere.exe from the
internet.  Even when building on linux.

> Then, if/when mu_nasm for arm64 becomes available, we will also be
> able to build OVMF from arm64 (although I am probably the only person
> in the world who does that regularly.)

Fedora build system does that too.  We have a patch to make x86 cross
builds work like arm cross builds, by just setting GCC5_${ARCH}_PREFIX
environment variable:
  https://github.com/kraxel/edk2/commit/6b2ca6f01bb76a3b9632e902b4bf0ef9e912ce40

Guess I should submit that one for upstream inclusion ;)

> iasl is a different matter, as we need it to build for arm64 as well.
> iasl is already available in the arm64 distros, so as I see it, there
> are 3 options here:
> - build iasl for Linux/arm64 and add it to the nuget repo
> - allow a fallback to system-wide iasl (how?)

Just use the system-wide tools is the best option IMHO.  The packages
are available in Fedora (other distros should be have them too), on both
x86_64 and aarch64, we only need to add them to the CI container image.
So why bother adding nuget builds?

That is also less fragile than downloading them on each CI run and have
checks fail now and then due to network problems.

Is it possible to run github actions (used to build containers) and
azure pipelines on aarch64 systems?  So we could move ArmVirt CI from
x86 cross builds to native arm builds?

Not natively, AFAIK, at least not without self-hosting a runner or using 3rd-party
services.
Most suggestions I came across suggest running in Qemu (tcg)...


take care,
  Gerd

[1] Offline builds are standard for linux distro builds to make sure
    all sources needed are to produce the binary package are actually
    included in the source package.


[-- Attachment #2: Type: text/html, Size: 45028 bytes --]

Re: [edk2-devel] arm64 support for stuart

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 4240 bytes --]

On Wed, Jan 25, 2023 at 8:00 PM Kinney, Michael D <
michael.d.kinney@intel.com> wrote:

> Sounds like a reasonable feature request to disable install of all
> external tools.  Pytools uses GitHub issues, so a feature request like
> this can be entered there.
>
There was a discussion about this
(https://github.com/tianocore/edk2-pytool-extensions/discussions/323),
where some possible
approaches were described.
Back then we were under the time pressure of getting the CI patches in
before Dec 1st (the
day the ubuntu-18 image was expected to disappear from Azure Pipelines).
In the end the solution of deleting the ext_dep files was accepted.
I opened a feature request here:
https://github.com/tianocore/edk2-pytool-extensions/issues/416


>
> One caution on this approach is that you make see passing conditions using
> your local versions of dependent tools,
>
> but when you submit a PR and the CI tools run with potentially different
> tools, there may be failures.
>
>
>
> If a failure it detected, then you will likely need to install the same
> version of tools the CI uses to reproduce,
>
> debug, and resolve.
>

Sure.  But at least there would be a good way to bring your own tools.

- Oliver


>
> Mike
>
>
>
> *From:* devel@edk2.groups.io <devel@edk2.groups.io> *On Behalf Of *Oliver
> Steffen
> *Sent:* Wednesday, January 25, 2023 10:28 AM
> *To:* Gerd Hoffmann <kraxel@redhat.com>
> *Cc:* Ard Biesheuvel <ardb@kernel.org>; Michael Kubacki <
> mikuback@linux.microsoft.com>; Sean Brogan <sean.brogan@microsoft.com>;
> Kinney, Michael D <michael.d.kinney@intel.com>; edk2-devel-groups-io <
> devel@edk2.groups.io>; Yao, Jiewen <jiewen.yao@intel.com>
> *Subject:* Re: [edk2-devel] arm64 support for stuart
>
>
>
>
>
>
>
> On Wed, Jan 25, 2023 at 5:56 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > Given that nasm is x86 specific, we should be able to work around this
> > by moving the nasm_ext_dep.yaml file into the right place.
>
> Overall stuart feels kind of alien to linux.  It just goes download
> stuff from the internet, even in case the tools are already available
> locally.  Oliver fixed some (but not all) of these when moving CI over
> to using containers.  IIRC at least the cross compilers are just the
> standard fedora cross compiler packages now.
>
> While being at it:  edk2-pytool-library fails to build with network
> access turned off[1] because it tries to download vswhere.exe from the
> internet.  Even when building on linux.
>
> > Then, if/when mu_nasm for arm64 becomes available, we will also be
> > able to build OVMF from arm64 (although I am probably the only person
> > in the world who does that regularly.)
>
> Fedora build system does that too.  We have a patch to make x86 cross
> builds work like arm cross builds, by just setting GCC5_${ARCH}_PREFIX
> environment variable:
>
> https://github.com/kraxel/edk2/commit/6b2ca6f01bb76a3b9632e902b4bf0ef9e912ce40
>
> Guess I should submit that one for upstream inclusion ;)
>
> > iasl is a different matter, as we need it to build for arm64 as well.
> > iasl is already available in the arm64 distros, so as I see it, there
> > are 3 options here:
> > - build iasl for Linux/arm64 and add it to the nuget repo
> > - allow a fallback to system-wide iasl (how?)
>
> Just use the system-wide tools is the best option IMHO.  The packages
> are available in Fedora (other distros should be have them too), on both
> x86_64 and aarch64, we only need to add them to the CI container image.
> So why bother adding nuget builds?
>
> That is also less fragile than downloading them on each CI run and have
> checks fail now and then due to network problems.
>
> Is it possible to run github actions (used to build containers) and
> azure pipelines on aarch64 systems?  So we could move ArmVirt CI from
> x86 cross builds to native arm builds?
>
>
>
> Not natively, AFAIK, at least not without self-hosting a runner or using
> 3rd-party
>
> services.
>
> Most suggestions I came across suggest running in Qemu (tcg)...
>
>
>
>
> take care,
>   Gerd
>
> [1] Offline builds are standard for linux distro builds to make sure
>     all sources needed are to produce the binary package are actually
>     included in the source package.
>
> 
>

[-- Attachment #2: Type: text/html, Size: 10192 bytes --]