[PATCH v7 0/6] CI: Use Fedora 35 container for Linux jobs

[PATCH v7 0/6] CI: Use Fedora 35 container for Linux jobs

[Oliver Steffen]

Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
containers.  This decouples the CI environment from the virtual machine
images that Azure DevOps provides.  The currently used ubuntu-18.04 image
has been deprecated for a while now and will finally be removed on
Dec 1st 2022.

The container image provides the required compiler toolchains and Qemu
for the supported architectures. These are then no longer downloaded at
runtime, avoiding CI failures due to download errors. This approach also
makes it easier to switch to other or newer compilers. It makes the CI
setup independent from the default images that Azure DevOps provides.
It can also help debugging CI problems, because the CI environment
can be reproduced on a local machine.

The container images are hosted on ghcr.io and are automatically
generated using GitHub Actions. The Dockerfiles are maintained in the
Tianocore "containers" repository:
https://github.com/tianocore/containers.

The current image is based on Fedora 35, with gcc 11. Fedora was chosen
because of its fast release cycle which makes it easy to keep the
toolchains up-to-date.

Some further possible changes not included in this series:
- The Tianocore/containers repository provides stack of layered images.
  One image for general purpose (build+test) and build-only jobs.
  The build+test image is based on the build-only one and adds Qemu,
  for the testing job that involve Qemu.  The work in the image side
  is done, we just need to change the CI setup accordingly.
  This patch set uses the build+test images for all jobs.
- Further reduce the number of external dependencies that need to be
  downloaded at runtime. Candidates are iasl and nasm, which are already
  included in the image but not used yet.

PR: https://github.com/tianocore/edk2/pull/3677

v7:
- Rebase to latest master branch.
- Use latest Fedora 35 CI image.
- Stop using the ubuntu-18.04 vm_image since this will no longer be available
  after Dec 1st.  Use ubuntu-latest instead.

v6:
- Include suggestions by Chris Fernald.
- Added a parameter for the container image to the job template, makes usage
  of containers optional.
- Added a parameter to configure the Python version to download. Allows
  using Python from the VM/container image also.
- Restructure the commits (no further functional changes).

v5:
- Update image

v4:
- Use the latest image from the tianocode/containers repository which
  - does not include acpica-tools
  - includes Pyhton 3.10

v3:
- Use the latest image from the tianocode/containers repository which
  pins down version numbers of gcc, iasl, and nasm in the Dockerfile.

v2:
- Images are now hosted under the Tianocore Organization
  https://github.com/tianocore/containers

v1:
- Thread: https://edk2.groups.io/g/devel/message/89058
- Images were hosted at https://github.com/osteffenrh/edk2-build-images

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Chris Fernald <chfernal@microsoft.com>

Oliver Steffen (6):
  CI: make Python version configurable
  CI: add ~/.local/bin to PATH (Linux only)
  CI: Allow running in a container.
  CI: Use Fedora 35 container (Linux only)
  BaseTools: Remove ext. gcc dependencies (Linux only)
  CI: use latest vm_image (Linux only)

 .azurepipelines/Ubuntu-GCC5.yml               |  2 ++
 .../templates/basetools-build-steps.yml       |  9 --------
 .../templates/platform-build-run-steps.yml    | 12 +++++++++-
 .../templates/pr-gate-build-job.yml           |  6 +++++
 .azurepipelines/templates/pr-gate-steps.yml   | 12 ++++++++--
 .../.azurepipelines/Ubuntu-GCC5.yml           |  9 ++++----
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 .../.azurepipelines/Ubuntu-GCC5.yml           |  5 ++++-
 .../.azurepipelines/Ubuntu-GCC5.yml           |  9 ++++----
 11 files changed, 41 insertions(+), 87 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

-- 
2.38.1

Re: [edk2-devel] [PATCH v7 0/6] CI: Use Fedora 35 container for Linux jobs

[Sean]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

Oliver,

Thanks for this changeset and i think this is getting closer but over 
the past few months there have been a few changes that I don't think 
this series takes into account.

A few comments i hope we can address quickly (since 18.04 is going 
offline tomorrow).

1. Can the whole block at 
https://github.com/tianocore/edk2/blob/dd3ba82d31a6d3cc4564dc83c9229e13773b55da/.pytool/CISettings.py#L172 
be removed.  The only reason these exist is to pick up and download the 
compilers.

2. what about supporting the loongarch64 compiler? Can we add to container?

3. If loongarch64 is included in container should you delete this file 
too: edk2/gcc_loongarch64_unknown_linux_ext_dep.yaml at master · 
tianocore/edk2 (github.com) 
<https://github.com/tianocore/edk2/blob/master/BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml>

4.  Patch 4 has changes in multiple packages.  I believe it is a 
requirement in edk2 to only change a single package per commit. It makes 
reviews and cherry-picking/bisection easier.

5. Since only Core CI as controlled by  file 
.azurepipelines/Ubuntu_GCC5.yml why not use the base container instead 
of test container?  The Core CI Process will not use qemu.  The platform 
Ci files have their own yml with container specifier.

6. what about removing the steps for installing node and cspell from the 
container build?  Since these are managed in the container they should 
be skipped.


Thanks

Sean



On 11/29/2022 11:26 AM, Oliver Steffen wrote:
> -18.04 vm_image since this will no longer be available
>    after Dec 1st.  Use ubuntu-latest instead.

[-- Attachment #2: Type: text/html, Size: 2505 bytes --]

Re: [edk2-devel] [PATCH v7 0/6] CI: Use Fedora 35 container for Linux jobs

[Oliver Steffen]

Sean,


Quoting Sean Brogan (2022-11-30 01:28:45)
> Oliver,
>
> Thanks for this changeset and i think this is getting closer but over the past
> few months there have been a few changes that I don't think this series takes
> into account.
>
> A few comments i hope we can address quickly (since 18.04 is going offline
> tomorrow).
>
> 1. Can the whole block at [1]https://github.com/tianocore/edk2/blob/
> dd3ba82d31a6d3cc4564dc83c9229e13773b55da/.pytool/CISettings.py#L172 be
> removed.  The only reason these exist is to pick up and download the
> compilers.

Yes. Done.

>
> 2. what about supporting the loongarch64 compiler? Can we add to container?

Added the gcc release from the LoongSon GitHub page.
This is a 13.0.0 20220906 (experimental), and not yet available via
regular repositories.  I opted for the "c-only" package which is only
70MB instead of the "full" one which was used in the ext_dep file
(921MB).

>
> 3. If loongarch64 is included in container should you delete this file too: [2]
> edk2/gcc_loongarch64_unknown_linux_ext_dep.yaml at master   tianocore/edk2
> (github.com)
>
Deleted.

> 4.  Patch 4 has changes in multiple packages.  I believe it is a requirement in
> edk2 to only change a single package per commit.  It makes reviews and
> cherry-picking/bisection easier.

Commit is now split by package.

> 5. Since only Core CI as controlled by  file .azurepipelines/Ubuntu_GCC5.yml
> why not use the base container instead of test container?  The Core CI Process
> will not use qemu.  The platform Ci files have their own yml with container
> specifier.

Changed.

>
> 6. what about removing the steps for installing node and cspell from the
> container build?  Since these are managed in the container they should be
> skipped.

I do not know what exactly what is was added for.  I'll open an issue on
the containers github for that.  I think this is not critical at the
moment.  We can still thin out the image after tomorrow.

>
> Thanks
>
> Sean
>

We'll need to merge the changes for the docker file and let the new
image be generated.  Then I can prepare the next version of the patches.


Thanks,
  Oliver