Re: [PATCH v11 00/17] CI: Use Fedora 35 container for Linux jobs

["Ard Biesheuvel" <ardb@kernel.org>]

On Mon, 16 Jan 2023 at 18:41, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
> containers.  This decouples the CI environment from the virtual machine
> images that Azure DevOps provides.  The currently used ubuntu-18.04 image
> has been deprecated for a while now and will be removed soon.
>
> The container image provides the required compiler toolchains and Qemu
> for the supported architectures. These are then no longer downloaded at
> runtime, avoiding CI failures due to download errors. This approach also
> makes it easier to switch to other or newer compilers. It makes the CI
> setup independent from the default images that Azure DevOps provides.
> It can also help debugging CI problems, because the CI environment
> can be reproduced on a local machine.
>
> The container images are hosted on ghcr.io and are automatically
> generated using GitHub Actions. The Dockerfiles are maintained in the
> Tianocore "containers" repository:
> https://github.com/tianocore/containers.
>
> The current image is based on Fedora 35, with gcc 11. Fedora was chosen
> because of its fast release cycle which makes it easy to keep the
> toolchains up-to-date.
>
> Some further possible changes not included in this series:
> - The Tianocore/containers repository provides stack of layered images.
>   One image for general purpose (build+test) and build-only jobs.
>   The build+test image is based on the build-only one and adds Qemu,
>   for the testing job that involve Qemu.  The work in the image side
>   is done, we just need to change the CI setup accordingly.
>   This patch set uses the build+test images for all jobs.
> - Further reduce the number of external dependencies that need to be
>   downloaded at runtime. Candidates are iasl and nasm, which are already
>   included in the image but not used yet.
>
> PR: https://github.com/tianocore/edk2/pull/3904
>
> v11:
> - Rebase to current master branch
> - Use lasted image, includes Qemu 7.2
> - Set container image URL in a central place (the defaults.yml template)
>
> v10:
> - Split commits by package
> - Use ubuntu-22.04 as vm_image (instead of the floating ubuntu-latest)
> - Some fixups around the Python version template argument and defaults.yml
> - Fix some commit messages
> - Add license statement to defaults.yml
>
> v9:
> - Drop the "Don't install cspell" patch
> - Use explicitly use ubuntu-22.04 as vm_image
>
> v8:
> - Use updated container image that contains gcc for LoongArch64.
> - Remove ext_dep files for the LoongArch64 gcc.
> - Don't change the scopes in .pytool/CISettings.py if running Linux CI
>   it.
> - Split commits that touch multiple packages.
> - Use the smaller "build" image for jobs that allow it.
> - Add a CI template file as a central place to define the default Python version
>   and use it where needed.
>
> v7:
> - Rebase to latest master branch.
> - Use latest Fedora 35 CI image.
> - Stop using the ubuntu-18.04 vm_image since this will no longer be available
>   after Dec 1st.  Use ubuntu-latest instead.
>
> v6:
> - Include suggestions by Chris Fernald.
> - Added a parameter for the container image to the job template, makes usage
>   of containers optional.
> - Added a parameter to configure the Python version to download. Allows
>   using Python from the VM/container image also.
> - Restructure the commits (no further functional changes).
>
> v5:
> - Update image
>
> v4:
> - Use the latest image from the tianocode/containers repository which
>   - does not include acpica-tools
>   - includes Pyhton 3.10
>
> v3:
> - Use the latest image from the tianocode/containers repository which
>   pins down version numbers of gcc, iasl, and nasm in the Dockerfile.
>
> v2:
> - Images are now hosted under the Tianocore Organization
>   https://github.com/tianocore/containers
>
> v1:
> - Thread: https://edk2.groups.io/g/devel/message/89058
> - Images were hosted at https://github.com/osteffenrh/edk2-build-images
>
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> Acked-by: Ard Biesheuvel <ardb@kernel.org>
> Acked-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Chris Fernald <chfernal@microsoft.com>
>
> Oliver Steffen (17):
>   CI: make Python version configurable
>   ArmVirtPkg: CI: use Python version from defaults template
>   EmulatorPkg: CI: use Python version from defaults template
>   OvmfPkg: CI: use Python version from defaults template
>   CI: add ~/.local/bin to PATH (Linux only)
>   CI: Allow running in a container.
>   CI: Use Fedora 35 container (Linux only)
>   ArmVirtPkg: CI: Use Fedora 35 container (Linux only)
>   EmulatorPkg: CI: Use Fedora 35 container (Linux only)
>   OvmfPkg: CI: Use Fedora 35 container (Linux only)
>   .pytool: CISettings.py: don't add scopes for GCC
>   BaseTools: remove ext_dep files for gcc
>   ArmVirtPkg: CI: use ubuntu-22.04 vm_image (Linux only)
>   EmulatorPgk: CI: use ubuntu-22.04 vm_image (Linux only)
>   OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only)
>   CI: use ubuntu-22.04 image (Linux only)

Thanks

I have created a PR here

https://github.com/tianocore/edk2/pull/3913

with all the acks added.

>   OfmvPkg: CI: Qemu tests: set CPU to Broadwell
>

... and I omitted this one (as discussed on the thread)


>  .azurepipelines/Ubuntu-GCC5.yml               | 12 +++++-----
>  .azurepipelines/Windows-VS2019.yml            |  4 ++++
>  .../templates/basetools-build-steps.yml       |  9 --------
>  .azurepipelines/templates/defaults.yml        | 12 ++++++++++
>  .../templates/platform-build-run-steps.yml    | 12 +++++++++-
>  .../templates/pr-gate-build-job.yml           |  6 +++++
>  .azurepipelines/templates/pr-gate-steps.yml   | 12 ++++++++--
>  .pytool/CISettings.py                         |  9 --------
>  .../.azurepipelines/Ubuntu-GCC5.yml           | 12 +++++-----
>  BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
>  BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
>  ...gcc_loongarch64_unknown_linux_ext_dep.yaml | 22 -------------------
>  .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  9 +++++++-
>  .../.azurepipelines/Windows-VS2019.yml        |  4 ++++
>  .../.azurepipelines/Ubuntu-GCC5.yml           | 12 +++++-----
>  .../.azurepipelines/Windows-VS2019.yml        |  5 +++++
>  OvmfPkg/PlatformCI/PlatformBuildLib.py        |  1 +
>  18 files changed, 82 insertions(+), 123 deletions(-)
>  create mode 100644 .azurepipelines/templates/defaults.yml
>  delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
>
> --
> 2.39.0
>