From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by mx.groups.io with SMTP id smtpd.web10.204807.1673979793535356628 for ; Tue, 17 Jan 2023 10:23:14 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=kayQeF5D; spf=pass (domain: kernel.org, ip: 145.40.68.75, mailfrom: ardb@kernel.org) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 841FBB819A4 for ; Tue, 17 Jan 2023 18:23:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A588DC4339C for ; Tue, 17 Jan 2023 18:23:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673979789; bh=fMN90DTeJNe7M+b6PuaaDenemsNedqnEIIs8HRsrdzA=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=kayQeF5De8YdWq5nZxqeCCcI37RJHb7mkG3hGU6V6zErShX7uCIgyZskraadxk970 7D31lnzLWq5koRyEKPrq+CewutAPiiQ0n3TxcR3cpDJkaDAkb+YL+shXPytJeC/AHS YWb7aHExCMB8QSPd1QnhtOZO3zLMk2rqUxGNFDL5TRO7WXrB5GxaoIeCilG52tj/ir WNfry7YIJmjkcXe4CY8dvoPuXBNmtRt34FGcMA9I+0M1bzLWvMmM6SoPOT9izqsh1A 0sIW3wWk+pB6WfhbW2n/f0zjezVW0PyJdOexI5j9DWgik3z1joqof5P1sLduVuoVj8 jCZhOfO1eK8/w== Received: by mail-lf1-f44.google.com with SMTP id br9so13030794lfb.4 for ; Tue, 17 Jan 2023 10:23:09 -0800 (PST) X-Gm-Message-State: AFqh2krs2jZg8WEyxyuU1/rJIGQpw1a1sazREDuaSdMKGikP4Qdd69YW j6b6BLR468s9Q0OJRuCuYOWuI5pPyNeNY1/N7YY= X-Google-Smtp-Source: AMrXdXvHwa9rUJg8KjsOslbIj2Go+kjiZvmeX7dXQLcobfQrdn5zrFcW1IzX0bDICCqmp9XKMKKDVBsTBEJiyUfuStY= X-Received: by 2002:ac2:5dd4:0:b0:4b5:964d:49a4 with SMTP id x20-20020ac25dd4000000b004b5964d49a4mr340630lfq.637.1673979787454; Tue, 17 Jan 2023 10:23:07 -0800 (PST) MIME-Version: 1.0 References: <20230116174044.3346383-1-osteffen@redhat.com> In-Reply-To: <20230116174044.3346383-1-osteffen@redhat.com> From: "Ard Biesheuvel" Date: Tue, 17 Jan 2023 19:22:56 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v11 00/17] CI: Use Fedora 35 container for Linux jobs To: Oliver Steffen Cc: devel@edk2.groups.io, Andrew Fish , Ard Biesheuvel , Bob Feng , Dandan Bi , Gerd Hoffmann , Jian J Wang , Jiewen Yao , Jordan Justen , Leif Lindholm , Liming Gao , Michael D Kinney , Michael Kubacki , Ray Ni , Sami Mujawar , Sean Brogan , Yuwei Chen , Pawel Polawski , chris.fernald@outlook.com, Chris Fernald Content-Type: text/plain; charset="UTF-8" On Mon, 16 Jan 2023 at 18:41, Oliver Steffen wrote: > > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom > containers. This decouples the CI environment from the virtual machine > images that Azure DevOps provides. The currently used ubuntu-18.04 image > has been deprecated for a while now and will be removed soon. > > The container image provides the required compiler toolchains and Qemu > for the supported architectures. These are then no longer downloaded at > runtime, avoiding CI failures due to download errors. This approach also > makes it easier to switch to other or newer compilers. It makes the CI > setup independent from the default images that Azure DevOps provides. > It can also help debugging CI problems, because the CI environment > can be reproduced on a local machine. > > The container images are hosted on ghcr.io and are automatically > generated using GitHub Actions. The Dockerfiles are maintained in the > Tianocore "containers" repository: > https://github.com/tianocore/containers. > > The current image is based on Fedora 35, with gcc 11. Fedora was chosen > because of its fast release cycle which makes it easy to keep the > toolchains up-to-date. > > Some further possible changes not included in this series: > - The Tianocore/containers repository provides stack of layered images. > One image for general purpose (build+test) and build-only jobs. > The build+test image is based on the build-only one and adds Qemu, > for the testing job that involve Qemu. The work in the image side > is done, we just need to change the CI setup accordingly. > This patch set uses the build+test images for all jobs. > - Further reduce the number of external dependencies that need to be > downloaded at runtime. Candidates are iasl and nasm, which are already > included in the image but not used yet. > > PR: https://github.com/tianocore/edk2/pull/3904 > > v11: > - Rebase to current master branch > - Use lasted image, includes Qemu 7.2 > - Set container image URL in a central place (the defaults.yml template) > > v10: > - Split commits by package > - Use ubuntu-22.04 as vm_image (instead of the floating ubuntu-latest) > - Some fixups around the Python version template argument and defaults.yml > - Fix some commit messages > - Add license statement to defaults.yml > > v9: > - Drop the "Don't install cspell" patch > - Use explicitly use ubuntu-22.04 as vm_image > > v8: > - Use updated container image that contains gcc for LoongArch64. > - Remove ext_dep files for the LoongArch64 gcc. > - Don't change the scopes in .pytool/CISettings.py if running Linux CI > it. > - Split commits that touch multiple packages. > - Use the smaller "build" image for jobs that allow it. > - Add a CI template file as a central place to define the default Python version > and use it where needed. > > v7: > - Rebase to latest master branch. > - Use latest Fedora 35 CI image. > - Stop using the ubuntu-18.04 vm_image since this will no longer be available > after Dec 1st. Use ubuntu-latest instead. > > v6: > - Include suggestions by Chris Fernald. > - Added a parameter for the container image to the job template, makes usage > of containers optional. > - Added a parameter to configure the Python version to download. Allows > using Python from the VM/container image also. > - Restructure the commits (no further functional changes). > > v5: > - Update image > > v4: > - Use the latest image from the tianocode/containers repository which > - does not include acpica-tools > - includes Pyhton 3.10 > > v3: > - Use the latest image from the tianocode/containers repository which > pins down version numbers of gcc, iasl, and nasm in the Dockerfile. > > v2: > - Images are now hosted under the Tianocore Organization > https://github.com/tianocore/containers > > v1: > - Thread: https://edk2.groups.io/g/devel/message/89058 > - Images were hosted at https://github.com/osteffenrh/edk2-build-images > > Signed-off-by: Oliver Steffen > Acked-by: Ard Biesheuvel > Acked-by: Gerd Hoffmann > Reviewed-by: Chris Fernald > > Oliver Steffen (17): > CI: make Python version configurable > ArmVirtPkg: CI: use Python version from defaults template > EmulatorPkg: CI: use Python version from defaults template > OvmfPkg: CI: use Python version from defaults template > CI: add ~/.local/bin to PATH (Linux only) > CI: Allow running in a container. > CI: Use Fedora 35 container (Linux only) > ArmVirtPkg: CI: Use Fedora 35 container (Linux only) > EmulatorPkg: CI: Use Fedora 35 container (Linux only) > OvmfPkg: CI: Use Fedora 35 container (Linux only) > .pytool: CISettings.py: don't add scopes for GCC > BaseTools: remove ext_dep files for gcc > ArmVirtPkg: CI: use ubuntu-22.04 vm_image (Linux only) > EmulatorPgk: CI: use ubuntu-22.04 vm_image (Linux only) > OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only) > CI: use ubuntu-22.04 image (Linux only) Thanks I have created a PR here https://github.com/tianocore/edk2/pull/3913 with all the acks added. > OfmvPkg: CI: Qemu tests: set CPU to Broadwell > ... and I omitted this one (as discussed on the thread) > .azurepipelines/Ubuntu-GCC5.yml | 12 +++++----- > .azurepipelines/Windows-VS2019.yml | 4 ++++ > .../templates/basetools-build-steps.yml | 9 -------- > .azurepipelines/templates/defaults.yml | 12 ++++++++++ > .../templates/platform-build-run-steps.yml | 12 +++++++++- > .../templates/pr-gate-build-job.yml | 6 +++++ > .azurepipelines/templates/pr-gate-steps.yml | 12 ++++++++-- > .pytool/CISettings.py | 9 -------- > .../.azurepipelines/Ubuntu-GCC5.yml | 12 +++++----- > BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ------------------ > BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ------------------ > ...gcc_loongarch64_unknown_linux_ext_dep.yaml | 22 ------------------- > .../Bin/gcc_riscv64_unknown_ext_dep.yaml | 22 ------------------- > .../.azurepipelines/Ubuntu-GCC5.yml | 9 +++++++- > .../.azurepipelines/Windows-VS2019.yml | 4 ++++ > .../.azurepipelines/Ubuntu-GCC5.yml | 12 +++++----- > .../.azurepipelines/Windows-VS2019.yml | 5 +++++ > OvmfPkg/PlatformCI/PlatformBuildLib.py | 1 + > 18 files changed, 82 insertions(+), 123 deletions(-) > create mode 100644 .azurepipelines/templates/defaults.yml > delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml > delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml > delete mode 100644 BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml > delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml > > -- > 2.39.0 >