From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web11.173928.1673890859432554862 for ; Mon, 16 Jan 2023 09:40:59 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fGoJc6ca; spf=pass (domain: redhat.com, ip: 170.10.133.124, mailfrom: osteffen@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673890858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FeQ6cvMBRDoEciXw9/QzTxPqLZrOLYTR/nx5KZrLucw=; b=fGoJc6cacLaldd6/xG5J+3k3pM++Pb02UE8KG7nLoJCUkhvRKvW7tsn5KVIcZVOBkXGJxN Q4C5VOsJD/CFpUb69zC26T6+WjGeIEdo0uy7ovYH3+KMRt613fDnuFnhsaagNsM161CVId Ioa9I8RvV8mesVKenTA9TUZ+USoFR8o= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-310-nmdmMG5NMjy9Itu2nNgpsQ-1; Mon, 16 Jan 2023 12:40:55 -0500 X-MC-Unique: nmdmMG5NMjy9Itu2nNgpsQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8E9851C0878F; Mon, 16 Jan 2023 17:40:54 +0000 (UTC) Received: from osteffen-laptop.redhat.com (unknown [10.39.192.49]) by smtp.corp.redhat.com (Postfix) with ESMTP id 550B7C15BA0; Mon, 16 Jan 2023 17:40:50 +0000 (UTC) From: "Oliver Steffen" To: devel@edk2.groups.io Cc: Andrew Fish , Ard Biesheuvel , Bob Feng , Dandan Bi , Gerd Hoffmann , Jian J Wang , Jiewen Yao , Jordan Justen , Leif Lindholm , Liming Gao , Michael D Kinney , Michael Kubacki , Ray Ni , Sami Mujawar , Sean Brogan , Yuwei Chen , Pawel Polawski , chris.fernald@outlook.com, Oliver Steffen , Ard Biesheuvel , Chris Fernald Subject: [PATCH v11 00/17] CI: Use Fedora 35 container for Linux jobs Date: Mon, 16 Jan 2023 18:40:27 +0100 Message-Id: <20230116174044.3346383-1-osteffen@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom containers. This decouples the CI environment from the virtual machine images that Azure DevOps provides. The currently used ubuntu-18.04 image has been deprecated for a while now and will be removed soon. The container image provides the required compiler toolchains and Qemu for the supported architectures. These are then no longer downloaded at runtime, avoiding CI failures due to download errors. This approach also makes it easier to switch to other or newer compilers. It makes the CI setup independent from the default images that Azure DevOps provides. It can also help debugging CI problems, because the CI environment can be reproduced on a local machine. The container images are hosted on ghcr.io and are automatically generated using GitHub Actions. The Dockerfiles are maintained in the Tianocore "containers" repository: https://github.com/tianocore/containers. The current image is based on Fedora 35, with gcc 11. Fedora was chosen because of its fast release cycle which makes it easy to keep the toolchains up-to-date. Some further possible changes not included in this series: - The Tianocore/containers repository provides stack of layered images. One image for general purpose (build+test) and build-only jobs. The build+test image is based on the build-only one and adds Qemu, for the testing job that involve Qemu. The work in the image side is done, we just need to change the CI setup accordingly. This patch set uses the build+test images for all jobs. - Further reduce the number of external dependencies that need to be downloaded at runtime. Candidates are iasl and nasm, which are already included in the image but not used yet. PR: https://github.com/tianocore/edk2/pull/3904 v11: - Rebase to current master branch - Use lasted image, includes Qemu 7.2 - Set container image URL in a central place (the defaults.yml template) v10: - Split commits by package - Use ubuntu-22.04 as vm_image (instead of the floating ubuntu-latest) - Some fixups around the Python version template argument and defaults.yml - Fix some commit messages - Add license statement to defaults.yml v9: - Drop the "Don't install cspell" patch - Use explicitly use ubuntu-22.04 as vm_image v8: - Use updated container image that contains gcc for LoongArch64. - Remove ext_dep files for the LoongArch64 gcc. - Don't change the scopes in .pytool/CISettings.py if running Linux CI it. - Split commits that touch multiple packages. - Use the smaller "build" image for jobs that allow it. - Add a CI template file as a central place to define the default Python version and use it where needed. v7: - Rebase to latest master branch. - Use latest Fedora 35 CI image. - Stop using the ubuntu-18.04 vm_image since this will no longer be available after Dec 1st. Use ubuntu-latest instead. v6: - Include suggestions by Chris Fernald. - Added a parameter for the container image to the job template, makes usage of containers optional. - Added a parameter to configure the Python version to download. Allows using Python from the VM/container image also. - Restructure the commits (no further functional changes). v5: - Update image v4: - Use the latest image from the tianocode/containers repository which - does not include acpica-tools - includes Pyhton 3.10 v3: - Use the latest image from the tianocode/containers repository which pins down version numbers of gcc, iasl, and nasm in the Dockerfile. v2: - Images are now hosted under the Tianocore Organization https://github.com/tianocore/containers v1: - Thread: https://edk2.groups.io/g/devel/message/89058 - Images were hosted at https://github.com/osteffenrh/edk2-build-images Signed-off-by: Oliver Steffen Acked-by: Ard Biesheuvel Acked-by: Gerd Hoffmann Reviewed-by: Chris Fernald Oliver Steffen (17): CI: make Python version configurable ArmVirtPkg: CI: use Python version from defaults template EmulatorPkg: CI: use Python version from defaults template OvmfPkg: CI: use Python version from defaults template CI: add ~/.local/bin to PATH (Linux only) CI: Allow running in a container. CI: Use Fedora 35 container (Linux only) ArmVirtPkg: CI: Use Fedora 35 container (Linux only) EmulatorPkg: CI: Use Fedora 35 container (Linux only) OvmfPkg: CI: Use Fedora 35 container (Linux only) .pytool: CISettings.py: don't add scopes for GCC BaseTools: remove ext_dep files for gcc ArmVirtPkg: CI: use ubuntu-22.04 vm_image (Linux only) EmulatorPgk: CI: use ubuntu-22.04 vm_image (Linux only) OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only) CI: use ubuntu-22.04 image (Linux only) OfmvPkg: CI: Qemu tests: set CPU to Broadwell .azurepipelines/Ubuntu-GCC5.yml | 12 +++++----- .azurepipelines/Windows-VS2019.yml | 4 ++++ .../templates/basetools-build-steps.yml | 9 -------- .azurepipelines/templates/defaults.yml | 12 ++++++++++ .../templates/platform-build-run-steps.yml | 12 +++++++++- .../templates/pr-gate-build-job.yml | 6 +++++ .azurepipelines/templates/pr-gate-steps.yml | 12 ++++++++-- .pytool/CISettings.py | 9 -------- .../.azurepipelines/Ubuntu-GCC5.yml | 12 +++++----- BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ------------------ BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ------------------ ...gcc_loongarch64_unknown_linux_ext_dep.yaml | 22 ------------------- .../Bin/gcc_riscv64_unknown_ext_dep.yaml | 22 ------------------- .../.azurepipelines/Ubuntu-GCC5.yml | 9 +++++++- .../.azurepipelines/Windows-VS2019.yml | 4 ++++ .../.azurepipelines/Ubuntu-GCC5.yml | 12 +++++----- .../.azurepipelines/Windows-VS2019.yml | 5 +++++ OvmfPkg/PlatformCI/PlatformBuildLib.py | 1 + 18 files changed, 82 insertions(+), 123 deletions(-) create mode 100644 .azurepipelines/templates/defaults.yml delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml -- 2.39.0