From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web11.160611.1669750002477607978 for ; Tue, 29 Nov 2022 11:26:42 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=DQdpHioM; spf=pass (domain: redhat.com, ip: 170.10.133.124, mailfrom: osteffen@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1669750001; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=Jc9myHciJNS3d0VFNLWjH4EH997i3fy/lqd/X6pv0aU=; b=DQdpHioMW0d5VElH2KlcHVv7mPmHorv0aWWo5HvJ4m+dlKFC9yXQHCxOppHuu57yAhAIaK 36xhJ7y6OSDX7uHkZINeyorpcQrSJWysNWu+bxsVeoVSBw9KFePMEZWpZVUgEzkK3e1C8o Vd0J0yXAura7GZyBrPa4qHGPDhKHRMo= Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-320-Igexqm8zNi2VI-qTTR-sLQ-1; Tue, 29 Nov 2022 14:26:39 -0500 X-MC-Unique: Igexqm8zNi2VI-qTTR-sLQ-1 Received: by mail-lf1-f71.google.com with SMTP id i14-20020ac25b4e000000b004b139c2c677so5628275lfp.12 for ; Tue, 29 Nov 2022 11:26:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:mime-version:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Jc9myHciJNS3d0VFNLWjH4EH997i3fy/lqd/X6pv0aU=; b=plvQ0N/VDCwPb/WRcRbjqN5WxZOCNrhqYzYueFR4upQzcuzDr4vawUYkuahNcI26G4 IuNhfGxgA+TFaez8DdzTxlg0h7dfGaRmDOrAo5/A+Aepgjvl0LJW9GBs3bEkAk8tj/th Sqz1KEJlu1Y/WGXQg+fFly3ysan7xNYok/EHrlgxTMlKp3Zs9YAhgGg4SbzFoawnYpF+ cZ+aNqPF5z+pHnz2jxbBamzT87yL2Lwsdn6Y8fCg9EgPXWfhTwGtlM2/JBwbbk25t9LS TddYX04zMM8gScHUaoE+phLA1Kp3OYxzmZWJc4hK3Od/MZKqG2NLqU2m3f3O1aBjus3U xu2Q== X-Gm-Message-State: ANoB5pkl0rgE7RB0mUPD/92rnfPbPfwwoDVkjqMgygsSDfeDhARIgGod FC8n36TVuiMk+L4RM8oOp+iGksDjrZx2IkNkSlSmFuYZ2DOFE0dsdOKjvRgRrv5FWNLR5reK8+C 3XWuy/eXKCtpXO/3N75Cj2qgIaeSWlQ== X-Received: by 2002:ac2:4c42:0:b0:4a5:bf09:a700 with SMTP id o2-20020ac24c42000000b004a5bf09a700mr12824624lfk.656.1669749997775; Tue, 29 Nov 2022 11:26:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf45HnY4pMjE8T6O/Z1UW/j5pBaAyPuIazMDpZmBOCmlfNeCxfJ9eEnzeN0o0ZLVu29w9mXP8Lg2Zdu3hX5cc14= X-Received: by 2002:ac2:4c42:0:b0:4a5:bf09:a700 with SMTP id o2-20020ac24c42000000b004a5bf09a700mr12824576lfk.656.1669749997392; Tue, 29 Nov 2022 11:26:37 -0800 (PST) Received: from 567203818698 named unknown by gmailapi.google.com with HTTPREST; Tue, 29 Nov 2022 14:26:36 -0500 From: "Oliver Steffen" X-Mailer: git-send-email 2.38.1 MIME-Version: 1.0 Date: Tue, 29 Nov 2022 14:26:36 -0500 Message-ID: Subject: [PATCH v7 0/6] CI: Use Fedora 35 container for Linux jobs To: devel@edk2.groups.io Cc: Abner Chang , Alexei Fedorov , Andrew Fish , Anthony Perard , Ard Biesheuvel , Bob Feng , Brijesh Singh , Chasel Chiu , Daniel Schaefer , Erdem Aktas , Eric Dong , Gerd Hoffmann , Gua Guo , Guo Dong , Guomin Jiang , Hao A Wu , Igor Kulchytskyy , James Bottomley , James Lu , Jian J Wang , Jiaxin Wu , Jiewen Yao , Jordan Justen , Julien Grall , Leif Lindholm , Liming Gao , Maciej Rabeda , Michael D Kinney , Michael Kubacki , Min Xu , Nate DeSimone , Nickle Wang , Peter Grehan , Pierre Gondois , Rahul Kumar , Ray Ni , Rebecca Cran , Sami Mujawar , Sean Brogan , Sean Rhodes , Sebastien Boeuf , Siyuan Fu , Star Zeng , Sunil V L , Supreeth Venkatesh , Tom Lendacky , Wei6 Xu , Xiaoyu Lu , Yuwei Chen , Zhichao Gao , Zhiguang Liu , Pawel Polawski , Chris Fernald , Oliver Steffen , Ard Biesheuvel , Chris Fernald X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom containers. This decouples the CI environment from the virtual machine images that Azure DevOps provides. The currently used ubuntu-18.04 image has been deprecated for a while now and will finally be removed on Dec 1st 2022. The container image provides the required compiler toolchains and Qemu for the supported architectures. These are then no longer downloaded at runtime, avoiding CI failures due to download errors. This approach also makes it easier to switch to other or newer compilers. It makes the CI setup independent from the default images that Azure DevOps provides. It can also help debugging CI problems, because the CI environment can be reproduced on a local machine. The container images are hosted on ghcr.io and are automatically generated using GitHub Actions. The Dockerfiles are maintained in the Tianocore "containers" repository: https://github.com/tianocore/containers. The current image is based on Fedora 35, with gcc 11. Fedora was chosen because of its fast release cycle which makes it easy to keep the toolchains up-to-date. Some further possible changes not included in this series: - The Tianocore/containers repository provides stack of layered images. One image for general purpose (build+test) and build-only jobs. The build+test image is based on the build-only one and adds Qemu, for the testing job that involve Qemu. The work in the image side is done, we just need to change the CI setup accordingly. This patch set uses the build+test images for all jobs. - Further reduce the number of external dependencies that need to be downloaded at runtime. Candidates are iasl and nasm, which are already included in the image but not used yet. PR: https://github.com/tianocore/edk2/pull/3677 v7: - Rebase to latest master branch. - Use latest Fedora 35 CI image. - Stop using the ubuntu-18.04 vm_image since this will no longer be available after Dec 1st. Use ubuntu-latest instead. v6: - Include suggestions by Chris Fernald. - Added a parameter for the container image to the job template, makes usage of containers optional. - Added a parameter to configure the Python version to download. Allows using Python from the VM/container image also. - Restructure the commits (no further functional changes). v5: - Update image v4: - Use the latest image from the tianocode/containers repository which - does not include acpica-tools - includes Pyhton 3.10 v3: - Use the latest image from the tianocode/containers repository which pins down version numbers of gcc, iasl, and nasm in the Dockerfile. v2: - Images are now hosted under the Tianocore Organization https://github.com/tianocore/containers v1: - Thread: https://edk2.groups.io/g/devel/message/89058 - Images were hosted at https://github.com/osteffenrh/edk2-build-images Signed-off-by: Oliver Steffen Acked-by: Ard Biesheuvel Acked-by: Gerd Hoffmann Reviewed-by: Chris Fernald Oliver Steffen (6): CI: make Python version configurable CI: add ~/.local/bin to PATH (Linux only) CI: Allow running in a container. CI: Use Fedora 35 container (Linux only) BaseTools: Remove ext. gcc dependencies (Linux only) CI: use latest vm_image (Linux only) .azurepipelines/Ubuntu-GCC5.yml | 2 ++ .../templates/basetools-build-steps.yml | 9 -------- .../templates/platform-build-run-steps.yml | 12 +++++++++- .../templates/pr-gate-build-job.yml | 6 +++++ .azurepipelines/templates/pr-gate-steps.yml | 12 ++++++++-- .../.azurepipelines/Ubuntu-GCC5.yml | 9 ++++---- BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ------------------ BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ------------------ .../Bin/gcc_riscv64_unknown_ext_dep.yaml | 22 ------------------- .../.azurepipelines/Ubuntu-GCC5.yml | 5 ++++- .../.azurepipelines/Ubuntu-GCC5.yml | 9 ++++---- 11 files changed, 41 insertions(+), 87 deletions(-) delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml -- 2.38.1