From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.groups.io with SMTP id smtpd.web08.43.1664210882326760130 for ; Mon, 26 Sep 2022 09:48:02 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=B2oMKjr3; spf=pass (domain: redhat.com, ip: 170.10.129.124, mailfrom: osteffen@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1664210881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=FQdy+cTxXWzwOFbzRHrnCRastf39w0bCwjHbUrtxJ+U=; b=B2oMKjr3PirDZpZlXbIW+oXYubU2DwSVYpN7Z7M5hlflAMYJ0dbQZsBNMZ6Yrx0vUwICr0 L0vobZYoqiGQYgwX3yGFreXcmiyalmsCDm1vkCF165OemrklC0umXTs+HnbfQFdL0IV0SP VPf9jKSwTh9RDDezIXDLOCC6RZPAwG0= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-640-lvYLp7ssPHS0C76FS-GIbA-1; Mon, 26 Sep 2022 12:31:34 -0400 X-MC-Unique: lvYLp7ssPHS0C76FS-GIbA-1 Received: by mail-lj1-f200.google.com with SMTP id z34-20020a2ebe22000000b0026c18a910fcso1828428ljq.23 for ; Mon, 26 Sep 2022 09:31:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:mime-version:from:x-gm-message-state :from:to:cc:subject:date; bh=FQdy+cTxXWzwOFbzRHrnCRastf39w0bCwjHbUrtxJ+U=; b=4CWgum3PS/K9swSfzdRRr7dHvUoQ1rk3+Y2mKSUQ1cld/lgHH6LQyTi0SHFZocjvPC llKHCj9fc80zxxifJQB+Egjki6V07zamUnpae6W5fWxX3+g4xAGp5bYJN4/JGxxH3sxG vBggScvaNT9aulgir3lVhEknBk1T203s4eGupPvfQoRi6vFh0/tI62IVHGcMCwF1t5A+ p9J6JQijaFzVqzLWsxy4Atq/VpPnPnuhg1o91Mfpawva7UpV2ycvaT1ymn3Wzae+XOVU BO6zB/C+Ziwt0jUwGR+pfR4nYsTK2DNgQAcRaDJLVUBmTr+sRAE7BY/91O9FsNytIwrb dDRg== X-Gm-Message-State: ACrzQf15CMhrSdTf0Y2huHVfrzn4nvD90tKQW4BM98OqaPdtKMl2ppVL 2f6yH9uzaV2hhyrzbeL8csUU27LrOL1aaBmJRXpDO5LA9BV4lMfflahfs+wEXXL8tkVrDRS8g94 /dqELk8ag7wO8odXn6n827Sc0DwLA8g== X-Received: by 2002:a05:6512:2a92:b0:4a1:d2c9:c2d0 with SMTP id dt18-20020a0565122a9200b004a1d2c9c2d0mr2367239lfb.278.1664209892074; Mon, 26 Sep 2022 09:31:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4D6x7wjFCsWMuNjmBOgJoKrrTkFc61vg2wHLCv7aMPuKwQFljYiyZQx1XXh82rS9GzBtvuhv7oe7XpXsM2WZQ= X-Received: by 2002:a05:6512:2a92:b0:4a1:d2c9:c2d0 with SMTP id dt18-20020a0565122a9200b004a1d2c9c2d0mr2367215lfb.278.1664209891638; Mon, 26 Sep 2022 09:31:31 -0700 (PDT) Received: from 567203818698 named unknown by gmailapi.google.com with HTTPREST; Mon, 26 Sep 2022 09:31:30 -0700 From: "Oliver Steffen" X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Date: Mon, 26 Sep 2022 09:31:30 -0700 Message-ID: Subject: [PATCH v6 0/5] CI: Use Fedora 35 container for Linux jobs To: devel@edk2.groups.io Cc: Oliver Steffen , Andrew Fish , Ard Biesheuvel , Bob Feng , Gerd Hoffmann , Jiewen Yao , Jordan Justen , Leif Lindholm , Liming Gao , Michael D Kinney , Michael Kubacki , Ray Ni , Sami Mujawar , Sean Brogan , Yuwei Chen , Pawel Polawski , Ard Biesheuvel X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom containers. The container image provides the required compiler toolchains and Qemu for the supported architectures. These are then no longer downloaded at runtime, avoiding CI failures due to download errors. This approach also makes it easier to switch to other or newer compilers. It makes the CI setup independent from the default images that Azure DevOps provides. It can also help debugging CI problems, because the CI environment can be reproduced on a local machine. The container images are hosted on ghcr.io and are automatically generated using GitHub Actions. The Dockerfiles are maintained in the Tianocore "containers" repository: https://github.com/tianocore/containers. The current image is based on Fedora 35, with gcc 11. Fedora was chosen because of its fast release cycle which makes it easy to keep the toolchains up-to-date. Some further possible changes not included in this series: - The Tianocore/containers repository provides stack of layered images. One image for general purpose (build+test) and build-only jobs. The build+test image is based on the build-only one and adds Qemu, for the testing job that involve Qemu. This reduces the total download size. This was suggested by Chris Fernald. The work in the image side is done, we just need to change the CI setup accordingly. This patch set uses the build+test images for all jobs. - Further reduce the number of external dependencies that need to be downloaded at runtime. Candidates are iasl and nasm, which are already included in the image but not used yet. PR: https://github.com/tianocore/edk2/pull/2935 v6: - Include suggestions by Chris Fernald. - Added a parameter for the container image to the job template, makes usage of containers optional. - Added a parameter to configure the Python version to download. Allows using Python from the VM/container image also. - Restructure the commits (no further functional changes). v5: - Update image v4: - Use the latest image from the tianocode/containers repository which - does not include acpica-tools - includes Pyhton 3.10 v3: - Use the latest image from the tianocode/containers repository which pins down version numbers of gcc, iasl, and nasm in the Dockerfile. v2: - Images are now hosted under the Tianocore Organization https://github.com/tianocore/containers v1: - Thread: https://edk2.groups.io/g/devel/message/89058 - Images were hosted at https://github.com/osteffenrh/edk2-build-images Signed-off-by: Oliver Steffen Acked-by: Ard Biesheuvel Acked-by: Gerd Hoffmann Oliver Steffen (5): CI: make Python version configurable CI: add ~/.local/bin to PATH (Linux only) CI: Allow running in a container. CI: Use Fedora 35 container (Linux only) BaseTools: Remove ext. gcc dependencies (Linux only) .azurepipelines/Ubuntu-GCC5.yml | 2 ++ .../templates/basetools-build-steps.yml | 9 -------- .../templates/platform-build-run-steps.yml | 12 +++++++++- .../templates/pr-gate-build-job.yml | 6 +++++ .azurepipelines/templates/pr-gate-steps.yml | 12 ++++++++-- .../.azurepipelines/Ubuntu-GCC5.yml | 7 +++--- BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ------------------ BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ------------------ .../Bin/gcc_riscv64_unknown_ext_dep.yaml | 22 ------------------- .../.azurepipelines/Ubuntu-GCC5.yml | 3 +++ .../.azurepipelines/Ubuntu-GCC5.yml | 7 +++--- 11 files changed, 38 insertions(+), 84 deletions(-) delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml -- 2.37.3