[PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[Oliver Steffen]

PR: https://github.com/tianocore/edk2/pull/2711

Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
Fedora 35 container. The image provides gcc 11 from Fedora for all
architectures. The external dependencies for gcc have been removed
so stuart does not download them. iasl and nasm are also
included in the image, but remain as ext-dep for now (CI jobs on
Windows need those).

The Dockerfiles are here:
https://github.com/osteffenrh/edk2-build-images, but they should, of
course, then move to a repo in the Tianocore group, or something like
that. The images are built automatically via GitHub Actions and then
pushed to ghcr.io.

Oliver Steffen (4):
  CI: don't force python verison (Linux only)
  CI: add ~/.local/bin to PATH (Linux only)
  CI: Use Fedora 35 container (Linux only)
  BaseTools: Remove ext. gcc dependencies (Linux only)

 .../templates/basetools-build-steps.yml       |  9 --------
 .../templates/platform-build-run-steps.yml    |  7 ++++++
 .../templates/pr-gate-build-job.yml           |  3 +++
 .azurepipelines/templates/pr-gate-steps.yml   |  7 ++++++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 .../.azurepipelines/Ubuntu-GCC5.yml           |  2 ++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 10 files changed, 23 insertions(+), 81 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

-- 
2.35.1

[PATCH 1/4] CI: don't force python verison (Linux only)

[Oliver Steffen]

Use the Python supplied by the container image.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 1 +
 .azurepipelines/templates/pr-gate-steps.yml            | 1 +
 2 files changed, 2 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
index 97e7faa26682..b0219bc61e67 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -44,6 +44,7 @@ steps:
   inputs:
     versionSpec: "3.8.x"
     architecture: "x64"
+  condition: not(eq('${{ parameters.tool_chain_tag }}', 'GCC5'))
 
 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index 70c19a462194..f5cf5ff0376f 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -22,6 +22,7 @@ steps:
   inputs:
     versionSpec: '3.8.x'
     architecture: 'x64'
+  condition: not(eq('${{ parameters.tool_chain_tag }}', 'GCC5'))
 
 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
-- 
2.35.1

[PATCH 2/4] CI: add ~/.local/bin to PATH (Linux only)

[Oliver Steffen]

Fix pip install.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 ++++++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
index b0219bc61e67..9d27f49a2b40 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -36,6 +36,12 @@ parameters:
   default: []
 
 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index f5cf5ff0376f..b78d2c4fff5d 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -14,6 +14,12 @@ parameters:
   build_archs: ''
 
 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
-- 
2.35.1

[PATCH 3/4] CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run all Linux based jobs in a container, using a custom Fedora 35 image
(gcc 11). The image is hosted on ghcr.io and the container-files are
here: https://github.com/osteffenrh/edk2-build-images.

Do not run apt-get to install qemu and gcc dependencies. Assume the
container image provides these.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/basetools-build-steps.yml    | 9 ---------
 .azurepipelines/templates/pr-gate-build-job.yml        | 3 +++
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml  | 6 ++----
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 ++
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     | 6 ++----
 5 files changed, 9 insertions(+), 17 deletions(-)

diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml
index d8c108c6e212..a72758bc3395 100644
--- a/.azurepipelines/templates/basetools-build-steps.yml
+++ b/.azurepipelines/templates/basetools-build-steps.yml
@@ -10,15 +10,6 @@ parameters:
   tool_chain_tag: ''
 
 steps:
-- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
-  - bash: sudo apt-get update
-    displayName: Update apt
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
-  - bash: sudo apt-get install gcc g++ make uuid-dev
-    displayName: Install required tools
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
 - task: CmdLine@1
   displayName: Build Base Tools from source
   inputs:
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index 69e4c50afdbb..716664391957 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -63,6 +63,9 @@ jobs:
   pool:
     vmImage: ${{ parameters.vm_image }}
 
+  ${{ if contains(parameters.tool_chain_tag, 'GCC') }}
+    container: 'ghcr.io/osteffenrh/edk2-build-images/fedora-35:latest'
+
   steps:
   - template: pr-gate-steps.yml
     parameters:
diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index b07e3199f143..a0a79b7108c9 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: 'ghcr.io/osteffenrh/edk2-build-images/fedora-35:latest'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -85,7 +87,3 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 416c15e70840..589eea7d1985 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: 'ghcr.io/osteffenrh/edk2-build-images/fedora-35:latest'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 6dec90d9fcaf..45a97cd246f7 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -166,6 +166,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: 'ghcr.io/osteffenrh/edk2-build-images/fedora-35:latest'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -176,7 +178,3 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
-- 
2.35.1

[PATCH 4/4] BaseTools: Remove ext. gcc dependencies (Linux only)

[Oliver Steffen]

Remove BaseTools/Bin/gcc*_linux_ext_dep.yaml to stop
downloading gcc from external locations; use the
toolchains provided by the container image instead.

The image needs to set the GCC5_*_PREFIX accordingly.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 3 files changed, 64 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
deleted file mode 100644
index ff8a9e868100..000000000000
--- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC AARCH64 compiler from Arm's release site
-# Set shell variable GCC5_AARCH64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_aarch64_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_aarch64_linux",
-  "type": "web",
-  "name": "gcc_aarch64_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_AARCH64_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
deleted file mode 100644
index 151cbfa4b532..000000000000
--- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC ARM compiler from Arm's release site
-# Set shell variable GCC5_ARM_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_arm_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_arm_linux",
-  "type": "web",
-  "name": "gcc_arm_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_ARM_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
deleted file mode 100644
index 8abbcd7ba040..000000000000
--- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-## @file
-# Download GCC RISCV64 compiler from RISC-V Organization release site
-# Set shell variable GCC5_RISCV64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_riscv64_unknown
-#
-# Copyright (c) Microsoft Corporation.
-# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_riscv64_unknown",
-  "type": "web",
-  "name": "gcc_riscv64_unknown",
-  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
-  "version": "9.2.0",
-  "compression_type": "tar",
-  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
-  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_RISCV64_INSTALL"
-}
-- 
2.35.1

Re: [PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[Ard Biesheuvel]

On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
>
> PR: https://github.com/tianocore/edk2/pull/2711
>
> Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
> Fedora 35 container.

Why?

> The image provides gcc 11 from Fedora for all
> architectures. The external dependencies for gcc have been removed
> so stuart does not download them. iasl and nasm are also
> included in the image, but remain as ext-dep for now (CI jobs on
> Windows need those).
>
> The Dockerfiles are here:
> https://github.com/osteffenrh/edk2-build-images, but they should, of
> course, then move to a repo in the Tianocore group, or something like
> that. The images are built automatically via GitHub Actions and then
> pushed to ghcr.io.
>

It would help tremendously if you could include why doing any of this is neces
> Oliver Steffen (4):
>   CI: don't force python verison (Linux only)
>   CI: add ~/.local/bin to PATH (Linux only)
>   CI: Use Fedora 35 container (Linux only)
>   BaseTools: Remove ext. gcc dependencies (Linux only)
>
>  .../templates/basetools-build-steps.yml       |  9 --------
>  .../templates/platform-build-run-steps.yml    |  7 ++++++
>  .../templates/pr-gate-build-job.yml           |  3 +++
>  .azurepipelines/templates/pr-gate-steps.yml   |  7 ++++++
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
>  BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
>  BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
>  .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  2 ++
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
>  10 files changed, 23 insertions(+), 81 deletions(-)
>  delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
>
> --
> 2.35.1
>

Re: [PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[Gerd Hoffmann]

On Tue, Apr 19, 2022 at 11:54:38PM +0200, Ard Biesheuvel wrote:
> On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > PR: https://github.com/tianocore/edk2/pull/2711
> >
> > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
> > Fedora 35 container.
> 
> Why?
> 
> > The image provides gcc 11 from Fedora for all
> > architectures. The external dependencies for gcc have been removed
> > so stuart does not download them. iasl and nasm are also
> > included in the image, but remain as ext-dep for now (CI jobs on
> > Windows need those).

^^^ this for example.

The idea is to (a) switch linux CI builds from vm images to container
images, and (b) use our own, custom container images.

This allows to roll the dependencies needed for CI into the container
images, so there is no need to download them in CI jobs.  Fixes
temporary CI failures due to network problems and download errors.

This also allows to remove any distribution-specific commands (apt-get
for example) from CI jobs.  This will be handled in the Dockerfiles used
to build our custom container images instead.

Oliver is also working on ubuntu containers, look here:

> > The Dockerfiles are here:
> > https://github.com/osteffenrh/edk2-build-images,

Fedora is a bit faster on updating stuff.  For example: ubuntu 22.04
uses gcc 11 whereas Fedora 36 uses gcc 12 (both distros are in beta
right now).  Whenever we want:

  (a) stick to ubuntu, or
  (b) switch to fedora, or
  (c) use both

is up for debate.

> > but they should, of
> > course, then move to a repo in the Tianocore group, or something like
> > that. The images are built automatically via GitHub Actions and then
> > pushed to ghcr.io.

That is the next question: where should we host the Dockerfiles and
container images?

For the Dockerfiles that should probably a git repo in the tianocore
group.

For the container images the obvious choices are github and azure.
Given that microsoft owns github I'd expect both github actions and
azure pipelines are running on the same cloud infrastructure and it
doesn't make much of a difference whenever github or azure container
registry is used to host container images.

take care,
  Gerd

Re: [PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[Ard Biesheuvel]

On Wed, 20 Apr 2022 at 09:30, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Tue, Apr 19, 2022 at 11:54:38PM +0200, Ard Biesheuvel wrote:
> > On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > PR: https://github.com/tianocore/edk2/pull/2711
> > >
> > > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
> > > Fedora 35 container.
> >
> > Why?
> >
> > > The image provides gcc 11 from Fedora for all
> > > architectures. The external dependencies for gcc have been removed
> > > so stuart does not download them. iasl and nasm are also
> > > included in the image, but remain as ext-dep for now (CI jobs on
> > > Windows need those).
>
> ^^^ this for example.
>
> The idea is to (a) switch linux CI builds from vm images to container
> images, and (b) use our own, custom container images.
>
> This allows to roll the dependencies needed for CI into the container
> images, so there is no need to download them in CI jobs.  Fixes
> temporary CI failures due to network problems and download errors.
>
> This also allows to remove any distribution-specific commands (apt-get
> for example) from CI jobs.  This will be handled in the Dockerfiles used
> to build our custom container images instead.
>
> Oliver is also working on ubuntu containers, look here:
>
> > > The Dockerfiles are here:
> > > https://github.com/osteffenrh/edk2-build-images,
>
> Fedora is a bit faster on updating stuff.  For example: ubuntu 22.04
> uses gcc 11 whereas Fedora 36 uses gcc 12 (both distros are in beta
> right now).  Whenever we want:
>
>   (a) stick to ubuntu, or
>   (b) switch to fedora, or
>   (c) use both
>
> is up for debate.
>
> > > but they should, of
> > > course, then move to a repo in the Tianocore group, or something like
> > > that. The images are built automatically via GitHub Actions and then
> > > pushed to ghcr.io.
>
> That is the next question: where should we host the Dockerfiles and
> container images?
>
> For the Dockerfiles that should probably a git repo in the tianocore
> group.
>
> For the container images the obvious choices are github and azure.
> Given that microsoft owns github I'd expect both github actions and
> azure pipelines are running on the same cloud infrastructure and it
> doesn't make much of a difference whenever github or azure container
> registry is used to host container images.
>

Thanks for the explanation.

Please put information like this in the commit logs, rather than
assuming it will obvious to everyone without providing any background.

For these changes,

Acked-by: Ard Biesheuvel <ardb@kernel.org>

Re: [edk2-devel] [PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[Oliver Steffen]

On 20.04.2022 11:16, Ard Biesheuvel wrote:
>On Wed, 20 Apr 2022 at 09:30, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>
>> On Tue, Apr 19, 2022 at 11:54:38PM +0200, Ard Biesheuvel wrote:
>> > On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
>> > >
>> > > PR: https://github.com/tianocore/edk2/pull/2711
>> > >
>> > > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
>> > > Fedora 35 container.
>> >
>> > Why?
>> >
>> > > The image provides gcc 11 from Fedora for all
>> > > architectures. The external dependencies for gcc have been removed
>> > > so stuart does not download them. iasl and nasm are also
>> > > included in the image, but remain as ext-dep for now (CI jobs on
>> > > Windows need those).
>>
>> ^^^ this for example.
>>
>> The idea is to (a) switch linux CI builds from vm images to container
>> images, and (b) use our own, custom container images.
>>
>> This allows to roll the dependencies needed for CI into the container
>> images, so there is no need to download them in CI jobs.  Fixes
>> temporary CI failures due to network problems and download errors.
>>
>> This also allows to remove any distribution-specific commands (apt-get
>> for example) from CI jobs.  This will be handled in the Dockerfiles used
>> to build our custom container images instead.
>>
>> Oliver is also working on ubuntu containers, look here:
>>
>> > > The Dockerfiles are here:
>> > > https://github.com/osteffenrh/edk2-build-images,
>>
>> Fedora is a bit faster on updating stuff.  For example: ubuntu 22.04
>> uses gcc 11 whereas Fedora 36 uses gcc 12 (both distros are in beta
>> right now).  Whenever we want:
>>
>>   (a) stick to ubuntu, or
>>   (b) switch to fedora, or
>>   (c) use both
>>
>> is up for debate.
>>
>> > > but they should, of
>> > > course, then move to a repo in the Tianocore group, or something like
>> > > that. The images are built automatically via GitHub Actions and then
>> > > pushed to ghcr.io.
>>
>> That is the next question: where should we host the Dockerfiles and
>> container images?
>>
>> For the Dockerfiles that should probably a git repo in the tianocore
>> group.
>>
>> For the container images the obvious choices are github and azure.
>> Given that microsoft owns github I'd expect both github actions and
>> azure pipelines are running on the same cloud infrastructure and it
>> doesn't make much of a difference whenever github or azure container
>> registry is used to host container images.
>>
>
>Thanks for the explanation.
>
>Please put information like this in the commit logs, rather than
>assuming it will obvious to everyone without providing any background.

Will do.

There is a related discussion about CI and containers here:
https://github.com/tianocore/edk2/discussions/2614#discussioncomment-2595253
as well as in the meeting recording attached in that post.

Cheers,
  Oliver

[PATCH 0/4] CI: Use Fedora 35 container for Linux jobs

[Oliver Steffen]

PR: https://github.com/tianocore/edk2/pull/2711

Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
Fedora 35 container. The image provides gcc 11 from Fedora for all
architectures. The external dependencies for gcc have been removed
so stuart does not download them. iasl and nasm are also
included in the image, but remain as ext-dep for now (CI jobs on
Windows need those).

The Dockerfiles are here:
https://github.com/osteffenrh/edk2-build-images, but they should, of
course, then move to a repo in the Tianocore group, or something like
that. The images are built automatically via GitHub Actions and then
pushed to ghcr.io.
Oliver Steffen (4):
  CI: don't force python verison (Linux only)
  CI: add ~/.local/bin to PATH (Linux only)
  CI: Use Fedora 35 container (Linux only)
  BaseTools: Remove ext. gcc dependencies (Linux only)

 .../templates/basetools-build-steps.yml       |  9 --------
 .../templates/platform-build-run-steps.yml    |  7 ++++++
 .../templates/pr-gate-build-job.yml           |  3 +++
 .azurepipelines/templates/pr-gate-steps.yml   |  7 ++++++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 .../.azurepipelines/Ubuntu-GCC5.yml           |  2 ++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 10 files changed, 23 insertions(+), 81 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

-- 
2.35.1