[PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs

public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed

* [PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs
@ 2022-09-20 14:45 Oliver Steffen
  2022-09-20 14:45 ` [PATCH v5 1/4] CI: don't force python verison (Linux only) Oliver Steffen
                   ` (3 more replies)
  0 siblings, 4 replies; 10+ messages in thread
From: Oliver Steffen @ 2022-09-20 14:45 UTC (permalink / raw)
  To: devel
  Cc: Oliver Steffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Gerd Hoffmann, Jiewen Yao, Jordan Justen, Leif Lindholm,
	Liming Gao, Michael D Kinney, Michael Kubacki, Ray Ni,
	Sami Mujawar, Sean Brogan, Yuwei Chen, Pawel Polawski,
	Ard Biesheuvel

Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
containers.

The container image provides the required compiler toolchains and Qemu
for the supported architectures. These are then no longer downloaded at
runtime, avoiding CI failures due to download errors. This approach also
makes it easier to switch to other or newer compilers. It makes the CI
setup independent from the default images that Azure DevOps provides.
It can also help debugging CI problems, because the CI environment
can be reproduced on a local machine.

The container images are hosted on ghcr.io and are automatically
generated using GitHub Actions. The Dockerfiles are maintained in the
Tianocore "containers" repository:
https://github.com/tianocore/containers.

The current image is based on Fedora 35, with gcc 11. Fedora was chosen
because of its fast release cycle which makes it easy to keep the
toolchains up-to-date.

Some further possible changes not included in this series:
- The Tianocore/containers repository provides stack of layered images.
  One image for general purpose (build+test) and build-only jobs.
  The build+test image is based on the build-only one and adds Qemu,
  for the testing job that involve Qemu. This reduces the total download
  size. This was suggested by Chris Fernald. The work in the image side
  is done, we just need to change the CI setup accordingly.
  This patch set uses the build+test images for all jobs.
- Further reduce the number of external dependencies that need to be
  downloaded at runtime. Candidates are iasl and nasm, which are already
  included in the image but not used yet.

PR: https://github.com/tianocore/edk2/pull/2935

v5:
- Update image

v4:
- Use the latest image from the tianocode/containers repository which
  - does not include acpica-tools
  - includes Pyhton 3.10

v3:
- Use the latest image from the tianocode/containers repository which
  pins down version numbers of gcc, iasl, and nasm in the Dockerfile.

v2:
- Images are now hosted under the Tianocore Organization
  https://github.com/tianocore/containers

v1:
- Thread: https://edk2.groups.io/g/devel/message/89058
- Images were hosted at https://github.com/osteffenrh/edk2-build-images

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Gerd Hoffmann <kraxel@redhat.com>

Oliver Steffen (4):
  CI: don't force python verison (Linux only)
  CI: add ~/.local/bin to PATH (Linux only)
  CI: Use Fedora 35 container (Linux only)
  BaseTools: Remove ext. gcc dependencies (Linux only)

 .../templates/basetools-build-steps.yml       |  9 --------
 .../templates/platform-build-run-steps.yml    |  7 ++++++
 .../templates/pr-gate-build-job.yml           |  3 +++
 .azurepipelines/templates/pr-gate-steps.yml   |  7 ++++++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 .../.azurepipelines/Ubuntu-GCC5.yml           |  2 ++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 10 files changed, 23 insertions(+), 81 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

-- 
2.37.3


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v5 1/4] CI: don't force python verison (Linux only)
  2022-09-20 14:45 [PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
@ 2022-09-20 14:45 ` Oliver Steffen
  2022-09-20 19:41   ` [edk2-devel] " Chris Fernald
  2022-09-20 14:45 ` [PATCH v5 2/4] CI: add ~/.local/bin to PATH " Oliver Steffen
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 10+ messages in thread
From: Oliver Steffen @ 2022-09-20 14:45 UTC (permalink / raw)
  To: devel
  Cc: Oliver Steffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Gerd Hoffmann, Jiewen Yao, Jordan Justen, Leif Lindholm,
	Liming Gao, Michael D Kinney, Michael Kubacki, Ray Ni,
	Sami Mujawar, Sean Brogan, Yuwei Chen, Pawel Polawski

Use the Python supplied by the container image.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 1 +
 .azurepipelines/templates/pr-gate-steps.yml            | 1 +
 2 files changed, 2 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
index 40a31a509fc5..3c16fa981d5a 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -44,6 +44,7 @@ steps:
   inputs:
     versionSpec: ">=3.10.6"
     architecture: "x64"
+  condition: not(eq('${{ parameters.tool_chain_tag }}', 'GCC5'))
 
 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index cb431e53fcd1..df8802718d99 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -22,6 +22,7 @@ steps:
   inputs:
     versionSpec: '>=3.10.6'
     architecture: 'x64'
+  condition: not(eq('${{ parameters.tool_chain_tag }}', 'GCC5'))
 
 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
-- 
2.37.3


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [edk2-devel] [PATCH v5 1/4] CI: don't force python verison (Linux only)
  2022-09-20 14:45 ` [PATCH v5 1/4] CI: don't force python verison (Linux only) Oliver Steffen
@ 2022-09-20 19:41   ` Chris Fernald
  2022-09-21  8:31     ` Gerd Hoffmann
  0 siblings, 1 reply; 10+ messages in thread
From: Chris Fernald @ 2022-09-20 19:41 UTC (permalink / raw)
  To: devel, osteffen
  Cc: Andrew Fish, Ard Biesheuvel, Bob Feng, Gerd Hoffmann, Jiewen Yao,
	Jordan Justen, Leif Lindholm, Liming Gao, Michael D Kinney,
	Michael Kubacki, Ray Ni, Sami Mujawar, Sean Brogan, Yuwei Chen,
	Pawel Polawski

I think we should avoid the expectation that we are in a container if 
and only if we are using GCC5. Instead I would prefer we add a boolean 
parameter to allows us to explicitly use the included python. So instead 
of the GCC check we would have this.

     condition: not(eq('${{ parameters.useIncludedPython }}', 'true'))

Chris Fernald
Microsoft

On 9/20/2022 7:45 AM, Oliver Steffen wrote:
> Use the Python supplied by the container image.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   .azurepipelines/templates/platform-build-run-steps.yml | 1 +
>   .azurepipelines/templates/pr-gate-steps.yml            | 1 +
>   2 files changed, 2 insertions(+)
> 
> diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
> index 40a31a509fc5..3c16fa981d5a 100644
> --- a/.azurepipelines/templates/platform-build-run-steps.yml
> +++ b/.azurepipelines/templates/platform-build-run-steps.yml
> @@ -44,6 +44,7 @@ steps:
>     inputs:
>       versionSpec: ">=3.10.6"
>       architecture: "x64"
> +  condition: not(eq('${{ parameters.tool_chain_tag }}', 'GCC5'))
>   
>   - script: pip install -r pip-requirements.txt --upgrade
>     displayName: 'Install/Upgrade pip modules'
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fcd1..df8802718d99 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -22,6 +22,7 @@ steps:
>     inputs:
>       versionSpec: '>=3.10.6'
>       architecture: 'x64'
> +  condition: not(eq('${{ parameters.tool_chain_tag }}', 'GCC5'))
>   
>   - script: pip install -r pip-requirements.txt --upgrade
>     displayName: 'Install/Upgrade pip modules'

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [edk2-devel] [PATCH v5 1/4] CI: don't force python verison (Linux only)
  2022-09-20 19:41   ` [edk2-devel] " Chris Fernald
@ 2022-09-21  8:31     ` Gerd Hoffmann
  0 siblings, 0 replies; 10+ messages in thread
From: Gerd Hoffmann @ 2022-09-21  8:31 UTC (permalink / raw)
  To: Chris Fernald
  Cc: devel, osteffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Jiewen Yao, Jordan Justen, Leif Lindholm, Liming Gao,
	Michael D Kinney, Michael Kubacki, Ray Ni, Sami Mujawar,
	Sean Brogan, Yuwei Chen, Pawel Polawski

On Tue, Sep 20, 2022 at 12:41:35PM -0700, Chris Fernald wrote:
> I think we should avoid the expectation that we are in a container if and
> only if we are using GCC5. Instead I would prefer we add a boolean parameter
> to allows us to explicitly use the included python. So instead of the GCC
> check we would have this.

When adding a toplevel container parameter (as suggested in the patch #3
comment) I think we can also simply use that parameter as condition.

take care,
  Gerd


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v5 2/4] CI: add ~/.local/bin to PATH (Linux only)
  2022-09-20 14:45 [PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
  2022-09-20 14:45 ` [PATCH v5 1/4] CI: don't force python verison (Linux only) Oliver Steffen
@ 2022-09-20 14:45 ` Oliver Steffen
  2022-09-20 14:45 ` [PATCH v5 3/4] CI: Use Fedora 35 container " Oliver Steffen
  2022-09-20 14:45 ` [PATCH v5 4/4] BaseTools: Remove ext. gcc dependencies " Oliver Steffen
  3 siblings, 0 replies; 10+ messages in thread
From: Oliver Steffen @ 2022-09-20 14:45 UTC (permalink / raw)
  To: devel
  Cc: Oliver Steffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Gerd Hoffmann, Jiewen Yao, Jordan Justen, Leif Lindholm,
	Liming Gao, Michael D Kinney, Michael Kubacki, Ray Ni,
	Sami Mujawar, Sean Brogan, Yuwei Chen, Pawel Polawski

Fix pip install.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 ++++++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
index 3c16fa981d5a..420d4858564f 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -36,6 +36,12 @@ parameters:
   default: []
 
 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index df8802718d99..aeff5731f2ec 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -14,6 +14,12 @@ parameters:
   build_archs: ''
 
 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
-- 
2.37.3


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH v5 3/4] CI: Use Fedora 35 container (Linux only)
  2022-09-20 14:45 [PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
  2022-09-20 14:45 ` [PATCH v5 1/4] CI: don't force python verison (Linux only) Oliver Steffen
  2022-09-20 14:45 ` [PATCH v5 2/4] CI: add ~/.local/bin to PATH " Oliver Steffen
@ 2022-09-20 14:45 ` Oliver Steffen
  2022-09-20 20:01   ` [edk2-devel] " Chris Fernald
  2022-09-20 14:45 ` [PATCH v5 4/4] BaseTools: Remove ext. gcc dependencies " Oliver Steffen
  3 siblings, 1 reply; 10+ messages in thread
From: Oliver Steffen @ 2022-09-20 14:45 UTC (permalink / raw)
  To: devel
  Cc: Oliver Steffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Gerd Hoffmann, Jiewen Yao, Jordan Justen, Leif Lindholm,
	Liming Gao, Michael D Kinney, Michael Kubacki, Ray Ni,
	Sami Mujawar, Sean Brogan, Yuwei Chen, Pawel Polawski

Run all Linux based jobs in a container, using a custom Fedora 35 image
(gcc 11). The image is hosted on ghcr.io and the Dockerfiles are
here: https://github.com/tianocore/containers
The version numbers of gcc, iasl, and nasm are pinned to avoid
unintended upgrades during image rebuild.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/basetools-build-steps.yml    | 9 ---------
 .azurepipelines/templates/pr-gate-build-job.yml        | 3 +++
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml  | 6 ++----
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 ++
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     | 6 ++----
 5 files changed, 9 insertions(+), 17 deletions(-)

diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml
index d8c108c6e212..a72758bc3395 100644
--- a/.azurepipelines/templates/basetools-build-steps.yml
+++ b/.azurepipelines/templates/basetools-build-steps.yml
@@ -10,15 +10,6 @@ parameters:
   tool_chain_tag: ''
 
 steps:
-- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
-  - bash: sudo apt-get update
-    displayName: Update apt
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
-  - bash: sudo apt-get install gcc g++ make uuid-dev
-    displayName: Install required tools
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
 - task: CmdLine@1
   displayName: Build Base Tools from source
   inputs:
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index 0e4ad019bf03..95ed03fb7e17 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -63,6 +63,9 @@ jobs:
   pool:
     vmImage: ${{ parameters.vm_image }}
 
+  ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
   steps:
   - template: pr-gate-steps.yml
     parameters:
diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index b07e3199f143..664edbfc8053 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -85,7 +87,3 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 416c15e70840..9cda8d1fbf34 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 7160d95f7e04..9c49ce16fccc 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -177,6 +177,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -187,7 +189,3 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
-- 
2.37.3


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [edk2-devel] [PATCH v5 3/4] CI: Use Fedora 35 container (Linux only)
  2022-09-20 14:45 ` [PATCH v5 3/4] CI: Use Fedora 35 container " Oliver Steffen
@ 2022-09-20 20:01   ` Chris Fernald
  2022-09-21  8:26     ` Gerd Hoffmann
  0 siblings, 1 reply; 10+ messages in thread
From: Chris Fernald @ 2022-09-20 20:01 UTC (permalink / raw)
  To: devel, osteffen
  Cc: Andrew Fish, Ard Biesheuvel, Bob Feng, Gerd Hoffmann, Jiewen Yao,
	Jordan Justen, Leif Lindholm, Liming Gao, Michael D Kinney,
	Michael Kubacki, Ray Ni, Sami Mujawar, Sean Brogan, Yuwei Chen,
	Pawel Polawski

Similar to my comment about the assumption in use of python, I believe 
the container should be determined and passed in as a parameter from the 
top level pipeline file rather then relying on tool chain tag. This lets 
the templates take action on being containerized rather then assuming 
the tool chain. Additionally, this would give consistency between the 
platform and PR templates which currently make opposite assumption about 
consumer vs template determining the container image.

Chris Fernald
Microsoft

On 9/20/2022 7:45 AM, Oliver Steffen wrote:
> Run all Linux based jobs in a container, using a custom Fedora 35 image
> (gcc 11). The image is hosted on ghcr.io and the Dockerfiles are
> here: https://github.com/tianocore/containers
> The version numbers of gcc, iasl, and nasm are pinned to avoid
> unintended upgrades during image rebuild.
> 
> Do not run apt-get in CI jobs to install qemu and gcc dependencies.
> Assume the container image provides these.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   .azurepipelines/templates/basetools-build-steps.yml    | 9 ---------
>   .azurepipelines/templates/pr-gate-build-job.yml        | 3 +++
>   ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml  | 6 ++----
>   EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 ++
>   OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     | 6 ++----
>   5 files changed, 9 insertions(+), 17 deletions(-)
> 
> diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml
> index d8c108c6e212..a72758bc3395 100644
> --- a/.azurepipelines/templates/basetools-build-steps.yml
> +++ b/.azurepipelines/templates/basetools-build-steps.yml
> @@ -10,15 +10,6 @@ parameters:
>     tool_chain_tag: ''
>   
>   steps:
> -- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
> -  - bash: sudo apt-get update
> -    displayName: Update apt
> -    condition: and(gt(variables.pkg_count, 0), succeeded())
> -
> -  - bash: sudo apt-get install gcc g++ make uuid-dev
> -    displayName: Install required tools
> -    condition: and(gt(variables.pkg_count, 0), succeeded())
> -
>   - task: CmdLine@1
>     displayName: Build Base Tools from source
>     inputs:
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
> index 0e4ad019bf03..95ed03fb7e17 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -63,6 +63,9 @@ jobs:
>     pool:
>       vmImage: ${{ parameters.vm_image }}
>   
> +  ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
> +    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
> +
>     steps:
>     - template: pr-gate-steps.yml
>       parameters:
> diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index b07e3199f143..664edbfc8053 100644
> --- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -75,6 +75,8 @@ jobs:
>       pool:
>         vmImage: $(vm_image)
>   
> +    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
> +
>       steps:
>       - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
>         parameters:
> @@ -85,7 +87,3 @@ jobs:
>           build_file: $(Build.File)
>           build_flags: $(Build.Flags)
>           run_flags: $(Run.Flags)
> -        extra_install_step:
> -        - bash: sudo apt-get install qemu
> -          displayName: Install qemu
> -          condition: and(gt(variables.pkg_count, 0), succeeded())
> diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 416c15e70840..9cda8d1fbf34 100644
> --- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -75,6 +75,8 @@ jobs:
>       pool:
>         vmImage: $(vm_image)
>   
> +    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
> +
>       steps:
>       - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
>         parameters:
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 7160d95f7e04..9c49ce16fccc 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -177,6 +177,8 @@ jobs:
>       pool:
>         vmImage: $(vm_image)
>   
> +    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
> +
>       steps:
>       - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
>         parameters:
> @@ -187,7 +189,3 @@ jobs:
>           build_file: $(Build.File)
>           build_flags: $(Build.Flags)
>           run_flags: $(Run.Flags)
> -        extra_install_step:
> -        - bash: sudo apt-get install qemu
> -          displayName: Install qemu
> -          condition: and(gt(variables.pkg_count, 0), succeeded())

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [edk2-devel] [PATCH v5 3/4] CI: Use Fedora 35 container (Linux only)
  2022-09-20 20:01   ` [edk2-devel] " Chris Fernald
@ 2022-09-21  8:26     ` Gerd Hoffmann
  2022-09-22  0:33       ` Chris Fernald
  0 siblings, 1 reply; 10+ messages in thread
From: Gerd Hoffmann @ 2022-09-21  8:26 UTC (permalink / raw)
  To: Chris Fernald
  Cc: devel, osteffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Jiewen Yao, Jordan Justen, Leif Lindholm, Liming Gao,
	Michael D Kinney, Michael Kubacki, Ray Ni, Sami Mujawar,
	Sean Brogan, Yuwei Chen, Pawel Polawski

On Tue, Sep 20, 2022 at 01:01:04PM -0700, Chris Fernald wrote:
> Similar to my comment about the assumption in use of python, I believe the
> container should be determined and passed in as a parameter from the top
> level pipeline file rather then relying on tool chain tag. This lets the
> templates take action on being containerized rather then assuming the tool
> chain. Additionally, this would give consistency between the platform and PR
> templates which currently make opposite assumption about consumer vs
> template determining the container image.

i.e. update .azurepipelines/Ubuntu-GCC5.yml and replace "vm_image: ..."
with "container: ..."?

take care,
  Gerd


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [edk2-devel] [PATCH v5 3/4] CI: Use Fedora 35 container (Linux only)
  2022-09-21  8:26     ` Gerd Hoffmann
@ 2022-09-22  0:33       ` Chris Fernald
  0 siblings, 0 replies; 10+ messages in thread
From: Chris Fernald @ 2022-09-22  0:33 UTC (permalink / raw)
  To: Gerd Hoffmann
  Cc: devel, osteffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Jiewen Yao, Jordan Justen, Leif Lindholm, Liming Gao,
	Michael D Kinney, Michael Kubacki, Ray Ni, Sami Mujawar,
	Sean Brogan, Yuwei Chen, Pawel Polawski

There is probably value in having both if we want to support various 
flavors of containers in the future, but yes adding a container option 
to the template similar to vm_image.

Thanks,
Chris

On 9/21/2022 1:26 AM, Gerd Hoffmann wrote:
> On Tue, Sep 20, 2022 at 01:01:04PM -0700, Chris Fernald wrote:
>> Similar to my comment about the assumption in use of python, I believe the
>> container should be determined and passed in as a parameter from the top
>> level pipeline file rather then relying on tool chain tag. This lets the
>> templates take action on being containerized rather then assuming the tool
>> chain. Additionally, this would give consistency between the platform and PR
>> templates which currently make opposite assumption about consumer vs
>> template determining the container image.
> 
> i.e. update .azurepipelines/Ubuntu-GCC5.yml and replace "vm_image: ..."
> with "container: ..."?
> 
> take care,
>    Gerd
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH v5 4/4] BaseTools: Remove ext. gcc dependencies (Linux only)
  2022-09-20 14:45 [PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
                   ` (2 preceding siblings ...)
  2022-09-20 14:45 ` [PATCH v5 3/4] CI: Use Fedora 35 container " Oliver Steffen
@ 2022-09-20 14:45 ` Oliver Steffen
  3 siblings, 0 replies; 10+ messages in thread
From: Oliver Steffen @ 2022-09-20 14:45 UTC (permalink / raw)
  To: devel
  Cc: Oliver Steffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
	Gerd Hoffmann, Jiewen Yao, Jordan Justen, Leif Lindholm,
	Liming Gao, Michael D Kinney, Michael Kubacki, Ray Ni,
	Sami Mujawar, Sean Brogan, Yuwei Chen, Pawel Polawski

Remove BaseTools/Bin/gcc*_linux_ext_dep.yaml to stop
downloading gcc from external locations; use the
toolchains provided by the container image instead.

The image needs to set the GCC5_*_PREFIX accordingly.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 3 files changed, 64 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
deleted file mode 100644
index ff8a9e868100..000000000000
--- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC AARCH64 compiler from Arm's release site
-# Set shell variable GCC5_AARCH64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_aarch64_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_aarch64_linux",
-  "type": "web",
-  "name": "gcc_aarch64_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_AARCH64_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
deleted file mode 100644
index 151cbfa4b532..000000000000
--- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC ARM compiler from Arm's release site
-# Set shell variable GCC5_ARM_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_arm_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_arm_linux",
-  "type": "web",
-  "name": "gcc_arm_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_ARM_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
deleted file mode 100644
index 8abbcd7ba040..000000000000
--- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-## @file
-# Download GCC RISCV64 compiler from RISC-V Organization release site
-# Set shell variable GCC5_RISCV64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_riscv64_unknown
-#
-# Copyright (c) Microsoft Corporation.
-# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_riscv64_unknown",
-  "type": "web",
-  "name": "gcc_riscv64_unknown",
-  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
-  "version": "9.2.0",
-  "compression_type": "tar",
-  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
-  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_RISCV64_INSTALL"
-}
-- 
2.37.3


^ permalink raw reply related	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2022-09-22  0:33 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-09-20 14:45 [PATCH v5 0/4] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
2022-09-20 14:45 ` [PATCH v5 1/4] CI: don't force python verison (Linux only) Oliver Steffen
2022-09-20 19:41   ` [edk2-devel] " Chris Fernald
2022-09-21  8:31     ` Gerd Hoffmann
2022-09-20 14:45 ` [PATCH v5 2/4] CI: add ~/.local/bin to PATH " Oliver Steffen
2022-09-20 14:45 ` [PATCH v5 3/4] CI: Use Fedora 35 container " Oliver Steffen
2022-09-20 20:01   ` [edk2-devel] " Chris Fernald
2022-09-21  8:26     ` Gerd Hoffmann
2022-09-22  0:33       ` Chris Fernald
2022-09-20 14:45 ` [PATCH v5 4/4] BaseTools: Remove ext. gcc dependencies " Oliver Steffen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox