[PATCH v6 1/5] CI: make Python version configurable

[PATCH v6 1/5] CI: make Python version configurable

[Oliver Steffen]

Add a new parameter "usePythonVersion" to the CI job templates.
This makes it possible to specify the version of Python to use.
The default value is ">=3.10.6".
If '' is specified, Python will not be downloaded at runtime and
the one provided by the VM/container image will be used.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 +++++-
 .azurepipelines/templates/pr-gate-build-job.yml        | 2 ++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++--
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
b/.azurepipelines/templates/platform-build-run-steps.yml
index 40a31a509fc5..6c9189a20fde 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -34,6 +34,9 @@ parameters:
 - name: extra_install_step
   type: stepList
   default: []
+- name: usePythonVersion
+  type: string
+  default: ">=3.10.6"

 steps:
 - checkout: self
@@ -42,8 +45,9 @@ steps:

 - task: UsePythonVersion@0
   inputs:
-    versionSpec: ">=3.10.6"
+    versionSpec: ${{ parameters.usePythonVersion }}
     architecture: "x64"
+  condition: ne('${{ parameters.usePythonVersion }}', '')

 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
b/.azurepipelines/templates/pr-gate-build-job.yml
index 0e4ad019bf03..24926ed64984 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -12,6 +12,7 @@ parameters:
   tool_chain_tag: ''
   vm_image: ''
   arch_list: ''
+  usePythonVersion: '>=3.10.6'

 # Build step
 jobs:
@@ -70,3 +71,4 @@ jobs:
       build_pkgs: $(Build.Pkgs)
       build_targets: $(Build.Targets)
       build_archs: ${{ parameters.arch_list }}
+      usePythonVersion: ${{ parameters.usePythonVersion }}
diff --git a/.azurepipelines/templates/pr-gate-steps.yml
b/.azurepipelines/templates/pr-gate-steps.yml
index cb431e53fcd1..bf6574b33692 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -12,6 +12,7 @@ parameters:
   build_pkgs: ''
   build_targets: ''
   build_archs: ''
+  usePythonVersion: '>=3.10.6'

 steps:
 - checkout: self
@@ -20,8 +21,9 @@ steps:

 - task: UsePythonVersion@0
   inputs:
-    versionSpec: '>=3.10.6'
-    architecture: 'x64'
+    versionSpec: ${{ parameters.usePythonVersion }}
+    architecture: "x64"
+  condition: ne('${{ parameters.usePythonVersion }}', '')

 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
-- 
2.37.3

[PATCH v6 2/5] CI: add ~/.local/bin to PATH (Linux only)

[Oliver Steffen]

Fix pip install under certain conditions.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 ++++++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
b/.azurepipelines/templates/platform-build-run-steps.yml
index 6c9189a20fde..38818ef0db44 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -39,6 +39,12 @@ parameters:
   default: ">=3.10.6"

 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
diff --git a/.azurepipelines/templates/pr-gate-steps.yml
b/.azurepipelines/templates/pr-gate-steps.yml
index bf6574b33692..7be33e425fe5 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -15,6 +15,12 @@ parameters:
   usePythonVersion: '>=3.10.6'

 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
-- 
2.37.3

[PATCH v6 3/5] CI: Allow running in a container.

[Oliver Steffen]

Add a parameter of the pr-gate-build-job template to specify a
container image URL.  If the value is not '' (default), then the
jobs will be run inside a container based on that image.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/pr-gate-build-job.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
b/.azurepipelines/templates/pr-gate-build-job.yml
index 24926ed64984..f42731a762cb 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -12,6 +12,7 @@ parameters:
   tool_chain_tag: ''
   vm_image: ''
   arch_list: ''
+  container: ''
   usePythonVersion: '>=3.10.6'

 # Build step
@@ -64,6 +65,9 @@ jobs:
   pool:
     vmImage: ${{ parameters.vm_image }}

+  ${{ if not(eq(parameters.container, '')) }}:
+    container: ${{ parameters.container }}
+
   steps:
   - template: pr-gate-steps.yml
     parameters:
-- 
2.37.3

[PATCH v6 4/5] CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run all Linux based jobs in a container, using a custom Fedora 35 image
(gcc 11). The image is hosted on ghcr.io and the Dockerfiles are
here: https://github.com/tianocore/containers
The version numbers of gcc, iasl, and nasm are pinned to avoid
unintended upgrades during image rebuild.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Use Python from the container image, do not download at runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/Ubuntu-GCC5.yml                        | 2 ++
 .azurepipelines/templates/basetools-build-steps.yml    | 9 ---------
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml  | 7 +++----
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 3 +++
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     | 7 +++----
 5 files changed, 11 insertions(+), 17 deletions(-)

diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index 3760c6efe105..bbd2a0bff0f7 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -17,5 +17,7 @@ jobs:
   parameters:
     tool_chain_tag: 'GCC5'
     vm_image: 'ubuntu-latest'
+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+    usePythonVersion: ''  # use Python from the container image
     arch_list: "IA32,X64,ARM,AARCH64,RISCV64"

diff --git a/.azurepipelines/templates/basetools-build-steps.yml
b/.azurepipelines/templates/basetools-build-steps.yml
index d8c108c6e212..a72758bc3395 100644
--- a/.azurepipelines/templates/basetools-build-steps.yml
+++ b/.azurepipelines/templates/basetools-build-steps.yml
@@ -10,15 +10,6 @@ parameters:
   tool_chain_tag: ''

 steps:
-- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
-  - bash: sudo apt-get update
-    displayName: Update apt
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
-  - bash: sudo apt-get install gcc g++ make uuid-dev
-    displayName: Install required tools
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
 - task: CmdLine@1
   displayName: Build Base Tools from source
   inputs:
diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index b07e3199f143..137cb3b8459e 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)

+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -85,7 +87,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
+        usePythonVersion: '' # use Python from the container image
diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 416c15e70840..72d33f4df741 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)

+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -85,3 +87,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: '' # use Python from the container image
diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 7160d95f7e04..d63d39afb482 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -177,6 +177,8 @@ jobs:
     pool:
       vmImage: $(vm_image)

+    container: 'ghcr.io/tianocore/containers/fedora-35-test:46de622'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -187,7 +189,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
+        usePythonVersion: '' # use Python from the container image
-- 
2.37.3

[PATCH v6 5/5] BaseTools: Remove ext. gcc dependencies (Linux only)

[Oliver Steffen]

Remove BaseTools/Bin/gcc*_linux_ext_dep.yaml to stop
downloading gcc from external locations; use the
toolchains provided by the container image instead.

The image needs to set the GCC5_*_PREFIX accordingly.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 3 files changed, 64 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
deleted file mode 100644
index ff8a9e868100..000000000000
--- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC AARCH64 compiler from Arm's release site
-# Set shell variable GCC5_AARCH64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_aarch64_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_aarch64_linux",
-  "type": "web",
-  "name": "gcc_aarch64_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_AARCH64_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
deleted file mode 100644
index 151cbfa4b532..000000000000
--- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC ARM compiler from Arm's release site
-# Set shell variable GCC5_ARM_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_arm_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_arm_linux",
-  "type": "web",
-  "name": "gcc_arm_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_ARM_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
deleted file mode 100644
index 8abbcd7ba040..000000000000
--- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-## @file
-# Download GCC RISCV64 compiler from RISC-V Organization release site
-# Set shell variable GCC5_RISCV64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_riscv64_unknown
-#
-# Copyright (c) Microsoft Corporation.
-# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All
rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_riscv64_unknown",
-  "type": "web",
-  "name": "gcc_riscv64_unknown",
-  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
-  "version": "9.2.0",
-  "compression_type": "tar",
-  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
-  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_RISCV64_INSTALL"
-}
-- 
2.37.3

Re: [edk2-devel] [PATCH v6 5/5] BaseTools: Remove ext. gcc dependencies (Linux only)

[Chris Fernald]

Over-all I think this change is good to go. My only outstanding concern 
would be around deleting these yaml files for the compilers. I fear 
deleting them may break others work-flows and wonder if instead we 
should update the python files to first check for the presence of the 
prefix shell variable to dynamically determine if the extdep is needed 
or not similar to how the compiler plugins determine whether to use the 
extdep compiler or not.

Thanks,

Chris

On 9/26/2022 9:31 AM, Oliver Steffen wrote:
> Remove BaseTools/Bin/gcc*_linux_ext_dep.yaml to stop
> downloading gcc from external locations; use the
> toolchains provided by the container image instead.
>
> The image needs to set the GCC5_*_PREFIX accordingly.
>
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
>   BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
>   .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
>   3 files changed, 64 deletions(-)
>   delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
>   delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
>   delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
>
> diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> deleted file mode 100644
> index ff8a9e868100..000000000000
> --- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> +++ /dev/null
> @@ -1,21 +0,0 @@
> -## @file
> -# Download GCC AARCH64 compiler from Arm's release site
> -# Set shell variable GCC5_AARCH64_INSTALL to this folder
> -#
> -# This is only downloaded when a build activates scope gcc_aarch64_linux
> -#
> -# Copyright (c) Microsoft Corporation.
> -# SPDX-License-Identifier: BSD-2-Clause-Patent
> -##
> -{
> -  "scope": "gcc_aarch64_linux",
> -  "type": "web",
> -  "name": "gcc_aarch64_linux",
> -  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
> -  "version": "11.2-2022.02",
> -  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
> -  "compression_type": "tar",
> -  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
> -  "flags": ["set_shell_var", ],
> -  "var_name": "GCC5_AARCH64_INSTALL"
> -}
> diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> deleted file mode 100644
> index 151cbfa4b532..000000000000
> --- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> +++ /dev/null
> @@ -1,21 +0,0 @@
> -## @file
> -# Download GCC ARM compiler from Arm's release site
> -# Set shell variable GCC5_ARM_INSTALL to this folder
> -#
> -# This is only downloaded when a build activates scope gcc_arm_linux
> -#
> -# Copyright (c) Microsoft Corporation.
> -# SPDX-License-Identifier: BSD-2-Clause-Patent
> -##
> -{
> -  "scope": "gcc_arm_linux",
> -  "type": "web",
> -  "name": "gcc_arm_linux",
> -  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
> -  "version": "11.2-2022.02",
> -  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
> -  "compression_type": "tar",
> -  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
> -  "flags": ["set_shell_var", ],
> -  "var_name": "GCC5_ARM_INSTALL"
> -}
> diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> deleted file mode 100644
> index 8abbcd7ba040..000000000000
> --- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> +++ /dev/null
> @@ -1,22 +0,0 @@
> -## @file
> -# Download GCC RISCV64 compiler from RISC-V Organization release site
> -# Set shell variable GCC5_RISCV64_INSTALL to this folder
> -#
> -# This is only downloaded when a build activates scope gcc_riscv64_unknown
> -#
> -# Copyright (c) Microsoft Corporation.
> -# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All
> rights reserved.<BR>
> -# SPDX-License-Identifier: BSD-2-Clause-Patent
> -##
> -{
> -  "scope": "gcc_riscv64_unknown",
> -  "type": "web",
> -  "name": "gcc_riscv64_unknown",
> -  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
> -  "version": "9.2.0",
> -  "compression_type": "tar",
> -  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
> -  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
> -  "flags": ["set_shell_var", ],
> -  "var_name": "GCC5_RISCV64_INSTALL"
> -}

Re: [edk2-devel] [PATCH v6 5/5] BaseTools: Remove ext. gcc dependencies (Linux only)

[Oliver Steffen]

Quoting Chris Fernald (2022-11-28 06:20:36)
> Over-all I think this change is good to go. My only outstanding concern
> would be around deleting these yaml files for the compilers. I fear
> deleting them may break others work-flows and wonder if instead we
> should update the python files to first check for the presence of the
> prefix shell variable to dynamically determine if the extdep is needed
> or not similar to how the compiler plugins determine whether to use the
> extdep compiler or not.

Yes.  Deleting the ext_dep files is for sure not the right way.

I think we need an option to allow the user (be it human or CI script)
to choose between using the ext_dep and the tools already present on the
system (=distro tools).  This should affect gcc as well as all other
tools (iasl, nasm, ...).

I am not sure how to implement this in a good way.  I had also started a
discussion on github about that [0].  But I have to admit that I was very
busy recently and had little time to work on this (sorry).

Thanks,
    Oliver

[0] https://github.com/tianocore/edk2-pytool-extensions/discussions/323

>
> Thanks,
>
> Chris
>
> On 9/26/2022 9:31 AM, Oliver Steffen wrote:
> > Remove BaseTools/Bin/gcc*_linux_ext_dep.yaml to stop
> > downloading gcc from external locations; use the
> > toolchains provided by the container image instead.
> >
> > The image needs to set the GCC5_*_PREFIX accordingly.
> >
> > Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> > ---
> >   BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
> >   BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
> >   .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
> >   3 files changed, 64 deletions(-)
> >   delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> >   delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> >   delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> >
> > diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> > b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> > deleted file mode 100644
> > index ff8a9e868100..000000000000
> > --- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> > +++ /dev/null
> > @@ -1,21 +0,0 @@
> > -## @file
> > -# Download GCC AARCH64 compiler from Arm's release site
> > -# Set shell variable GCC5_AARCH64_INSTALL to this folder
> > -#
> > -# This is only downloaded when a build activates scope gcc_aarch64_linux
> > -#
> > -# Copyright (c) Microsoft Corporation.
> > -# SPDX-License-Identifier: BSD-2-Clause-Patent
> > -##
> > -{
> > -  "scope": "gcc_aarch64_linux",
> > -  "type": "web",
> > -  "name": "gcc_aarch64_linux",
> > -  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
> > -  "version": "11.2-2022.02",
> > -  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
> > -  "compression_type": "tar",
> > -  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
> > -  "flags": ["set_shell_var", ],
> > -  "var_name": "GCC5_AARCH64_INSTALL"
> > -}
> > diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> > b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> > deleted file mode 100644
> > index 151cbfa4b532..000000000000
> > --- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> > +++ /dev/null
> > @@ -1,21 +0,0 @@
> > -## @file
> > -# Download GCC ARM compiler from Arm's release site
> > -# Set shell variable GCC5_ARM_INSTALL to this folder
> > -#
> > -# This is only downloaded when a build activates scope gcc_arm_linux
> > -#
> > -# Copyright (c) Microsoft Corporation.
> > -# SPDX-License-Identifier: BSD-2-Clause-Patent
> > -##
> > -{
> > -  "scope": "gcc_arm_linux",
> > -  "type": "web",
> > -  "name": "gcc_arm_linux",
> > -  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
> > -  "version": "11.2-2022.02",
> > -  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
> > -  "compression_type": "tar",
> > -  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
> > -  "flags": ["set_shell_var", ],
> > -  "var_name": "GCC5_ARM_INSTALL"
> > -}
> > diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> > b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> > deleted file mode 100644
> > index 8abbcd7ba040..000000000000
> > --- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
> > +++ /dev/null
> > @@ -1,22 +0,0 @@
> > -## @file
> > -# Download GCC RISCV64 compiler from RISC-V Organization release site
> > -# Set shell variable GCC5_RISCV64_INSTALL to this folder
> > -#
> > -# This is only downloaded when a build activates scope gcc_riscv64_unknown
> > -#
> > -# Copyright (c) Microsoft Corporation.
> > -# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All
> > rights reserved.<BR>
> > -# SPDX-License-Identifier: BSD-2-Clause-Patent
> > -##
> > -{
> > -  "scope": "gcc_riscv64_unknown",
> > -  "type": "web",
> > -  "name": "gcc_riscv64_unknown",
> > -  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
> > -  "version": "9.2.0",
> > -  "compression_type": "tar",
> > -  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
> > -  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
> > -  "flags": ["set_shell_var", ],
> > -  "var_name": "GCC5_RISCV64_INSTALL"
> > -}
>

--
🎩Oliver Steffen (he/him) - Software Engineer, Virtualization
Red Hat GmbH <https://www.redhat.com/de/global/dach>,
Registered seat: Werner-von-Siemens-Ring 12, D-85630 Grasbrunn, Germany
Commercial register: Amtsgericht München/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill,
Amy Ross

Everyone has different working hours… Please do not feel obligated to
reply outside of your normal work schedule.