[PATCH v7 1/6] CI: make Python version configurable

[PATCH v7 1/6] CI: make Python version configurable

[Oliver Steffen]

Add a new parameter "usePythonVersion" to the CI job templates.
This makes it possible to specify the version of Python to use.
The default value is ">=3.10.6".
If '' is specified, Python will not be downloaded at runtime and
the one provided by the VM/container image will be used.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 +++++-
 .azurepipelines/templates/pr-gate-build-job.yml        | 2 ++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++--
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
b/.azurepipelines/templates/platform-build-run-steps.yml
index 40a31a509fc5..6c9189a20fde 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -34,6 +34,9 @@ parameters:
 - name: extra_install_step
   type: stepList
   default: []
+- name: usePythonVersion
+  type: string
+  default: ">=3.10.6"

 steps:
 - checkout: self
@@ -42,8 +45,9 @@ steps:

 - task: UsePythonVersion@0
   inputs:
-    versionSpec: ">=3.10.6"
+    versionSpec: ${{ parameters.usePythonVersion }}
     architecture: "x64"
+  condition: ne('${{ parameters.usePythonVersion }}', '')

 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
b/.azurepipelines/templates/pr-gate-build-job.yml
index 7f88b41dc8d3..d0a6ccd10700 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -12,6 +12,7 @@ parameters:
   tool_chain_tag: ''
   vm_image: ''
   arch_list: ''
+  usePythonVersion: '>=3.10.6'

 # Build step
 jobs:
@@ -77,3 +78,4 @@ jobs:
       build_pkgs: $(Build.Pkgs)
       build_targets: $(Build.Targets)
       build_archs: ${{ parameters.arch_list }}
+      usePythonVersion: ${{ parameters.usePythonVersion }}
diff --git a/.azurepipelines/templates/pr-gate-steps.yml
b/.azurepipelines/templates/pr-gate-steps.yml
index cb431e53fcd1..bf6574b33692 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -12,6 +12,7 @@ parameters:
   build_pkgs: ''
   build_targets: ''
   build_archs: ''
+  usePythonVersion: '>=3.10.6'

 steps:
 - checkout: self
@@ -20,8 +21,9 @@ steps:

 - task: UsePythonVersion@0
   inputs:
-    versionSpec: '>=3.10.6'
-    architecture: 'x64'
+    versionSpec: ${{ parameters.usePythonVersion }}
+    architecture: "x64"
+  condition: ne('${{ parameters.usePythonVersion }}', '')

 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
-- 
2.38.1

[PATCH v7 2/6] CI: add ~/.local/bin to PATH (Linux only)

[Oliver Steffen]

Fix pip install under certain conditions.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 ++++++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
b/.azurepipelines/templates/platform-build-run-steps.yml
index 6c9189a20fde..38818ef0db44 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -39,6 +39,12 @@ parameters:
   default: ">=3.10.6"

 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
diff --git a/.azurepipelines/templates/pr-gate-steps.yml
b/.azurepipelines/templates/pr-gate-steps.yml
index bf6574b33692..7be33e425fe5 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -15,6 +15,12 @@ parameters:
   usePythonVersion: '>=3.10.6'

 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
-- 
2.38.1

[PATCH v7 3/6] CI: Allow running in a container.

[Oliver Steffen]

Add a parameter of the pr-gate-build-job template to specify a
container image URL.  If the value is not '' (default), then the
jobs will be run inside a container based on that image.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/templates/pr-gate-build-job.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
b/.azurepipelines/templates/pr-gate-build-job.yml
index d0a6ccd10700..52e6260b1c77 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -12,6 +12,7 @@ parameters:
   tool_chain_tag: ''
   vm_image: ''
   arch_list: ''
+  container: ''
   usePythonVersion: '>=3.10.6'

 # Build step
@@ -71,6 +72,9 @@ jobs:
   pool:
     vmImage: ${{ parameters.vm_image }}

+  ${{ if not(eq(parameters.container, '')) }}:
+    container: ${{ parameters.container }}
+
   steps:
   - template: pr-gate-steps.yml
     parameters:
-- 
2.38.1

[PATCH v7 4/6] CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run all Linux based jobs in a container, using a custom Fedora 35 image
(gcc 11). The image is hosted on ghcr.io and the Dockerfiles are
here: https://github.com/tianocore/containers
The version numbers of gcc, iasl, and nasm are pinned to avoid
unintended upgrades during image rebuild.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Use Python from the container image, do not download at runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/Ubuntu-GCC5.yml                        | 2 ++
 .azurepipelines/templates/basetools-build-steps.yml    | 9 ---------
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml  | 7 +++----
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 3 +++
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     | 7 +++----
 5 files changed, 11 insertions(+), 17 deletions(-)

diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index 1acd8d2a46a7..7c7732d66fc0 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -18,5 +18,7 @@ jobs:
   parameters:
     tool_chain_tag: 'GCC5'
     vm_image: 'ubuntu-latest'
+    container: 'ghcr.io/tianocore/containers/fedora-35-test:0b7ecfa'
+    usePythonVersion: ''  # use Python from the container image
     arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"

diff --git a/.azurepipelines/templates/basetools-build-steps.yml
b/.azurepipelines/templates/basetools-build-steps.yml
index d8c108c6e212..a72758bc3395 100644
--- a/.azurepipelines/templates/basetools-build-steps.yml
+++ b/.azurepipelines/templates/basetools-build-steps.yml
@@ -10,15 +10,6 @@ parameters:
   tool_chain_tag: ''

 steps:
-- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
-  - bash: sudo apt-get update
-    displayName: Update apt
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
-  - bash: sudo apt-get install gcc g++ make uuid-dev
-    displayName: Install required tools
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
 - task: CmdLine@1
   displayName: Build Base Tools from source
   inputs:
diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index b07e3199f143..e29168537048 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)

+    container: 'ghcr.io/tianocore/containers/fedora-35-test:0b7ecfa'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -85,7 +87,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
+        usePythonVersion: '' # use Python from the container image
diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 416c15e70840..88f5df8a37d9 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -75,6 +75,8 @@ jobs:
     pool:
       vmImage: $(vm_image)

+    container: 'ghcr.io/tianocore/containers/fedora-35-test:0b7ecfa'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -85,3 +87,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: '' # use Python from the container image
diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 7160d95f7e04..836e73467ca2 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -177,6 +177,8 @@ jobs:
     pool:
       vmImage: $(vm_image)

+    container: 'ghcr.io/tianocore/containers/fedora-35-test:0b7ecfa'
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -187,7 +189,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
+        usePythonVersion: '' # use Python from the container image
-- 
2.38.1

[PATCH v7 5/6] BaseTools: Remove ext. gcc dependencies (Linux only)

[Oliver Steffen]

Remove BaseTools/Bin/gcc*_linux_ext_dep.yaml to stop
downloading gcc from external locations; use the
toolchains provided by the container image instead.

The image needs to set the GCC5_*_PREFIX accordingly.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 3 files changed, 64 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
deleted file mode 100644
index ff8a9e868100..000000000000
--- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC AARCH64 compiler from Arm's release site
-# Set shell variable GCC5_AARCH64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_aarch64_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_aarch64_linux",
-  "type": "web",
-  "name": "gcc_aarch64_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_AARCH64_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
deleted file mode 100644
index 151cbfa4b532..000000000000
--- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC ARM compiler from Arm's release site
-# Set shell variable GCC5_ARM_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_arm_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_arm_linux",
-  "type": "web",
-  "name": "gcc_arm_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_ARM_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
deleted file mode 100644
index 8abbcd7ba040..000000000000
--- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-## @file
-# Download GCC RISCV64 compiler from RISC-V Organization release site
-# Set shell variable GCC5_RISCV64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_riscv64_unknown
-#
-# Copyright (c) Microsoft Corporation.
-# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All
rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_riscv64_unknown",
-  "type": "web",
-  "name": "gcc_riscv64_unknown",
-  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
-  "version": "9.2.0",
-  "compression_type": "tar",
-  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
-  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_RISCV64_INSTALL"
-}
-- 
2.38.1

[PATCH v7 6/6] CI: use latest vm_image (Linux only)

[Oliver Steffen]

Switch over to `ubuntu-latest` as the vm_image for Linux
CI jobs (EmulatorPkg, ArmVirtPkg, OvmfPkg).
The previously used ubuntu-18.04 will not be available
after Dec 1st 2022.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml  | 2 +-
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 +-
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index e29168537048..001441f86432 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -19,7 +19,7 @@ jobs:
   - job: Platform_CI
     variables:
       package: 'ArmVirtPkg'
-      vm_image: 'ubuntu-18.04'
+      vm_image: 'ubuntu-latest'
       should_run: true
       run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"

diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 88f5df8a37d9..97918c86f1e5 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -19,7 +19,7 @@ jobs:
   - job: Platform_CI
     variables:
       package: 'EmulatorPkg'
-      vm_image: 'ubuntu-18.04'
+      vm_image: 'ubuntu-latest'
       should_run: false
       run_flags: "MAKE_STARTUP_NSH=TRUE"

diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 836e73467ca2..ce1c4f851526 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -19,7 +19,7 @@ jobs:
   - job: Platform_CI
     variables:
       package: 'OvmfPkg'
-      vm_image: 'ubuntu-18.04'
+      vm_image: 'ubuntu-latest'
       should_run: true
       run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"

-- 
2.38.1

Re: [PATCH v7 1/6] CI: make Python version configurable

[Michael Kubacki]

This distributes the Python version (as a default value) across several 
files - ">=3.10.6". For maintainability, we'd like to manage versioning 
more centrally to reduce likelihood of misses on updates.

You could potentially adjust it such that the default does not specify a 
version, put the Python version in a template and call that when needed, 
etc.


Thanks,
Michael

On 11/29/2022 2:26 PM, Oliver Steffen wrote:
> Add a new parameter "usePythonVersion" to the CI job templates.
> This makes it possible to specify the version of Python to use.
> The default value is ">=3.10.6".
> If '' is specified, Python will not be downloaded at runtime and
> the one provided by the VM/container image will be used.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   .azurepipelines/templates/platform-build-run-steps.yml | 6 +++++-
>   .azurepipelines/templates/pr-gate-build-job.yml        | 2 ++
>   .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++--
>   3 files changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
> b/.azurepipelines/templates/platform-build-run-steps.yml
> index 40a31a509fc5..6c9189a20fde 100644
> --- a/.azurepipelines/templates/platform-build-run-steps.yml
> +++ b/.azurepipelines/templates/platform-build-run-steps.yml
> @@ -34,6 +34,9 @@ parameters:
>   - name: extra_install_step
>     type: stepList
>     default: []
> +- name: usePythonVersion
> +  type: string
> +  default: ">=3.10.6"
> 
>   steps:
>   - checkout: self
> @@ -42,8 +45,9 @@ steps:
> 
>   - task: UsePythonVersion@0
>     inputs:
> -    versionSpec: ">=3.10.6"
> +    versionSpec: ${{ parameters.usePythonVersion }}
>       architecture: "x64"
> +  condition: ne('${{ parameters.usePythonVersion }}', '')
> 
>   - script: pip install -r pip-requirements.txt --upgrade
>     displayName: 'Install/Upgrade pip modules'
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
> b/.azurepipelines/templates/pr-gate-build-job.yml
> index 7f88b41dc8d3..d0a6ccd10700 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -12,6 +12,7 @@ parameters:
>     tool_chain_tag: ''
>     vm_image: ''
>     arch_list: ''
> +  usePythonVersion: '>=3.10.6'
> 
>   # Build step
>   jobs:
> @@ -77,3 +78,4 @@ jobs:
>         build_pkgs: $(Build.Pkgs)
>         build_targets: $(Build.Targets)
>         build_archs: ${{ parameters.arch_list }}
> +      usePythonVersion: ${{ parameters.usePythonVersion }}
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml
> b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fcd1..bf6574b33692 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -12,6 +12,7 @@ parameters:
>     build_pkgs: ''
>     build_targets: ''
>     build_archs: ''
> +  usePythonVersion: '>=3.10.6'
> 
>   steps:
>   - checkout: self
> @@ -20,8 +21,9 @@ steps:
> 
>   - task: UsePythonVersion@0
>     inputs:
> -    versionSpec: '>=3.10.6'
> -    architecture: 'x64'
> +    versionSpec: ${{ parameters.usePythonVersion }}
> +    architecture: "x64"
> +  condition: ne('${{ parameters.usePythonVersion }}', '')
> 
>   - script: pip install -r pip-requirements.txt --upgrade
>     displayName: 'Install/Upgrade pip modules'

Re: [PATCH v7 1/6] CI: make Python version configurable

[Oliver Steffen]

Hi Michael

On Wed, Nov 30, 2022 at 3:13 PM Michael Kubacki
<mikuback@linux.microsoft.com> wrote:
>
> This distributes the Python version (as a default value) across several
> files - ">=3.10.6". For maintainability, we'd like to manage versioning
> more centrally to reduce likelihood of misses on updates.
>
> You could potentially adjust it such that the default does not specify a
> version, put the Python version in a template and call that when needed,
> etc.
Yes, it sounds like a good idea.

Can we work on this in a separate patch set?
I am experimenting with this now, but I fear this might delay
these patches too much and go for a minimal version for now.
Maybe we have a chance to merge something before the CI stops
working tomorrow.

Thanks,
  Oliver

> Thanks,
> Michael
>
> On 11/29/2022 2:26 PM, Oliver Steffen wrote:
> > Add a new parameter "usePythonVersion" to the CI job templates.
> > This makes it possible to specify the version of Python to use.
> > The default value is ">=3.10.6".
> > If '' is specified, Python will not be downloaded at runtime and
> > the one provided by the VM/container image will be used.
> >
> > Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> > ---
> >   .azurepipelines/templates/platform-build-run-steps.yml | 6 +++++-
> >   .azurepipelines/templates/pr-gate-build-job.yml        | 2 ++
> >   .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++--
> >   3 files changed, 11 insertions(+), 3 deletions(-)
> >
> > diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
> > b/.azurepipelines/templates/platform-build-run-steps.yml
> > index 40a31a509fc5..6c9189a20fde 100644
> > --- a/.azurepipelines/templates/platform-build-run-steps.yml
> > +++ b/.azurepipelines/templates/platform-build-run-steps.yml
> > @@ -34,6 +34,9 @@ parameters:
> >   - name: extra_install_step
> >     type: stepList
> >     default: []
> > +- name: usePythonVersion
> > +  type: string
> > +  default: ">=3.10.6"
> >
> >   steps:
> >   - checkout: self
> > @@ -42,8 +45,9 @@ steps:
> >
> >   - task: UsePythonVersion@0
> >     inputs:
> > -    versionSpec: ">=3.10.6"
> > +    versionSpec: ${{ parameters.usePythonVersion }}
> >       architecture: "x64"
> > +  condition: ne('${{ parameters.usePythonVersion }}', '')
> >
> >   - script: pip install -r pip-requirements.txt --upgrade
> >     displayName: 'Install/Upgrade pip modules'
> > diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
> > b/.azurepipelines/templates/pr-gate-build-job.yml
> > index 7f88b41dc8d3..d0a6ccd10700 100644
> > --- a/.azurepipelines/templates/pr-gate-build-job.yml
> > +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> > @@ -12,6 +12,7 @@ parameters:
> >     tool_chain_tag: ''
> >     vm_image: ''
> >     arch_list: ''
> > +  usePythonVersion: '>=3.10.6'
> >
> >   # Build step
> >   jobs:
> > @@ -77,3 +78,4 @@ jobs:
> >         build_pkgs: $(Build.Pkgs)
> >         build_targets: $(Build.Targets)
> >         build_archs: ${{ parameters.arch_list }}
> > +      usePythonVersion: ${{ parameters.usePythonVersion }}
> > diff --git a/.azurepipelines/templates/pr-gate-steps.yml
> > b/.azurepipelines/templates/pr-gate-steps.yml
> > index cb431e53fcd1..bf6574b33692 100644
> > --- a/.azurepipelines/templates/pr-gate-steps.yml
> > +++ b/.azurepipelines/templates/pr-gate-steps.yml
> > @@ -12,6 +12,7 @@ parameters:
> >     build_pkgs: ''
> >     build_targets: ''
> >     build_archs: ''
> > +  usePythonVersion: '>=3.10.6'
> >
> >   steps:
> >   - checkout: self
> > @@ -20,8 +21,9 @@ steps:
> >
> >   - task: UsePythonVersion@0
> >     inputs:
> > -    versionSpec: '>=3.10.6'
> > -    architecture: 'x64'
> > +    versionSpec: ${{ parameters.usePythonVersion }}
> > +    architecture: "x64"
> > +  condition: ne('${{ parameters.usePythonVersion }}', '')
> >
> >   - script: pip install -r pip-requirements.txt --upgrade
> >     displayName: 'Install/Upgrade pip modules'
>

Re: [PATCH v7 1/6] CI: make Python version configurable

[Michael Kubacki]

I'm fine for it to come in a separate patch if necessary. If you end up 
deferring it, maybe file a brief BZ to track it.

Thanks,
Michael

On 11/30/2022 12:52 PM, Oliver Steffen wrote:
> Hi Michael
> 
> On Wed, Nov 30, 2022 at 3:13 PM Michael Kubacki
> <mikuback@linux.microsoft.com> wrote:
>>
>> This distributes the Python version (as a default value) across several
>> files - ">=3.10.6". For maintainability, we'd like to manage versioning
>> more centrally to reduce likelihood of misses on updates.
>>
>> You could potentially adjust it such that the default does not specify a
>> version, put the Python version in a template and call that when needed,
>> etc.
> Yes, it sounds like a good idea.
> 
> Can we work on this in a separate patch set?
> I am experimenting with this now, but I fear this might delay
> these patches too much and go for a minimal version for now.
> Maybe we have a chance to merge something before the CI stops
> working tomorrow.
> 
> Thanks,
>    Oliver
> 
>> Thanks,
>> Michael
>>
>> On 11/29/2022 2:26 PM, Oliver Steffen wrote:
>>> Add a new parameter "usePythonVersion" to the CI job templates.
>>> This makes it possible to specify the version of Python to use.
>>> The default value is ">=3.10.6".
>>> If '' is specified, Python will not be downloaded at runtime and
>>> the one provided by the VM/container image will be used.
>>>
>>> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
>>> ---
>>>    .azurepipelines/templates/platform-build-run-steps.yml | 6 +++++-
>>>    .azurepipelines/templates/pr-gate-build-job.yml        | 2 ++
>>>    .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++--
>>>    3 files changed, 11 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/.azurepipelines/templates/platform-build-run-steps.yml
>>> b/.azurepipelines/templates/platform-build-run-steps.yml
>>> index 40a31a509fc5..6c9189a20fde 100644
>>> --- a/.azurepipelines/templates/platform-build-run-steps.yml
>>> +++ b/.azurepipelines/templates/platform-build-run-steps.yml
>>> @@ -34,6 +34,9 @@ parameters:
>>>    - name: extra_install_step
>>>      type: stepList
>>>      default: []
>>> +- name: usePythonVersion
>>> +  type: string
>>> +  default: ">=3.10.6"
>>>
>>>    steps:
>>>    - checkout: self
>>> @@ -42,8 +45,9 @@ steps:
>>>
>>>    - task: UsePythonVersion@0
>>>      inputs:
>>> -    versionSpec: ">=3.10.6"
>>> +    versionSpec: ${{ parameters.usePythonVersion }}
>>>        architecture: "x64"
>>> +  condition: ne('${{ parameters.usePythonVersion }}', '')
>>>
>>>    - script: pip install -r pip-requirements.txt --upgrade
>>>      displayName: 'Install/Upgrade pip modules'
>>> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
>>> b/.azurepipelines/templates/pr-gate-build-job.yml
>>> index 7f88b41dc8d3..d0a6ccd10700 100644
>>> --- a/.azurepipelines/templates/pr-gate-build-job.yml
>>> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
>>> @@ -12,6 +12,7 @@ parameters:
>>>      tool_chain_tag: ''
>>>      vm_image: ''
>>>      arch_list: ''
>>> +  usePythonVersion: '>=3.10.6'
>>>
>>>    # Build step
>>>    jobs:
>>> @@ -77,3 +78,4 @@ jobs:
>>>          build_pkgs: $(Build.Pkgs)
>>>          build_targets: $(Build.Targets)
>>>          build_archs: ${{ parameters.arch_list }}
>>> +      usePythonVersion: ${{ parameters.usePythonVersion }}
>>> diff --git a/.azurepipelines/templates/pr-gate-steps.yml
>>> b/.azurepipelines/templates/pr-gate-steps.yml
>>> index cb431e53fcd1..bf6574b33692 100644
>>> --- a/.azurepipelines/templates/pr-gate-steps.yml
>>> +++ b/.azurepipelines/templates/pr-gate-steps.yml
>>> @@ -12,6 +12,7 @@ parameters:
>>>      build_pkgs: ''
>>>      build_targets: ''
>>>      build_archs: ''
>>> +  usePythonVersion: '>=3.10.6'
>>>
>>>    steps:
>>>    - checkout: self
>>> @@ -20,8 +21,9 @@ steps:
>>>
>>>    - task: UsePythonVersion@0
>>>      inputs:
>>> -    versionSpec: '>=3.10.6'
>>> -    architecture: 'x64'
>>> +    versionSpec: ${{ parameters.usePythonVersion }}
>>> +    architecture: "x64"
>>> +  condition: ne('${{ parameters.usePythonVersion }}', '')
>>>
>>>    - script: pip install -r pip-requirements.txt --upgrade
>>>      displayName: 'Install/Upgrade pip modules'
>>