[PATCH v11 00/17] CI: Use Fedora 35 container for Linux jobs

[PATCH v11 00/17] CI: Use Fedora 35 container for Linux jobs

[Oliver Steffen]

Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
containers.  This decouples the CI environment from the virtual machine
images that Azure DevOps provides.  The currently used ubuntu-18.04 image
has been deprecated for a while now and will be removed soon.

The container image provides the required compiler toolchains and Qemu
for the supported architectures. These are then no longer downloaded at
runtime, avoiding CI failures due to download errors. This approach also
makes it easier to switch to other or newer compilers. It makes the CI
setup independent from the default images that Azure DevOps provides.
It can also help debugging CI problems, because the CI environment
can be reproduced on a local machine.

The container images are hosted on ghcr.io and are automatically
generated using GitHub Actions. The Dockerfiles are maintained in the
Tianocore "containers" repository:
https://github.com/tianocore/containers.

The current image is based on Fedora 35, with gcc 11. Fedora was chosen
because of its fast release cycle which makes it easy to keep the
toolchains up-to-date.

Some further possible changes not included in this series:
- The Tianocore/containers repository provides stack of layered images.
  One image for general purpose (build+test) and build-only jobs.
  The build+test image is based on the build-only one and adds Qemu,
  for the testing job that involve Qemu.  The work in the image side
  is done, we just need to change the CI setup accordingly.
  This patch set uses the build+test images for all jobs.
- Further reduce the number of external dependencies that need to be
  downloaded at runtime. Candidates are iasl and nasm, which are already
  included in the image but not used yet.

PR: https://github.com/tianocore/edk2/pull/3904

v11:
- Rebase to current master branch
- Use lasted image, includes Qemu 7.2
- Set container image URL in a central place (the defaults.yml template)

v10:
- Split commits by package
- Use ubuntu-22.04 as vm_image (instead of the floating ubuntu-latest)
- Some fixups around the Python version template argument and defaults.yml
- Fix some commit messages
- Add license statement to defaults.yml

v9:
- Drop the "Don't install cspell" patch
- Use explicitly use ubuntu-22.04 as vm_image

v8:
- Use updated container image that contains gcc for LoongArch64.
- Remove ext_dep files for the LoongArch64 gcc.
- Don't change the scopes in .pytool/CISettings.py if running Linux CI
  it.
- Split commits that touch multiple packages.
- Use the smaller "build" image for jobs that allow it.
- Add a CI template file as a central place to define the default Python version
  and use it where needed.

v7:
- Rebase to latest master branch.
- Use latest Fedora 35 CI image.
- Stop using the ubuntu-18.04 vm_image since this will no longer be available
  after Dec 1st.  Use ubuntu-latest instead.

v6:
- Include suggestions by Chris Fernald.
- Added a parameter for the container image to the job template, makes usage
  of containers optional.
- Added a parameter to configure the Python version to download. Allows
  using Python from the VM/container image also.
- Restructure the commits (no further functional changes).

v5:
- Update image

v4:
- Use the latest image from the tianocode/containers repository which
  - does not include acpica-tools
  - includes Pyhton 3.10

v3:
- Use the latest image from the tianocode/containers repository which
  pins down version numbers of gcc, iasl, and nasm in the Dockerfile.

v2:
- Images are now hosted under the Tianocore Organization
  https://github.com/tianocore/containers

v1:
- Thread: https://edk2.groups.io/g/devel/message/89058
- Images were hosted at https://github.com/osteffenrh/edk2-build-images

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Chris Fernald <chfernal@microsoft.com>

Oliver Steffen (17):
  CI: make Python version configurable
  ArmVirtPkg: CI: use Python version from defaults template
  EmulatorPkg: CI: use Python version from defaults template
  OvmfPkg: CI: use Python version from defaults template
  CI: add ~/.local/bin to PATH (Linux only)
  CI: Allow running in a container.
  CI: Use Fedora 35 container (Linux only)
  ArmVirtPkg: CI: Use Fedora 35 container (Linux only)
  EmulatorPkg: CI: Use Fedora 35 container (Linux only)
  OvmfPkg: CI: Use Fedora 35 container (Linux only)
  .pytool: CISettings.py: don't add scopes for GCC
  BaseTools: remove ext_dep files for gcc
  ArmVirtPkg: CI: use ubuntu-22.04 vm_image (Linux only)
  EmulatorPgk: CI: use ubuntu-22.04 vm_image (Linux only)
  OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only)
  CI: use ubuntu-22.04 image (Linux only)
  OfmvPkg: CI: Qemu tests: set CPU to Broadwell

 .azurepipelines/Ubuntu-GCC5.yml               | 12 +++++-----
 .azurepipelines/Windows-VS2019.yml            |  4 ++++
 .../templates/basetools-build-steps.yml       |  9 --------
 .azurepipelines/templates/defaults.yml        | 12 ++++++++++
 .../templates/platform-build-run-steps.yml    | 12 +++++++++-
 .../templates/pr-gate-build-job.yml           |  6 +++++
 .azurepipelines/templates/pr-gate-steps.yml   | 12 ++++++++--
 .pytool/CISettings.py                         |  9 --------
 .../.azurepipelines/Ubuntu-GCC5.yml           | 12 +++++-----
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 ...gcc_loongarch64_unknown_linux_ext_dep.yaml | 22 -------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 .../.azurepipelines/Ubuntu-GCC5.yml           |  9 +++++++-
 .../.azurepipelines/Windows-VS2019.yml        |  4 ++++
 .../.azurepipelines/Ubuntu-GCC5.yml           | 12 +++++-----
 .../.azurepipelines/Windows-VS2019.yml        |  5 +++++
 OvmfPkg/PlatformCI/PlatformBuildLib.py        |  1 +
 18 files changed, 82 insertions(+), 123 deletions(-)
 create mode 100644 .azurepipelines/templates/defaults.yml
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

-- 
2.39.0

[PATCH v11 01/17] CI: make Python version configurable

[Oliver Steffen]

Add a new parameter "usePythonVersion" to the CI job templates.
This makes it possible to specify the version of Python to use.
The default value is '', in which case Python will not be downloaded
at runtime and the one provided by the VM/container image will be used.

Additionally, add a template .azurepipelines/templates/defaults.yml,
from which the default Pyhton version string can be obtained.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 .azurepipelines/Ubuntu-GCC5.yml                       |  9 +++++----
 .azurepipelines/Windows-VS2019.yml                    |  4 ++++
 .azurepipelines/templates/defaults.yml                | 11 +++++++++++
 .../templates/platform-build-run-steps.yml            |  6 +++++-
 .azurepipelines/templates/pr-gate-build-job.yml       |  2 ++
 .azurepipelines/templates/pr-gate-steps.yml           |  6 ++++--
 6 files changed, 31 insertions(+), 7 deletions(-)
 create mode 100644 .azurepipelines/templates/defaults.yml

diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index f83951eeaf86..4ed6cb601b8e 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -13,13 +13,14 @@ pr:
 - master
 - stable/*
 
+variables:
+  - template: templates/defaults.yml
+
 jobs:
 - template: templates/pr-gate-build-job.yml
   parameters:
     tool_chain_tag: 'GCC5'
     vm_image: 'ubuntu-latest'
     arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
-    extra_install_step:
-    - bash: sudo apt-get install -y lcov
-      displayName: Install Code Coverage Tools
-      condition: and(gt(variables.pkg_count, 0), succeeded())
+    usePythonVersion: ${{ variables.default_python_version }}
+
diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml
index c07e5bb43424..58bb98d42b28 100644
--- a/.azurepipelines/Windows-VS2019.yml
+++ b/.azurepipelines/Windows-VS2019.yml
@@ -12,12 +12,16 @@ pr:
 - master
 - stable/*
 
+variables:
+  - template: templates/defaults.yml
+
 jobs:
 - template: templates/pr-gate-build-job.yml
   parameters:
     tool_chain_tag: 'VS2019'
     vm_image: 'windows-2019'
     arch_list: "IA32,X64"
+    usePythonVersion: ${{ variables.default_python_version }}
     extra_install_step:
     - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
       displayName: Install Code Coverage Tool
diff --git a/.azurepipelines/templates/defaults.yml b/.azurepipelines/templates/defaults.yml
new file mode 100644
index 000000000000..b4909448bdea
--- /dev/null
+++ b/.azurepipelines/templates/defaults.yml
@@ -0,0 +1,11 @@
+## @file
+# File templates/default.yml
+#
+# template file containing common default values
+#
+# Copyright (c) Red Hat, Inc.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+variables:
+  default_python_version: ">=3.10.6"
diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
index 40a31a509fc5..8803d80cf51c 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -34,6 +34,9 @@ parameters:
 - name: extra_install_step
   type: stepList
   default: []
+- name: usePythonVersion
+  type: string
+  default: ''
 
 steps:
 - checkout: self
@@ -42,8 +45,9 @@ steps:
 
 - task: UsePythonVersion@0
   inputs:
-    versionSpec: ">=3.10.6"
+    versionSpec: ${{ parameters.usePythonVersion }}
     architecture: "x64"
+  condition: ne('${{ parameters.usePythonVersion }}', '')
 
 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index fff61a3193b0..b8573b90da1c 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -13,6 +13,7 @@ parameters:
   vm_image: ''
   arch_list: ''
   extra_install_step: []
+  usePythonVersion: ''
 
 # Build step
 jobs:
@@ -78,6 +79,7 @@ jobs:
       build_pkgs: $(Build.Pkgs)
       build_targets: $(Build.Targets)
       build_archs: ${{ parameters.arch_list }}
+      usePythonVersion: ${{ parameters.usePythonVersion }}
       extra_install_step: ${{ parameters.extra_install_step }}
 
 - job: Build_${{ parameters.tool_chain_tag }}_TARGET_CODE_COVERAGE
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index 080f60aea6ab..ebc1e86c37d4 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -12,6 +12,7 @@ parameters:
   build_pkgs: ''
   build_targets: ''
   build_archs: ''
+  usePythonVersion: ''
   extra_install_step: []
 
 steps:
@@ -21,8 +22,9 @@ steps:
 
 - task: UsePythonVersion@0
   inputs:
-    versionSpec: '>=3.10.6'
-    architecture: 'x64'
+    versionSpec: ${{ parameters.usePythonVersion }}
+    architecture: "x64"
+  condition: ne('${{ parameters.usePythonVersion }}', '')
 
 - script: pip install -r pip-requirements.txt --upgrade
   displayName: 'Install/Upgrade pip modules'
-- 
2.39.0

[PATCH v11 02/17] ArmVirtPkg: CI: use Python version from defaults template

[Oliver Steffen]

Use the default Python version from the defaults template
(.azurepipelines/templates/defaults.yml) in the Windows and
Linux CI jobs.

Previous changes to the CI job templates make it necessary
to specify a version number, if Python shall be pulled
at CI runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index b07e3199f143..5a0e589ed4a7 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -15,6 +15,9 @@ pr:
   - master
   - stable/*
 
+variables:
+  - template: ../../../.azurepipelines/templates/defaults.yml
+
 jobs:
   - job: Platform_CI
     variables:
@@ -85,6 +88,7 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: ${{ variables.default_python_version }}
         extra_install_step:
         - bash: sudo apt-get install qemu
           displayName: Install qemu
-- 
2.39.0

[PATCH v11 03/17] EmulatorPkg: CI: use Python version from defaults template

[Oliver Steffen]

Use the default Python version from the defaults template
(.azurepipelines/templates/defaults.yml) in the Windows and
Linux CI jobs.

Previous changes to the CI job templates make it necessary
to specify a version number, if Python shall be pulled
at CI runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml    | 5 +++++
 EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml | 4 ++++
 2 files changed, 9 insertions(+)

diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 416c15e70840..a32c57d4aab4 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -15,6 +15,10 @@ trigger:
 pr:
   - master
   - stable/*
+
+variables:
+  - template: ../../../.azurepipelines/templates/defaults.yml
+
 jobs:
   - job: Platform_CI
     variables:
@@ -85,3 +89,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: ${{ variables.default_python_version }}
diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
index e7ead06ae266..09960e7c7a6e 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
@@ -16,6 +16,9 @@ pr:
   - master
   - stable/*
 
+variables:
+  - template: ../../../.azurepipelines/templates/defaults.yml
+
 jobs:
   - job: Platform_CI
     variables:
@@ -128,3 +131,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: ${{ variables.default_python_version }}
-- 
2.39.0

[PATCH v11 04/17] OvmfPkg: CI: use Python version from defaults template

[Oliver Steffen]

Use the default Python version from the defaults template
(.azurepipelines/templates/defaults.yml) in the Windows and
Linux CI jobs.

Previous changes to the CI job templates make it necessary
to specify a version number, if Python shall be pulled
at CI runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml    | 4 ++++
 OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml | 5 +++++
 2 files changed, 9 insertions(+)

diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 7160d95f7e04..6dd90711ac70 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -15,6 +15,9 @@ pr:
   - master
   - stable/*
 
+variables:
+  - template: ../../../.azurepipelines/templates/defaults.yml
+
 jobs:
   - job: Platform_CI
     variables:
@@ -187,6 +190,7 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: ${{ variables.default_python_version }}
         extra_install_step:
         - bash: sudo apt-get install qemu
           displayName: Install qemu
diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
index 7d6344d6383d..7e63f419b26b 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
@@ -14,6 +14,10 @@ trigger:
 pr:
   - master
   - stable/*
+
+variables:
+  - template: ../../../.azurepipelines/templates/defaults.yml
+
 jobs:
   - job: Platform_CI
     variables:
@@ -133,6 +137,7 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
+        usePythonVersion: ${{ variables.default_python_version }}
         extra_install_step:
         - powershell: choco install qemu --version=2021.5.5; Write-Host "##vso[task.prependpath]c:\Program Files\qemu"
           displayName: Install QEMU and Set QEMU on path # friendly name displayed in the UI
-- 
2.39.0

[PATCH v11 05/17] CI: add ~/.local/bin to PATH (Linux only)

[Oliver Steffen]

Without adding ~/.local/bin to PATH, `pip install` will throw
an error when running inside a container.
Containers will be introduced to the CI in the following commits.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 .azurepipelines/templates/platform-build-run-steps.yml | 6 ++++++
 .azurepipelines/templates/pr-gate-steps.yml            | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
index 8803d80cf51c..087f460d7fec 100644
--- a/.azurepipelines/templates/platform-build-run-steps.yml
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -39,6 +39,12 @@ parameters:
   default: ''
 
 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index ebc1e86c37d4..78b2b2c8d934 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -16,6 +16,12 @@ parameters:
   extra_install_step: []
 
 steps:
+- bash: |
+    echo "##vso[task.prependpath]${HOME}/.local/bin"
+    echo "new PATH=${PATH}"
+  displayName: Set PATH
+  condition: eq('${{ parameters.tool_chain_tag }}', 'GCC5')
+
 - checkout: self
   clean: true
   fetchDepth: 1
-- 
2.39.0

[PATCH v11 06/17] CI: Allow running in a container.

[Oliver Steffen]

Add a parameter of the pr-gate-build-job template to specify a
container image URL.  If the value is not '' (default), then the
jobs will be run inside a container based on that image.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 .azurepipelines/templates/pr-gate-build-job.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index b8573b90da1c..3999bb166813 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -14,6 +14,7 @@ parameters:
   arch_list: ''
   extra_install_step: []
   usePythonVersion: ''
+  container: ''
 
 # Build step
 jobs:
@@ -72,6 +73,9 @@ jobs:
   pool:
     vmImage: ${{ parameters.vm_image }}
 
+  ${{ if not(eq(parameters.container, '')) }}:
+    container: ${{ parameters.container }}
+
   steps:
   - template: pr-gate-steps.yml
     parameters:
-- 
2.39.0

[PATCH v11 07/17] CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run all Linux based jobs in a container, using a custom Fedora 35 image
(gcc 11).  The image URL specified in the defaults.yml template, so that
all CI jobs can use it. The image is hosted on ghcr.io and the
Dockerfiles are here: https://github.com/tianocore/containers The
version numbers of gcc, iasl, and nasm are pinned to avoid unintended
upgrades during image rebuild.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Use Python from the container image, do not download at runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 .azurepipelines/Ubuntu-GCC5.yml                     | 3 ++-
 .azurepipelines/templates/basetools-build-steps.yml | 9 ---------
 .azurepipelines/templates/defaults.yml              | 1 +
 3 files changed, 3 insertions(+), 10 deletions(-)

diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index 4ed6cb601b8e..246e2d272e20 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -21,6 +21,7 @@ jobs:
   parameters:
     tool_chain_tag: 'GCC5'
     vm_image: 'ubuntu-latest'
+    container: ${{ variables.default_linux_image }}
     arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
-    usePythonVersion: ${{ variables.default_python_version }}
+    usePythonVersion: ''  # use Python from the container image
 
diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml
index d8c108c6e212..a72758bc3395 100644
--- a/.azurepipelines/templates/basetools-build-steps.yml
+++ b/.azurepipelines/templates/basetools-build-steps.yml
@@ -10,15 +10,6 @@ parameters:
   tool_chain_tag: ''
 
 steps:
-- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
-  - bash: sudo apt-get update
-    displayName: Update apt
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
-  - bash: sudo apt-get install gcc g++ make uuid-dev
-    displayName: Install required tools
-    condition: and(gt(variables.pkg_count, 0), succeeded())
-
 - task: CmdLine@1
   displayName: Build Base Tools from source
   inputs:
diff --git a/.azurepipelines/templates/defaults.yml b/.azurepipelines/templates/defaults.yml
index b4909448bdea..74d6b417839d 100644
--- a/.azurepipelines/templates/defaults.yml
+++ b/.azurepipelines/templates/defaults.yml
@@ -9,3 +9,4 @@
 
 variables:
   default_python_version: ">=3.10.6"
+  default_linux_image: "ghcr.io/tianocore/containers/fedora-35-test:47addc9"
-- 
2.39.0

[PATCH v11 08/17] ArmVirtPkg: CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run the Linux jobs of the ArmVirtPkg platform CI inside a container,
in the same way the general CI does now. Make use of the default image
specified in the defaults.yml template.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Use Python from the container image, do not download at runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 5a0e589ed4a7..c0a2e0f20d4f 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -78,6 +78,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: ${{ variables.default_linux_image }}
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -88,8 +90,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        usePythonVersion: ${{ variables.default_python_version }}
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
+        usePythonVersion: '' # use Python from the container image
-- 
2.39.0

[PATCH v11 09/17] EmulatorPkg: CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run the Linux jobs of the EmulatorPkg platform CI inside a container,
in the same way the general CI does now. Make use of the default image
specified in the defaults.yml template.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Use Python from the container image, do not download at runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index a32c57d4aab4..3861457ac70a 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -79,6 +79,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: ${{ variables.default_linux_image }}
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -89,4 +91,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        usePythonVersion: ${{ variables.default_python_version }}
+        usePythonVersion: '' # use Python from the container image
-- 
2.39.0

[PATCH v11 10/17] OvmfPkg: CI: Use Fedora 35 container (Linux only)

[Oliver Steffen]

Run the Linux jobs of the OvmfPkg platform CI inside a container,
in the same way the general CI does now. Make use of the default image
specified in the defaults.yml template.

Do not run apt-get in CI jobs to install qemu and gcc dependencies.
Assume the container image provides these.

Use Python from the container image, do not download at runtime.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 6dd90711ac70..85be431ca5e7 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -180,6 +180,8 @@ jobs:
     pool:
       vmImage: $(vm_image)
 
+    container: ${{ variables.default_linux_image }}
+
     steps:
     - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
       parameters:
@@ -190,8 +192,4 @@ jobs:
         build_file: $(Build.File)
         build_flags: $(Build.Flags)
         run_flags: $(Run.Flags)
-        usePythonVersion: ${{ variables.default_python_version }}
-        extra_install_step:
-        - bash: sudo apt-get install qemu
-          displayName: Install qemu
-          condition: and(gt(variables.pkg_count, 0), succeeded())
+        usePythonVersion: '' # use Python from the container image
-- 
2.39.0

[PATCH v11 11/17] .pytool: CISettings.py: don't add scopes for GCC

[Oliver Steffen]

All ext_dep.yml files for gcc have been removed and gcc
is expected to be installed on the system (GCC5_*_PREFIX
may indicate the location).
No need to adjust the toolchain scopes for Linux builds
anymore.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 .pytool/CISettings.py | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py
index 76ac2b09dba6..d87c8e838e61 100644
--- a/.pytool/CISettings.py
+++ b/.pytool/CISettings.py
@@ -169,15 +169,6 @@ class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSettingsManag
             else:
                 logging.warning("Falling back to using in-tree BaseTools")
 
-            if is_linux and self.ActualToolChainTag.upper().startswith("GCC"):
-                if "AARCH64" in self.ActualArchitectures:
-                    scopes += ("gcc_aarch64_linux",)
-                if "ARM" in self.ActualArchitectures:
-                    scopes += ("gcc_arm_linux",)
-                if "RISCV64" in self.ActualArchitectures:
-                    scopes += ("gcc_riscv64_unknown",)
-                if "LOONGARCH64" in self.ActualArchitectures:
-                    scopes += ("gcc_loongarch64_unknown_linux",)
             self.ActualScopes = scopes
         return self.ActualScopes
 
-- 
2.39.0

[PATCH v11 12/17] BaseTools: remove ext_dep files for gcc

[Oliver Steffen]

Remove BaseTools/Bin/gcc_*_ext_dep.yaml to stop
downloading gcc from external locations; use the
gcc provided by the container image instead.
The container image sets the variable GCC5_*_PREFIX accordingly.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 ...gcc_loongarch64_unknown_linux_ext_dep.yaml | 22 -------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 4 files changed, 86 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
deleted file mode 100644
index ff8a9e868100..000000000000
--- a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC AARCH64 compiler from Arm's release site
-# Set shell variable GCC5_AARCH64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_aarch64_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_aarch64_linux",
-  "type": "web",
-  "name": "gcc_aarch64_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "52dbac3eb71dbe0916f60a8c5ab9b7dc9b66b3ce513047baa09fae56234e53f3",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_AARCH64_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
deleted file mode 100644
index 151cbfa4b532..000000000000
--- a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-## @file
-# Download GCC ARM compiler from Arm's release site
-# Set shell variable GCC5_ARM_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_arm_linux
-#
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_arm_linux",
-  "type": "web",
-  "name": "gcc_arm_linux",
-  "source": "https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf.tar.xz",
-  "version": "11.2-2022.02",
-  "sha256": "c254f7199261fe76c32ef42187502839bda7efad0a66646cf739d074eff45fad",
-  "compression_type": "tar",
-  "internal_path": "/gcc-arm-11.2-2022.02-x86_64-arm-none-linux-gnueabihf/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_ARM_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml b/BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml
deleted file mode 100644
index 6ebc93a07c66..000000000000
--- a/BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-## @file
-# Download GCC LoongArch64 compiler from LoongArch GitHub release site
-# Set shell variable GCC5_LOONGARCH64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_loongarch64_unknown_linux
-#
-# Copyright (c) Microsoft Corporation.
-# Copyright (c) 2022, Loongson Technology Corporation Limited. All rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_loongarch64_unknown_linux",
-  "type": "web",
-  "name": "gcc_loongarch64_unknown_linux",
-  "source":"https://github.com/loongson/build-tools/releases/download/2022.09.06/loongarch64-clfs-6.3-cross-tools-c-only.tar.xz",
-  "version": "13.0.0",
-  "sha256":"744cbb50c6ef07c96551bddf07a8b85cee3250bb18b6040f6ed051e389514951",
-  "compression_type": "tar",
-  "internal_path": "/cross-tools/",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_LOONGARCH64_INSTALL"
-}
diff --git a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml b/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
deleted file mode 100644
index 8abbcd7ba040..000000000000
--- a/BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-## @file
-# Download GCC RISCV64 compiler from RISC-V Organization release site
-# Set shell variable GCC5_RISCV64_INSTALL to this folder
-#
-# This is only downloaded when a build activates scope gcc_riscv64_unknown
-#
-# Copyright (c) Microsoft Corporation.
-# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-##
-{
-  "scope": "gcc_riscv64_unknown",
-  "type": "web",
-  "name": "gcc_riscv64_unknown",
-  "source": "https://raw.githubusercontent.com/riscv/riscv-uefi-edk2-docs/master/gcc-riscv-edk2-ci-toolchain/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu.tar.xz",
-  "version": "9.2.0",
-  "compression_type": "tar",
-  "sha256": "28373643b69f0ce008273c3dc63f172aa1121952f1b9ae94d7485ac94af7f344",
-  "internal_path": "/gcc-riscv-9.2.0-2020.04-x86_64_riscv64-unknown-gnu",
-  "flags": ["set_shell_var", ],
-  "var_name": "GCC5_RISCV64_INSTALL"
-}
-- 
2.39.0

[PATCH v11 13/17] ArmVirtPkg: CI: use ubuntu-22.04 vm_image (Linux only)

[Oliver Steffen]

Switch over to ubuntu-22.04 as the vm_image for Linux CI jobs.  The
previously used ubuntu-18.04 which is not available anymore since
Dec 1st 2022.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index c0a2e0f20d4f..5fa7518d2c5e 100644
--- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -22,7 +22,7 @@ jobs:
   - job: Platform_CI
     variables:
       package: 'ArmVirtPkg'
-      vm_image: 'ubuntu-18.04'
+      vm_image: 'ubuntu-22.04'
       should_run: true
       run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
 
-- 
2.39.0

[PATCH v11 14/17] EmulatorPgk: CI: use ubuntu-22.04 vm_image (Linux only)

[Oliver Steffen]

Switch over to ubuntu-22.04 as the vm_image for Linux CI jobs.  The
previously used ubuntu-18.04 which is not available anymore since
Dec 1st 2022.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 3861457ac70a..8af4c9c778cb 100644
--- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -23,7 +23,7 @@ jobs:
   - job: Platform_CI
     variables:
       package: 'EmulatorPkg'
-      vm_image: 'ubuntu-18.04'
+      vm_image: 'ubuntu-22.04'
       should_run: false
       run_flags: "MAKE_STARTUP_NSH=TRUE"
 
-- 
2.39.0

[PATCH v11 15/17] OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only)

[Oliver Steffen]

Switch over to ubuntu-22.04 as the vm_image for Linux CI jobs.  The
previously used ubuntu-18.04 which is not available anymore since
Dec 1st 2022.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
index 85be431ca5e7..1cf9bdf8b9c2 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
@@ -22,7 +22,7 @@ jobs:
   - job: Platform_CI
     variables:
       package: 'OvmfPkg'
-      vm_image: 'ubuntu-18.04'
+      vm_image: 'ubuntu-22.04'
       should_run: true
       run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
 
-- 
2.39.0

[PATCH v11 16/17] CI: use ubuntu-22.04 image (Linux only)

[Oliver Steffen]

Use the same vm_image as the other Linux CIs in
ArmVirtPkg, EmulatorPkg, and OvmfPkg.
Switch over to ubuntu-22.04 form ubuntu-latest.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 .azurepipelines/Ubuntu-GCC5.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index 246e2d272e20..d93604fadbdb 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -20,7 +20,7 @@ jobs:
 - template: templates/pr-gate-build-job.yml
   parameters:
     tool_chain_tag: 'GCC5'
-    vm_image: 'ubuntu-latest'
+    vm_image: 'ubuntu-22.04'
     container: ${{ variables.default_linux_image }}
     arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
     usePythonVersion: ''  # use Python from the container image
-- 
2.39.0

[PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Oliver Steffen]

Work around a bug in Ovmf that causes Ovmf to crash on startup
when running in recent versions of Qemu.  Setting the cpu model
to something older, for example "Broadwell" works around the issue.

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
---
 OvmfPkg/PlatformCI/PlatformBuildLib.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py b/OvmfPkg/PlatformCI/PlatformBuildLib.py
index bfef9849c749..93f7d7aa13d6 100644
--- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
+++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
@@ -194,6 +194,7 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
         cmd = "qemu-system-x86_64"
         args  = "-debugcon stdio"                                           # write messages to stdio
         args += " -global isa-debugcon.iobase=0x402"                        # debug messages out thru virtual io port
+        args += " -cpu Broadwell"                                           # workaround Qvmf bug
         args += " -net none"                                                # turn off network
         args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk" # Mount disk with startup.nsh
 
-- 
2.39.0

Re: [edk2-devel] [PATCH v11 04/17] OvmfPkg: CI: use Python version from defaults template

[Yao, Jiewen]

Acked-by: Jiewen Yao <Jiewen.yao@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Oliver
> Steffen
> Sent: Tuesday, January 17, 2023 1:41 AM
> To: devel@edk2.groups.io
> Cc: Andrew Fish <afish@apple.com>; Ard Biesheuvel
> <ardb+tianocore@kernel.org>; Feng, Bob C <bob.c.feng@intel.com>; Bi,
> Dandan <dandan.bi@intel.com>; Gerd Hoffmann <kraxel@redhat.com>;
> Wang, Jian J <jian.j.wang@intel.com>; Yao, Jiewen <jiewen.yao@intel.com>;
> Justen, Jordan L <jordan.l.justen@intel.com>; Leif Lindholm
> <quic_llindhol@quicinc.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> Mujawar <sami.mujawar@arm.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com; Oliver
> Steffen <osteffen@redhat.com>; Kubacki, Michael
> <michael.kubacki@microsoft.com>
> Subject: [edk2-devel] [PATCH v11 04/17] OvmfPkg: CI: use Python version
> from defaults template
> 
> Use the default Python version from the defaults template
> (.azurepipelines/templates/defaults.yml) in the Windows and
> Linux CI jobs.
> 
> Previous changes to the CI job templates make it necessary
> to specify a version number, if Python shall be pulled
> at CI runtime.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
> ---
>  OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml    | 4 ++++
>  OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml | 5 +++++
>  2 files changed, 9 insertions(+)
> 
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 7160d95f7e04..6dd90711ac70 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -15,6 +15,9 @@ pr:
>    - master
>    - stable/*
> 
> +variables:
> +  - template: ../../../.azurepipelines/templates/defaults.yml
> +
>  jobs:
>    - job: Platform_CI
>      variables:
> @@ -187,6 +190,7 @@ jobs:
>          build_file: $(Build.File)
>          build_flags: $(Build.Flags)
>          run_flags: $(Run.Flags)
> +        usePythonVersion: ${{ variables.default_python_version }}
>          extra_install_step:
>          - bash: sudo apt-get install qemu
>            displayName: Install qemu
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> index 7d6344d6383d..7e63f419b26b 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> @@ -14,6 +14,10 @@ trigger:
>  pr:
>    - master
>    - stable/*
> +
> +variables:
> +  - template: ../../../.azurepipelines/templates/defaults.yml
> +
>  jobs:
>    - job: Platform_CI
>      variables:
> @@ -133,6 +137,7 @@ jobs:
>          build_file: $(Build.File)
>          build_flags: $(Build.Flags)
>          run_flags: $(Run.Flags)
> +        usePythonVersion: ${{ variables.default_python_version }}
>          extra_install_step:
>          - powershell: choco install qemu --version=2021.5.5; Write-Host
> "##vso[task.prependpath]c:\Program Files\qemu"
>            displayName: Install QEMU and Set QEMU on path # friendly name
> displayed in the UI
> --
> 2.39.0
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH v11 10/17] OvmfPkg: CI: Use Fedora 35 container (Linux only)

[Yao, Jiewen]

Acked-by: Jiewen Yao <Jiewen.yao@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Oliver
> Steffen
> Sent: Tuesday, January 17, 2023 1:41 AM
> To: devel@edk2.groups.io
> Cc: Andrew Fish <afish@apple.com>; Ard Biesheuvel
> <ardb+tianocore@kernel.org>; Feng, Bob C <bob.c.feng@intel.com>; Bi,
> Dandan <dandan.bi@intel.com>; Gerd Hoffmann <kraxel@redhat.com>;
> Wang, Jian J <jian.j.wang@intel.com>; Yao, Jiewen <jiewen.yao@intel.com>;
> Justen, Jordan L <jordan.l.justen@intel.com>; Leif Lindholm
> <quic_llindhol@quicinc.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> Mujawar <sami.mujawar@arm.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com; Oliver
> Steffen <osteffen@redhat.com>
> Subject: [edk2-devel] [PATCH v11 10/17] OvmfPkg: CI: Use Fedora 35
> container (Linux only)
> 
> Run the Linux jobs of the OvmfPkg platform CI inside a container,
> in the same way the general CI does now. Make use of the default image
> specified in the defaults.yml template.
> 
> Do not run apt-get in CI jobs to install qemu and gcc dependencies.
> Assume the container image provides these.
> 
> Use Python from the container image, do not download at runtime.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>  OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 8 +++-----
>  1 file changed, 3 insertions(+), 5 deletions(-)
> 
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 6dd90711ac70..85be431ca5e7 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -180,6 +180,8 @@ jobs:
>      pool:
>        vmImage: $(vm_image)
> 
> +    container: ${{ variables.default_linux_image }}
> +
>      steps:
>      - template: ../../../.azurepipelines/templates/platform-build-run-
> steps.yml
>        parameters:
> @@ -190,8 +192,4 @@ jobs:
>          build_file: $(Build.File)
>          build_flags: $(Build.Flags)
>          run_flags: $(Run.Flags)
> -        usePythonVersion: ${{ variables.default_python_version }}
> -        extra_install_step:
> -        - bash: sudo apt-get install qemu
> -          displayName: Install qemu
> -          condition: and(gt(variables.pkg_count, 0), succeeded())
> +        usePythonVersion: '' # use Python from the container image
> --
> 2.39.0
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH v11 15/17] OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only)

[Yao, Jiewen]

Acked-by: Jiewen Yao <Jiewen.yao@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Oliver
> Steffen
> Sent: Tuesday, January 17, 2023 1:41 AM
> To: devel@edk2.groups.io
> Cc: Andrew Fish <afish@apple.com>; Ard Biesheuvel
> <ardb+tianocore@kernel.org>; Feng, Bob C <bob.c.feng@intel.com>; Bi,
> Dandan <dandan.bi@intel.com>; Gerd Hoffmann <kraxel@redhat.com>;
> Wang, Jian J <jian.j.wang@intel.com>; Yao, Jiewen <jiewen.yao@intel.com>;
> Justen, Jordan L <jordan.l.justen@intel.com>; Leif Lindholm
> <quic_llindhol@quicinc.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> Mujawar <sami.mujawar@arm.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com; Oliver
> Steffen <osteffen@redhat.com>; Kubacki, Michael
> <michael.kubacki@microsoft.com>
> Subject: [edk2-devel] [PATCH v11 15/17] OvmfPkg: CI: use ubuntu-22.04
> vm_image (Linux only)
> 
> Switch over to ubuntu-22.04 as the vm_image for Linux CI jobs.  The
> previously used ubuntu-18.04 which is not available anymore since
> Dec 1st 2022.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
> ---
>  OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 85be431ca5e7..1cf9bdf8b9c2 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -22,7 +22,7 @@ jobs:
>    - job: Platform_CI
>      variables:
>        package: 'OvmfPkg'
> -      vm_image: 'ubuntu-18.04'
> +      vm_image: 'ubuntu-22.04'
>        should_run: true
>        run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
> 
> --
> 2.39.0
> 
> 
> 
> 
> 

Re: [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Yao, Jiewen]

Not sure the detail.

Could you please add more comment, such as, what issue? Why crash? Why changing to Broadwell can resolve it?
any URL?



> -----Original Message-----
> From: Oliver Steffen <osteffen@redhat.com>
> Sent: Tuesday, January 17, 2023 1:41 AM
> To: devel@edk2.groups.io
> Cc: Andrew Fish <afish@apple.com>; Ard Biesheuvel
> <ardb+tianocore@kernel.org>; Feng, Bob C <bob.c.feng@intel.com>; Bi,
> Dandan <dandan.bi@intel.com>; Gerd Hoffmann <kraxel@redhat.com>;
> Wang, Jian J <jian.j.wang@intel.com>; Yao, Jiewen <jiewen.yao@intel.com>;
> Justen, Jordan L <jordan.l.justen@intel.com>; Leif Lindholm
> <quic_llindhol@quicinc.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> Mujawar <sami.mujawar@arm.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com; Oliver
> Steffen <osteffen@redhat.com>
> Subject: [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell
> 
> Work around a bug in Ovmf that causes Ovmf to crash on startup
> when running in recent versions of Qemu.  Setting the cpu model
> to something older, for example "Broadwell" works around the issue.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>  OvmfPkg/PlatformCI/PlatformBuildLib.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> index bfef9849c749..93f7d7aa13d6 100644
> --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> @@ -194,6 +194,7 @@ class PlatformBuilder( UefiBuilder,
> BuildSettingsManager):
>          cmd = "qemu-system-x86_64"
>          args  = "-debugcon stdio"                                           # write messages to
> stdio
>          args += " -global isa-debugcon.iobase=0x402"                        # debug
> messages out thru virtual io port
> +        args += " -cpu Broadwell"                                           # workaround Qvmf
> bug
>          args += " -net none"                                                # turn off network
>          args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk" #
> Mount disk with startup.nsh
> 
> --
> 2.39.0

Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

It is this one:
https://bugzilla.tianocore.org/show_bug.cgi?id=4251
Should have put that into the commit message, sorry.


On Tue, Jan 17, 2023 at 1:16 AM Yao, Jiewen <jiewen.yao@intel.com> wrote:

> Not sure the detail.
>
> Could you please add more comment, such as, what issue? Why crash? Why
> changing to Broadwell can resolve it?
> any URL?
>
>
>
> > -----Original Message-----
> > From: Oliver Steffen <osteffen@redhat.com>
> > Sent: Tuesday, January 17, 2023 1:41 AM
> > To: devel@edk2.groups.io
> > Cc: Andrew Fish <afish@apple.com>; Ard Biesheuvel
> > <ardb+tianocore@kernel.org>; Feng, Bob C <bob.c.feng@intel.com>; Bi,
> > Dandan <dandan.bi@intel.com>; Gerd Hoffmann <kraxel@redhat.com>;
> > Wang, Jian J <jian.j.wang@intel.com>; Yao, Jiewen <jiewen.yao@intel.com
> >;
> > Justen, Jordan L <jordan.l.justen@intel.com>; Leif Lindholm
> > <quic_llindhol@quicinc.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> > Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki
> > <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> > Mujawar <sami.mujawar@arm.com>; Sean Brogan
> > <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> > Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com; Oliver
> > Steffen <osteffen@redhat.com>
> > Subject: [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell
> >
> > Work around a bug in Ovmf that causes Ovmf to crash on startup
> > when running in recent versions of Qemu.  Setting the cpu model
> > to something older, for example "Broadwell" works around the issue.
> >
> > Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> > ---
> >  OvmfPkg/PlatformCI/PlatformBuildLib.py | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > index bfef9849c749..93f7d7aa13d6 100644
> > --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > @@ -194,6 +194,7 @@ class PlatformBuilder( UefiBuilder,
> > BuildSettingsManager):
> >          cmd = "qemu-system-x86_64"
> >          args  = "-debugcon stdio"
>      # write messages to
> > stdio
> >          args += " -global isa-debugcon.iobase=0x402"
>     # debug
> > messages out thru virtual io port
> > +        args += " -cpu Broadwell"
>      # workaround Qvmf
> > bug
> >          args += " -net none"
>     # turn off network
> >          args += f" -drive
> file=fat:rw:{VirtualDrive},format=raw,media=disk" #
> > Mount disk with startup.nsh
> >
> > --
> > 2.39.0
>
>
>
> 
>
>
>

[-- Attachment #2: Type: text/html, Size: 5234 bytes --]

Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Yao, Jiewen]

[-- Attachment #1: Type: text/plain, Size: 4495 bytes --]

I have seen Gerd sent out V4 patch set for #4251. Is that not enough?

In general, I prefer no work-around, unless there is strong reason.

I would like to have Gerd and Ard to feedback.

Thank you
Yao, Jiewen

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Oliver Steffen
Sent: Tuesday, January 17, 2023 6:37 PM
To: devel@edk2.groups.io; Yao, Jiewen <jiewen.yao@intel.com>
Cc: Andrew Fish <afish@apple.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Gerd Hoffmann <kraxel@redhat.com>; Wang, Jian J <jian.j.wang@intel.com>; Justen, Jordan L <jordan.l.justen@intel.com>; Leif Lindholm <quic_llindhol@quicinc.com>; Gao, Liming <gaoliming@byosoft.com.cn>; Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami Mujawar <sami.mujawar@arm.com>; Sean Brogan <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>; Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com
Subject: Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

It is this one:
https://bugzilla.tianocore.org/show_bug.cgi?id=4251
Should have put that into the commit message, sorry.


On Tue, Jan 17, 2023 at 1:16 AM Yao, Jiewen <jiewen.yao@intel.com<mailto:jiewen.yao@intel.com>> wrote:
Not sure the detail.

Could you please add more comment, such as, what issue? Why crash? Why changing to Broadwell can resolve it?
any URL?



> -----Original Message-----
> From: Oliver Steffen <osteffen@redhat.com<mailto:osteffen@redhat.com>>
> Sent: Tuesday, January 17, 2023 1:41 AM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
> Cc: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>; Ard Biesheuvel
> <ardb+tianocore@kernel.org<mailto:ardb%2Btianocore@kernel.org>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Bi,
> Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gerd Hoffmann <kraxel@redhat.com<mailto:kraxel@redhat.com>>;
> Wang, Jian J <jian.j.wang@intel.com<mailto:jian.j.wang@intel.com>>; Yao, Jiewen <jiewen.yao@intel.com<mailto:jiewen.yao@intel.com>>;
> Justen, Jordan L <jordan.l.justen@intel.com<mailto:jordan.l.justen@intel.com>>; Leif Lindholm
> <quic_llindhol@quicinc.com<mailto:quic_llindhol@quicinc.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>;
> Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Michael Kubacki
> <mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Sami
> Mujawar <sami.mujawar@arm.com<mailto:sami.mujawar@arm.com>>; Sean Brogan
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Chen, Christine <yuwei.chen@intel.com<mailto:yuwei.chen@intel.com>>;
> Pawel Polawski <ppolawsk@redhat.com<mailto:ppolawsk@redhat.com>>; chris.fernald@outlook.com<mailto:chris.fernald@outlook.com>; Oliver
> Steffen <osteffen@redhat.com<mailto:osteffen@redhat.com>>
> Subject: [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell
>
> Work around a bug in Ovmf that causes Ovmf to crash on startup
> when running in recent versions of Qemu.  Setting the cpu model
> to something older, for example "Broadwell" works around the issue.
>
> Signed-off-by: Oliver Steffen <osteffen@redhat.com<mailto:osteffen@redhat.com>>
> ---
>  OvmfPkg/PlatformCI/PlatformBuildLib.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> index bfef9849c749..93f7d7aa13d6 100644
> --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> @@ -194,6 +194,7 @@ class PlatformBuilder( UefiBuilder,
> BuildSettingsManager):
>          cmd = "qemu-system-x86_64"
>          args  = "-debugcon stdio"                                           # write messages to
> stdio
>          args += " -global isa-debugcon.iobase=0x402"                        # debug
> messages out thru virtual io port
> +        args += " -cpu Broadwell"                                           # workaround Qvmf
> bug
>          args += " -net none"                                                # turn off network
>          args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk" #
> Mount disk with startup.nsh
>
> --
> 2.39.0







[-- Attachment #2: Type: text/html, Size: 9738 bytes --]

Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Gerd Hoffmann]

On Tue, Jan 17, 2023 at 12:20:16PM +0000, Yao, Jiewen wrote:
> I have seen Gerd sent out V4 patch set for #4251. Is that not enough?

If that is merged first patch 17 should not be needed.

take care,
  Gerd

Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Yao, Jiewen]

Cool!
Then I suggest to drop patch 17.

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Gerd
> Hoffmann
> Sent: Tuesday, January 17, 2023 8:39 PM
> To: Yao, Jiewen <jiewen.yao@intel.com>
> Cc: devel@edk2.groups.io; osteffen@redhat.com; Andrew Fish
> <afish@apple.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Feng,
> Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Wang,
> Jian J <jian.j.wang@intel.com>; Justen, Jordan L <jordan.l.justen@intel.com>;
> Leif Lindholm <quic_llindhol@quicinc.com>; Gao, Liming
> <gaoliming@byosoft.com.cn>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> Mujawar <sami.mujawar@arm.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com
> Subject: Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set
> CPU to Broadwell
> 
> On Tue, Jan 17, 2023 at 12:20:16PM +0000, Yao, Jiewen wrote:
> > I have seen Gerd sent out V4 patch set for #4251. Is that not enough?
> 
> If that is merged first patch 17 should not be needed.
> 
> take care,
>   Gerd
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set CPU to Broadwell

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

On Tue, Jan 17, 2023 at 3:50 PM Yao, Jiewen <jiewen.yao@intel.com> wrote:

> Cool!
> Then I suggest to drop patch 17.
>

Perfectly fine with me.


>
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Gerd
> > Hoffmann
> > Sent: Tuesday, January 17, 2023 8:39 PM
> > To: Yao, Jiewen <jiewen.yao@intel.com>
> > Cc: devel@edk2.groups.io; osteffen@redhat.com; Andrew Fish
> > <afish@apple.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Feng,
> > Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Wang,
> > Jian J <jian.j.wang@intel.com>; Justen, Jordan L <
> jordan.l.justen@intel.com>;
> > Leif Lindholm <quic_llindhol@quicinc.com>; Gao, Liming
> > <gaoliming@byosoft.com.cn>; Kinney, Michael D
> > <michael.d.kinney@intel.com>; Michael Kubacki
> > <mikuback@linux.microsoft.com>; Ni, Ray <ray.ni@intel.com>; Sami
> > Mujawar <sami.mujawar@arm.com>; Sean Brogan
> > <sean.brogan@microsoft.com>; Chen, Christine <yuwei.chen@intel.com>;
> > Pawel Polawski <ppolawsk@redhat.com>; chris.fernald@outlook.com
> > Subject: Re: [edk2-devel] [PATCH v11 17/17] OfmvPkg: CI: Qemu tests: set
> > CPU to Broadwell
> >
> > On Tue, Jan 17, 2023 at 12:20:16PM +0000, Yao, Jiewen wrote:
> > > I have seen Gerd sent out V4 patch set for #4251. Is that not enough?
> >
> > If that is merged first patch 17 should not be needed.
> >
> > take care,
> >   Gerd
> >
>
>

[-- Attachment #2: Type: text/html, Size: 3618 bytes --]

Re: [PATCH v11 07/17] CI: Use Fedora 35 container (Linux only)

[Michael Kubacki]

Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>

On 1/16/2023 12:40 PM, Oliver Steffen wrote:
> Run all Linux based jobs in a container, using a custom Fedora 35 image
> (gcc 11).  The image URL specified in the defaults.yml template, so that
> all CI jobs can use it. The image is hosted on ghcr.io and the
> Dockerfiles are here: https://github.com/tianocore/containers The
> version numbers of gcc, iasl, and nasm are pinned to avoid unintended
> upgrades during image rebuild.
> 
> Do not run apt-get in CI jobs to install qemu and gcc dependencies.
> Assume the container image provides these.
> 
> Use Python from the container image, do not download at runtime.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   .azurepipelines/Ubuntu-GCC5.yml                     | 3 ++-
>   .azurepipelines/templates/basetools-build-steps.yml | 9 ---------
>   .azurepipelines/templates/defaults.yml              | 1 +
>   3 files changed, 3 insertions(+), 10 deletions(-)
> 
> diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
> index 4ed6cb601b8e..246e2d272e20 100644
> --- a/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/.azurepipelines/Ubuntu-GCC5.yml
> @@ -21,6 +21,7 @@ jobs:
>     parameters:
>       tool_chain_tag: 'GCC5'
>       vm_image: 'ubuntu-latest'
> +    container: ${{ variables.default_linux_image }}
>       arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
> -    usePythonVersion: ${{ variables.default_python_version }}
> +    usePythonVersion: ''  # use Python from the container image
>   
> diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml
> index d8c108c6e212..a72758bc3395 100644
> --- a/.azurepipelines/templates/basetools-build-steps.yml
> +++ b/.azurepipelines/templates/basetools-build-steps.yml
> @@ -10,15 +10,6 @@ parameters:
>     tool_chain_tag: ''
>   
>   steps:
> -- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
> -  - bash: sudo apt-get update
> -    displayName: Update apt
> -    condition: and(gt(variables.pkg_count, 0), succeeded())
> -
> -  - bash: sudo apt-get install gcc g++ make uuid-dev
> -    displayName: Install required tools
> -    condition: and(gt(variables.pkg_count, 0), succeeded())
> -
>   - task: CmdLine@1
>     displayName: Build Base Tools from source
>     inputs:
> diff --git a/.azurepipelines/templates/defaults.yml b/.azurepipelines/templates/defaults.yml
> index b4909448bdea..74d6b417839d 100644
> --- a/.azurepipelines/templates/defaults.yml
> +++ b/.azurepipelines/templates/defaults.yml
> @@ -9,3 +9,4 @@
>   
>   variables:
>     default_python_version: ">=3.10.6"
> +  default_linux_image: "ghcr.io/tianocore/containers/fedora-35-test:47addc9"

Re: [PATCH v11 08/17] ArmVirtPkg: CI: Use Fedora 35 container (Linux only)

[Michael Kubacki]

Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>

On 1/16/2023 12:40 PM, Oliver Steffen wrote:
> Run the Linux jobs of the ArmVirtPkg platform CI inside a container,
> in the same way the general CI does now. Make use of the default image
> specified in the defaults.yml template.
> 
> Do not run apt-get in CI jobs to install qemu and gcc dependencies.
> Assume the container image provides these.
> 
> Use Python from the container image, do not download at runtime.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 8 +++-----
>   1 file changed, 3 insertions(+), 5 deletions(-)
> 
> diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 5a0e589ed4a7..c0a2e0f20d4f 100644
> --- a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -78,6 +78,8 @@ jobs:
>       pool:
>         vmImage: $(vm_image)
>   
> +    container: ${{ variables.default_linux_image }}
> +
>       steps:
>       - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
>         parameters:
> @@ -88,8 +90,4 @@ jobs:
>           build_file: $(Build.File)
>           build_flags: $(Build.Flags)
>           run_flags: $(Run.Flags)
> -        usePythonVersion: ${{ variables.default_python_version }}
> -        extra_install_step:
> -        - bash: sudo apt-get install qemu
> -          displayName: Install qemu
> -          condition: and(gt(variables.pkg_count, 0), succeeded())
> +        usePythonVersion: '' # use Python from the container image

Re: [PATCH v11 09/17] EmulatorPkg: CI: Use Fedora 35 container (Linux only)

[Michael Kubacki]

Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>

On 1/16/2023 12:40 PM, Oliver Steffen wrote:
> Run the Linux jobs of the EmulatorPkg platform CI inside a container,
> in the same way the general CI does now. Make use of the default image
> specified in the defaults.yml template.
> 
> Do not run apt-get in CI jobs to install qemu and gcc dependencies.
> Assume the container image provides these.
> 
> Use Python from the container image, do not download at runtime.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index a32c57d4aab4..3861457ac70a 100644
> --- a/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -79,6 +79,8 @@ jobs:
>       pool:
>         vmImage: $(vm_image)
>   
> +    container: ${{ variables.default_linux_image }}
> +
>       steps:
>       - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
>         parameters:
> @@ -89,4 +91,4 @@ jobs:
>           build_file: $(Build.File)
>           build_flags: $(Build.Flags)
>           run_flags: $(Run.Flags)
> -        usePythonVersion: ${{ variables.default_python_version }}
> +        usePythonVersion: '' # use Python from the container image

Re: [edk2-devel] [PATCH v11 10/17] OvmfPkg: CI: Use Fedora 35 container (Linux only)

[Michael Kubacki]

Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>

On 1/16/2023 12:40 PM, Oliver Steffen wrote:
> Run the Linux jobs of the OvmfPkg platform CI inside a container,
> in the same way the general CI does now. Make use of the default image
> specified in the defaults.yml template.
> 
> Do not run apt-get in CI jobs to install qemu and gcc dependencies.
> Assume the container image provides these.
> 
> Use Python from the container image, do not download at runtime.
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> ---
>   OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 8 +++-----
>   1 file changed, 3 insertions(+), 5 deletions(-)
> 
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> index 6dd90711ac70..85be431ca5e7 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
> @@ -180,6 +180,8 @@ jobs:
>       pool:
>         vmImage: $(vm_image)
>   
> +    container: ${{ variables.default_linux_image }}
> +
>       steps:
>       - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml
>         parameters:
> @@ -190,8 +192,4 @@ jobs:
>           build_file: $(Build.File)
>           build_flags: $(Build.Flags)
>           run_flags: $(Run.Flags)
> -        usePythonVersion: ${{ variables.default_python_version }}
> -        extra_install_step:
> -        - bash: sudo apt-get install qemu
> -          displayName: Install qemu
> -          condition: and(gt(variables.pkg_count, 0), succeeded())
> +        usePythonVersion: '' # use Python from the container image

Re: [PATCH v11 00/17] CI: Use Fedora 35 container for Linux jobs

[Ard Biesheuvel]

On Mon, 16 Jan 2023 at 18:41, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
> containers.  This decouples the CI environment from the virtual machine
> images that Azure DevOps provides.  The currently used ubuntu-18.04 image
> has been deprecated for a while now and will be removed soon.
>
> The container image provides the required compiler toolchains and Qemu
> for the supported architectures. These are then no longer downloaded at
> runtime, avoiding CI failures due to download errors. This approach also
> makes it easier to switch to other or newer compilers. It makes the CI
> setup independent from the default images that Azure DevOps provides.
> It can also help debugging CI problems, because the CI environment
> can be reproduced on a local machine.
>
> The container images are hosted on ghcr.io and are automatically
> generated using GitHub Actions. The Dockerfiles are maintained in the
> Tianocore "containers" repository:
> https://github.com/tianocore/containers.
>
> The current image is based on Fedora 35, with gcc 11. Fedora was chosen
> because of its fast release cycle which makes it easy to keep the
> toolchains up-to-date.
>
> Some further possible changes not included in this series:
> - The Tianocore/containers repository provides stack of layered images.
>   One image for general purpose (build+test) and build-only jobs.
>   The build+test image is based on the build-only one and adds Qemu,
>   for the testing job that involve Qemu.  The work in the image side
>   is done, we just need to change the CI setup accordingly.
>   This patch set uses the build+test images for all jobs.
> - Further reduce the number of external dependencies that need to be
>   downloaded at runtime. Candidates are iasl and nasm, which are already
>   included in the image but not used yet.
>
> PR: https://github.com/tianocore/edk2/pull/3904
>
> v11:
> - Rebase to current master branch
> - Use lasted image, includes Qemu 7.2
> - Set container image URL in a central place (the defaults.yml template)
>
> v10:
> - Split commits by package
> - Use ubuntu-22.04 as vm_image (instead of the floating ubuntu-latest)
> - Some fixups around the Python version template argument and defaults.yml
> - Fix some commit messages
> - Add license statement to defaults.yml
>
> v9:
> - Drop the "Don't install cspell" patch
> - Use explicitly use ubuntu-22.04 as vm_image
>
> v8:
> - Use updated container image that contains gcc for LoongArch64.
> - Remove ext_dep files for the LoongArch64 gcc.
> - Don't change the scopes in .pytool/CISettings.py if running Linux CI
>   it.
> - Split commits that touch multiple packages.
> - Use the smaller "build" image for jobs that allow it.
> - Add a CI template file as a central place to define the default Python version
>   and use it where needed.
>
> v7:
> - Rebase to latest master branch.
> - Use latest Fedora 35 CI image.
> - Stop using the ubuntu-18.04 vm_image since this will no longer be available
>   after Dec 1st.  Use ubuntu-latest instead.
>
> v6:
> - Include suggestions by Chris Fernald.
> - Added a parameter for the container image to the job template, makes usage
>   of containers optional.
> - Added a parameter to configure the Python version to download. Allows
>   using Python from the VM/container image also.
> - Restructure the commits (no further functional changes).
>
> v5:
> - Update image
>
> v4:
> - Use the latest image from the tianocode/containers repository which
>   - does not include acpica-tools
>   - includes Pyhton 3.10
>
> v3:
> - Use the latest image from the tianocode/containers repository which
>   pins down version numbers of gcc, iasl, and nasm in the Dockerfile.
>
> v2:
> - Images are now hosted under the Tianocore Organization
>   https://github.com/tianocore/containers
>
> v1:
> - Thread: https://edk2.groups.io/g/devel/message/89058
> - Images were hosted at https://github.com/osteffenrh/edk2-build-images
>
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> Acked-by: Ard Biesheuvel <ardb@kernel.org>
> Acked-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Chris Fernald <chfernal@microsoft.com>
>
> Oliver Steffen (17):
>   CI: make Python version configurable
>   ArmVirtPkg: CI: use Python version from defaults template
>   EmulatorPkg: CI: use Python version from defaults template
>   OvmfPkg: CI: use Python version from defaults template
>   CI: add ~/.local/bin to PATH (Linux only)
>   CI: Allow running in a container.
>   CI: Use Fedora 35 container (Linux only)
>   ArmVirtPkg: CI: Use Fedora 35 container (Linux only)
>   EmulatorPkg: CI: Use Fedora 35 container (Linux only)
>   OvmfPkg: CI: Use Fedora 35 container (Linux only)
>   .pytool: CISettings.py: don't add scopes for GCC
>   BaseTools: remove ext_dep files for gcc
>   ArmVirtPkg: CI: use ubuntu-22.04 vm_image (Linux only)
>   EmulatorPgk: CI: use ubuntu-22.04 vm_image (Linux only)
>   OvmfPkg: CI: use ubuntu-22.04 vm_image (Linux only)
>   CI: use ubuntu-22.04 image (Linux only)

Thanks

I have created a PR here

https://github.com/tianocore/edk2/pull/3913

with all the acks added.

>   OfmvPkg: CI: Qemu tests: set CPU to Broadwell
>

... and I omitted this one (as discussed on the thread)


>  .azurepipelines/Ubuntu-GCC5.yml               | 12 +++++-----
>  .azurepipelines/Windows-VS2019.yml            |  4 ++++
>  .../templates/basetools-build-steps.yml       |  9 --------
>  .azurepipelines/templates/defaults.yml        | 12 ++++++++++
>  .../templates/platform-build-run-steps.yml    | 12 +++++++++-
>  .../templates/pr-gate-build-job.yml           |  6 +++++
>  .azurepipelines/templates/pr-gate-steps.yml   | 12 ++++++++--
>  .pytool/CISettings.py                         |  9 --------
>  .../.azurepipelines/Ubuntu-GCC5.yml           | 12 +++++-----
>  BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
>  BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
>  ...gcc_loongarch64_unknown_linux_ext_dep.yaml | 22 -------------------
>  .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  9 +++++++-
>  .../.azurepipelines/Windows-VS2019.yml        |  4 ++++
>  .../.azurepipelines/Ubuntu-GCC5.yml           | 12 +++++-----
>  .../.azurepipelines/Windows-VS2019.yml        |  5 +++++
>  OvmfPkg/PlatformCI/PlatformBuildLib.py        |  1 +
>  18 files changed, 82 insertions(+), 123 deletions(-)
>  create mode 100644 .azurepipelines/templates/defaults.yml
>  delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_loongarch64_unknown_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
>
> --
> 2.39.0
>