* [PATCH v6 0/5] CI: Use Fedora 35 container for Linux jobs
@ 2022-09-26 16:31 Oliver Steffen
2022-11-28 22:55 ` [edk2-devel] " Chris Fernald
0 siblings, 1 reply; 2+ messages in thread
From: Oliver Steffen @ 2022-09-26 16:31 UTC (permalink / raw)
To: devel
Cc: Oliver Steffen, Andrew Fish, Ard Biesheuvel, Bob Feng,
Gerd Hoffmann, Jiewen Yao, Jordan Justen, Leif Lindholm,
Liming Gao, Michael D Kinney, Michael Kubacki, Ray Ni,
Sami Mujawar, Sean Brogan, Yuwei Chen, Pawel Polawski,
Ard Biesheuvel
Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
containers.
The container image provides the required compiler toolchains and Qemu
for the supported architectures. These are then no longer downloaded at
runtime, avoiding CI failures due to download errors. This approach also
makes it easier to switch to other or newer compilers. It makes the CI
setup independent from the default images that Azure DevOps provides.
It can also help debugging CI problems, because the CI environment
can be reproduced on a local machine.
The container images are hosted on ghcr.io and are automatically
generated using GitHub Actions. The Dockerfiles are maintained in the
Tianocore "containers" repository:
https://github.com/tianocore/containers.
The current image is based on Fedora 35, with gcc 11. Fedora was chosen
because of its fast release cycle which makes it easy to keep the
toolchains up-to-date.
Some further possible changes not included in this series:
- The Tianocore/containers repository provides stack of layered images.
One image for general purpose (build+test) and build-only jobs.
The build+test image is based on the build-only one and adds Qemu,
for the testing job that involve Qemu. This reduces the total download
size. This was suggested by Chris Fernald. The work in the image side
is done, we just need to change the CI setup accordingly.
This patch set uses the build+test images for all jobs.
- Further reduce the number of external dependencies that need to be
downloaded at runtime. Candidates are iasl and nasm, which are already
included in the image but not used yet.
PR: https://github.com/tianocore/edk2/pull/2935
v6:
- Include suggestions by Chris Fernald.
- Added a parameter for the container image to the job template, makes usage
of containers optional.
- Added a parameter to configure the Python version to download. Allows
using Python from the VM/container image also.
- Restructure the commits (no further functional changes).
v5:
- Update image
v4:
- Use the latest image from the tianocode/containers repository which
- does not include acpica-tools
- includes Pyhton 3.10
v3:
- Use the latest image from the tianocode/containers repository which
pins down version numbers of gcc, iasl, and nasm in the Dockerfile.
v2:
- Images are now hosted under the Tianocore Organization
https://github.com/tianocore/containers
v1:
- Thread: https://edk2.groups.io/g/devel/message/89058
- Images were hosted at https://github.com/osteffenrh/edk2-build-images
Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Gerd Hoffmann <kraxel@redhat.com>
Oliver Steffen (5):
CI: make Python version configurable
CI: add ~/.local/bin to PATH (Linux only)
CI: Allow running in a container.
CI: Use Fedora 35 container (Linux only)
BaseTools: Remove ext. gcc dependencies (Linux only)
.azurepipelines/Ubuntu-GCC5.yml | 2 ++
.../templates/basetools-build-steps.yml | 9 --------
.../templates/platform-build-run-steps.yml | 12 +++++++++-
.../templates/pr-gate-build-job.yml | 6 +++++
.azurepipelines/templates/pr-gate-steps.yml | 12 ++++++++--
.../.azurepipelines/Ubuntu-GCC5.yml | 7 +++---
BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ------------------
BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ------------------
.../Bin/gcc_riscv64_unknown_ext_dep.yaml | 22 -------------------
.../.azurepipelines/Ubuntu-GCC5.yml | 3 +++
.../.azurepipelines/Ubuntu-GCC5.yml | 7 +++---
11 files changed, 38 insertions(+), 84 deletions(-)
delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
--
2.37.3
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [edk2-devel] [PATCH v6 0/5] CI: Use Fedora 35 container for Linux jobs
2022-09-26 16:31 [PATCH v6 0/5] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
@ 2022-11-28 22:55 ` Chris Fernald
0 siblings, 0 replies; 2+ messages in thread
From: Chris Fernald @ 2022-11-28 22:55 UTC (permalink / raw)
To: devel, osteffen
Cc: Andrew Fish, Ard Biesheuvel, Bob Feng, Gerd Hoffmann, Jiewen Yao,
Jordan Justen, Leif Lindholm, Liming Gao, Michael D Kinney,
Michael Kubacki, Ray Ni, Sami Mujawar, Sean Brogan, Yuwei Chen,
Pawel Polawski, Ard Biesheuvel
After discussing this a bit more internally, I think I'm actually coming
around to removing the compiler external dependencies. Leaving behind
the unused external dependencies is likely to lead to them collecting
dust, so it would probably be better left for consumers to define at the
platform level if they need it.
One additional change I would like, but will not hold my review on is to
remove the "gcc_<ARCH>_linux" scopes from PlatformBuild.py and
CISettings.py as they will no longer be used once the extdep files are
removed.
Reviewed-by: Chris Fernald <chfernal@microsoft.com>
On 9/26/2022 9:31 AM, Oliver Steffen wrote:
> Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in custom
> containers.
>
> The container image provides the required compiler toolchains and Qemu
> for the supported architectures. These are then no longer downloaded at
> runtime, avoiding CI failures due to download errors. This approach also
> makes it easier to switch to other or newer compilers. It makes the CI
> setup independent from the default images that Azure DevOps provides.
> It can also help debugging CI problems, because the CI environment
> can be reproduced on a local machine.
>
> The container images are hosted on ghcr.io and are automatically
> generated using GitHub Actions. The Dockerfiles are maintained in the
> Tianocore "containers" repository:
> https://github.com/tianocore/containers.
>
> The current image is based on Fedora 35, with gcc 11. Fedora was chosen
> because of its fast release cycle which makes it easy to keep the
> toolchains up-to-date.
>
> Some further possible changes not included in this series:
> - The Tianocore/containers repository provides stack of layered images.
> One image for general purpose (build+test) and build-only jobs.
> The build+test image is based on the build-only one and adds Qemu,
> for the testing job that involve Qemu. This reduces the total download
> size. This was suggested by Chris Fernald. The work in the image side
> is done, we just need to change the CI setup accordingly.
> This patch set uses the build+test images for all jobs.
> - Further reduce the number of external dependencies that need to be
> downloaded at runtime. Candidates are iasl and nasm, which are already
> included in the image but not used yet.
>
> PR: https://github.com/tianocore/edk2/pull/2935
>
> v6:
> - Include suggestions by Chris Fernald.
> - Added a parameter for the container image to the job template, makes usage
> of containers optional.
> - Added a parameter to configure the Python version to download. Allows
> using Python from the VM/container image also.
> - Restructure the commits (no further functional changes).
>
> v5:
> - Update image
>
> v4:
> - Use the latest image from the tianocode/containers repository which
> - does not include acpica-tools
> - includes Pyhton 3.10
>
> v3:
> - Use the latest image from the tianocode/containers repository which
> pins down version numbers of gcc, iasl, and nasm in the Dockerfile.
>
> v2:
> - Images are now hosted under the Tianocore Organization
> https://github.com/tianocore/containers
>
> v1:
> - Thread: https://edk2.groups.io/g/devel/message/89058
> - Images were hosted at https://github.com/osteffenrh/edk2-build-images
>
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> Acked-by: Ard Biesheuvel <ardb@kernel.org>
> Acked-by: Gerd Hoffmann <kraxel@redhat.com>
>
> Oliver Steffen (5):
> CI: make Python version configurable
> CI: add ~/.local/bin to PATH (Linux only)
> CI: Allow running in a container.
> CI: Use Fedora 35 container (Linux only)
> BaseTools: Remove ext. gcc dependencies (Linux only)
>
> .azurepipelines/Ubuntu-GCC5.yml | 2 ++
> .../templates/basetools-build-steps.yml | 9 --------
> .../templates/platform-build-run-steps.yml | 12 +++++++++-
> .../templates/pr-gate-build-job.yml | 6 +++++
> .azurepipelines/templates/pr-gate-steps.yml | 12 ++++++++--
> .../.azurepipelines/Ubuntu-GCC5.yml | 7 +++---
> BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ------------------
> BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ------------------
> .../Bin/gcc_riscv64_unknown_ext_dep.yaml | 22 -------------------
> .../.azurepipelines/Ubuntu-GCC5.yml | 3 +++
> .../.azurepipelines/Ubuntu-GCC5.yml | 7 +++---
> 11 files changed, 38 insertions(+), 84 deletions(-)
> delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
> delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
> delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-11-28 22:55 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-09-26 16:31 [PATCH v6 0/5] CI: Use Fedora 35 container for Linux jobs Oliver Steffen
2022-11-28 22:55 ` [edk2-devel] " Chris Fernald
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox