* [Patch v5 00/22] Enable Phase 1 of EDK II CI
@ 2019-11-11 8:40 Michael D Kinney
2019-11-11 8:40 ` [Patch v5 01/22] Maintainers.txt: Add continuous integration(CI) directories Michael D Kinney
` (21 more replies)
0 siblings, 22 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel
Cc: Sean Brogan, Bret Barkelew, Liming Gao, Bob Feng, Andrew Fish,
Laszlo Ersek, Leif Lindholm, Jian J Wang, Xiaoyu Lu, Ray Ni,
Hao A Wu, Jiaxin Wu, Siyuan Fu, Jiewen Yao, Chao Zhang,
Zhichao Gao, Eric Dong
https://bugzilla.tianocore.org/show_bug.cgi?id=2315
RFC Proposals:
* https://edk2.groups.io/g/rfc/message/93
* https://edk2.groups.io/g/devel/message/46607
Wiki Page:
* https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-Integration
Branch for review:
* https://github.com/tianocore/edk2-staging/tree/edk2-ci_V5
Changes in V5
* Rename requirements.txt -> pip_requirements.txt
* Use HTML syntax for Build Status in Readme.md in root
* Add @file to YML and YAML file headers
* Format Readme.md files to 80 columns
* Add missing Readme.md in .pytool CompilerPlugin
* Add missing Readme.md in .pytool DependencyCheck
* Rename readme.md -> Readme.md in .pytool DscCompleteCheck
* Rename readme.md -> Readme.md in .pytool LibraryClassCheck
* Remove commented out SecurityPkg dependency from CryptoPkg YAML
* Remove SerialPortLib IgnoreDuplicates from MdeModulePkg YAML
Changes in V4
* Only copy BaseTools build logs if BaseTools are built
* Add FINISHED and FAILED jobs to prevent multiple email notifications from
Mergify for PR close and PR comment actions
* Reformat .pytools/Readme.md to 80 columns
* Set WINSDK_PATH_FOR_RC_EXE in BaseTools/set_vsprefix_env.bat for VS2017/VS2019
* Update .gitignore to alphabetic ordering
* Update Maintainers.txt to alphabetic ordering
* Add file header comment block to requirements.txt
* Remove license badge and formatting changes from Readme.md
Previous versions of branches for reference:
* https://github.com/tianocore/edk2-staging/tree/edk2-ci_V4
* https://github.com/tianocore/edk2-staging/tree/edk2-ci_V3
* https://github.com/tianocore/edk2-staging/tree/edk2-ci_V2
* https://github.com/tianocore/edk2-staging/tree/edk2-ci_V1
Active branch for testing/evaluation:
* https://github.com/tianocore/edk2-staging/tree/edk2-ci
* To test, fork edk2-staging repo, create a branch with a change, and submit
a pull request targeting edk2-staging/edk2-ci. NOTE: the default branch for
the edk2-staging is 'about'. You must select the 'edk2-ci' branch when
a pull request is opened. Set the 'push' label to require commit if all
checks pass.
Pull request history on active branch for testing/evaluation:
* https://github.com/tianocore/edk2-staging/pulls?q=is%3Apr+is%3Aclosed
This patch series enables pre-commit and post-commit checks on edk2/master The
goal is to improve quality of code submissions by requiring all pre-commit
checks to pass before any change is committed to edk2/master. In order to
minimize the impact to the developer process, when an EDK II Maintainer is ready
to commit changes to edk2/master, the EDK II Maintainer pushes a branch to their
personal fork of the edk2 repository and opens a pull request targeting
edk2/master and sets the 'push' label. If all checks pass, then the pull
request is automatically committed.
A developer that wants to know if a patch series passes all checks can either
run the checks locally or open a pull request without setting the 'push' label.
The post-commit checks build all packages and provide status badges and links to
reports in the Readme.md file at the root of edk2/master.
A combination of GitHub, Azure Pipelines, Mergify, and edk2-pytool features are
used to implement this feature. GitHub and Azure Pipelines configuration steps
are required to activate this feature on edk2/master.
Once this feature is active and shown to be stable for 1-2 weeks, write access
to the edk2 repository will be removed for all EDK II Maintainers. Only admins
will retain write access, and admins should avoid use of direct push to resolve
an issue unless there is no method to resolve the issue using the pull request
method.
The following checks are made available by this patch series. Not all checks
are enabled for all packages. Enabling all checks on all packages can be done
incrementally as well as adding more checks over time.
* PatchCheck
* CharEncodingCheck
* CompilerPlugin
* DependencyCheck
* DscCompleteCheck
* GuidCheck
* LibraryClassCheck
* SpellCheck
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Xiaoyu Lu <xiaoyux.lu@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Cc: Zhichao Gao <zhichao.gao@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Michael D Kinney (14):
Maintainers.txt: Add continuous integration(CI) directories
CryptoPkg: Add YAML file for CI builds
FatPkg: Add YAML file for CI builds
FmpDevicePkg: Add YAML file for CI builds
MdeModulePkg: Add YAML file for CI builds
MdePkg: Add YAML file for CI builds
NetworkPkg: Add YAML file for CI builds
PcAtChipsetPkg: Add YAML files for CI builds
SecurityPkg: Add YAML files for CI builds
ShellPkg: Add YAML file for CI builds
UefiCpuPkg: Add YAML file for CI builds
SignedCapsulePkg: Use BaseCryptLibNull to reduce package CI time
.mergify: Add Mergify YML pull request rules configuration file
Readme.md: Add CI build status badges
Sean Brogan (8):
.gitignore: Ignore python compiled files, extdeps, and vscode
pip_requirements.txt: Add python pip requirements file
BaseTools: Add RC_PATH define for VS2017/2019
BaseTools: Add YAML files with path env and tool extdeps
BaseTools: Add BaseTools plugins to support CI
.pytool/Plugin: Add CI plugins
.pytool: Add CISettings.py and Readme.md
.azurepipelines: Add Azure Pipelines YML configuration files
.azurepipelines/Ubuntu-GCC5.yml | 18 ++
.azurepipelines/Ubuntu-PatchCheck.yml | 35 +++
.azurepipelines/Windows-VS2019.yml | 18 ++
.../templates/basetools-build-steps.yml | 37 +++
.../templates/pr-gate-build-job.yml | 80 ++++++
.azurepipelines/templates/pr-gate-steps.yml | 130 +++++++++
.../templates/spell-check-prereq-steps.yml | 22 ++
.gitignore | 6 +-
.mergify/config.yml | 97 +++++++
.pytool/CISettings.py | 173 ++++++++++++
.../CharEncodingCheck/CharEncodingCheck.py | 118 ++++++++
.../CharEncodingCheck_plug_in.yaml | 11 +
.pytool/Plugin/CharEncodingCheck/Readme.md | 18 ++
.../Plugin/CompilerPlugin/CompilerPlugin.py | 102 +++++++
.../CompilerPlugin/Compiler_plug_in.yaml | 11 +
.pytool/Plugin/CompilerPlugin/Readme.md | 17 ++
.../Plugin/DependencyCheck/DependencyCheck.py | 120 +++++++++
.../DependencyCheck_plug_in.yaml | 13 +
.pytool/Plugin/DependencyCheck/Readme.md | 31 +++
.../DscCompleteCheck/DscCompleteCheck.py | 118 ++++++++
.../DscCompleteCheck_plug_in.yaml | 12 +
.pytool/Plugin/DscCompleteCheck/Readme.md | 27 ++
.pytool/Plugin/GuidCheck/GuidCheck.py | 251 ++++++++++++++++++
.../Plugin/GuidCheck/GuidCheck_plug_in.yaml | 11 +
.pytool/Plugin/GuidCheck/Readme.md | 80 ++++++
.../LibraryClassCheck/LibraryClassCheck.py | 153 +++++++++++
.../LibraryClassCheck_plug_in.yaml | 11 +
.pytool/Plugin/LibraryClassCheck/Readme.md | 25 ++
.pytool/Plugin/SpellCheck/Readme.md | 127 +++++++++
.pytool/Plugin/SpellCheck/SpellCheck.py | 216 +++++++++++++++
.../Plugin/SpellCheck/SpellCheck_plug_in.yaml | 11 +
.pytool/Plugin/SpellCheck/cspell.base.yaml | 165 ++++++++++++
.pytool/Readme.md | 223 ++++++++++++++++
BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 ++
BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 ++
BaseTools/Bin/iasl_ext_dep.yaml | 21 ++
BaseTools/Bin/nasm_ext_dep.yaml | 18 ++
.../BinWrappers/PosixLike/posix_path_env.yaml | 10 +
.../WindowsLike/win_build_tools_path_env.yaml | 10 +
BaseTools/Conf/tools_def.template | 18 +-
BaseTools/Edk2ToolsBuild.py | 163 ++++++++++++
.../BuildToolsReportGenerator.py | 69 +++++
.../BuildToolsReportGenerator_plug_in.yaml | 12 +
.../BuildToolsReport_Template.html | 126 +++++++++
.../LinuxGcc5ToolChain/LinuxGcc5ToolChain.py | 85 ++++++
.../LinuxGcc5ToolChain_plug_in.yaml | 12 +
.../WindowsResourceCompiler/WinRcPath.py | 29 ++
.../WinRcPath_plug_in.yaml | 13 +
.../WindowsVsToolChain/WindowsVsToolChain.py | 126 +++++++++
.../WindowsVsToolChain_plug_in.yaml | 11 +
.../basetool_tiano_python_path_env.yaml | 11 +
BaseTools/basetools_calling_path_env.yaml | 11 +
BaseTools/basetools_path_env.yaml | 11 +
BaseTools/set_vsprefix_envs.bat | 10 +
CryptoPkg/CryptoPkg.ci.yaml | 47 ++++
FatPkg/FatPkg.ci.yaml | 50 ++++
FmpDevicePkg/FmpDevicePkg.ci.yaml | 43 +++
FmpDevicePkg/FmpDevicePkg.dsc | 6 +-
Maintainers.txt | 23 ++
MdeModulePkg/MdeModulePkg.ci.yaml | 80 ++++++
MdePkg/MdePkg.ci.yaml | 90 +++++++
NetworkPkg/NetworkPkg.ci.yaml | 62 +++++
NetworkPkg/NetworkPkg.dsc | 5 +
PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml | 46 ++++
Readme.md | 51 ++++
SecurityPkg/SecurityPkg.ci.yaml | 80 ++++++
SecurityPkg/SecurityPkg.dsc | 32 ++-
ShellPkg/ShellPkg.ci.yaml | 55 ++++
SignedCapsulePkg/SignedCapsulePkg.dsc | 38 ++-
UefiCpuPkg/UefiCpuPkg.ci.yaml | 51 ++++
pip_requirements.txt | 17 ++
71 files changed, 4063 insertions(+), 8 deletions(-)
create mode 100644 .azurepipelines/Ubuntu-GCC5.yml
create mode 100644 .azurepipelines/Ubuntu-PatchCheck.yml
create mode 100644 .azurepipelines/Windows-VS2019.yml
create mode 100644 .azurepipelines/templates/basetools-build-steps.yml
create mode 100644 .azurepipelines/templates/pr-gate-build-job.yml
create mode 100644 .azurepipelines/templates/pr-gate-steps.yml
create mode 100644 .azurepipelines/templates/spell-check-prereq-steps.yml
create mode 100644 .mergify/config.yml
create mode 100644 .pytool/CISettings.py
create mode 100644 .pytool/Plugin/CharEncodingCheck/CharEncodingCheck.py
create mode 100644 .pytool/Plugin/CharEncodingCheck/CharEncodingCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/CharEncodingCheck/Readme.md
create mode 100644 .pytool/Plugin/CompilerPlugin/CompilerPlugin.py
create mode 100644 .pytool/Plugin/CompilerPlugin/Compiler_plug_in.yaml
create mode 100644 .pytool/Plugin/CompilerPlugin/Readme.md
create mode 100644 .pytool/Plugin/DependencyCheck/DependencyCheck.py
create mode 100644 .pytool/Plugin/DependencyCheck/DependencyCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/DependencyCheck/Readme.md
create mode 100644 .pytool/Plugin/DscCompleteCheck/DscCompleteCheck.py
create mode 100644 .pytool/Plugin/DscCompleteCheck/DscCompleteCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/DscCompleteCheck/Readme.md
create mode 100644 .pytool/Plugin/GuidCheck/GuidCheck.py
create mode 100644 .pytool/Plugin/GuidCheck/GuidCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/GuidCheck/Readme.md
create mode 100644 .pytool/Plugin/LibraryClassCheck/LibraryClassCheck.py
create mode 100644 .pytool/Plugin/LibraryClassCheck/LibraryClassCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/LibraryClassCheck/Readme.md
create mode 100644 .pytool/Plugin/SpellCheck/Readme.md
create mode 100644 .pytool/Plugin/SpellCheck/SpellCheck.py
create mode 100644 .pytool/Plugin/SpellCheck/SpellCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/SpellCheck/cspell.base.yaml
create mode 100644 .pytool/Readme.md
create mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
create mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
create mode 100644 BaseTools/Bin/iasl_ext_dep.yaml
create mode 100644 BaseTools/Bin/nasm_ext_dep.yaml
create mode 100644 BaseTools/BinWrappers/PosixLike/posix_path_env.yaml
create mode 100644 BaseTools/BinWrappers/WindowsLike/win_build_tools_path_env.yaml
create mode 100644 BaseTools/Edk2ToolsBuild.py
create mode 100644 BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator.py
create mode 100644 BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator_plug_in.yaml
create mode 100644 BaseTools/Plugin/BuildToolsReport/BuildToolsReport_Template.html
create mode 100644 BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain.py
create mode 100644 BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain_plug_in.yaml
create mode 100644 BaseTools/Plugin/WindowsResourceCompiler/WinRcPath.py
create mode 100644 BaseTools/Plugin/WindowsResourceCompiler/WinRcPath_plug_in.yaml
create mode 100644 BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain.py
create mode 100644 BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain_plug_in.yaml
create mode 100644 BaseTools/Source/Python/basetool_tiano_python_path_env.yaml
create mode 100644 BaseTools/basetools_calling_path_env.yaml
create mode 100644 BaseTools/basetools_path_env.yaml
create mode 100644 CryptoPkg/CryptoPkg.ci.yaml
create mode 100644 FatPkg/FatPkg.ci.yaml
create mode 100644 FmpDevicePkg/FmpDevicePkg.ci.yaml
create mode 100644 MdeModulePkg/MdeModulePkg.ci.yaml
create mode 100644 MdePkg/MdePkg.ci.yaml
create mode 100644 NetworkPkg/NetworkPkg.ci.yaml
create mode 100644 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
create mode 100644 SecurityPkg/SecurityPkg.ci.yaml
create mode 100644 ShellPkg/ShellPkg.ci.yaml
create mode 100644 UefiCpuPkg/UefiCpuPkg.ci.yaml
create mode 100644 pip_requirements.txt
--
2.21.0.windows.1
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Patch v5 01/22] Maintainers.txt: Add continuous integration(CI) directories
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 02/22] .gitignore: Ignore python compiled files, extdeps, and vscode Michael D Kinney
` (20 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Andrew Fish, Laszlo Ersek, Leif Lindholm
Add maintainers and reviewers for the directories associated
with continuous integration steps.
* .azurepipelines
* .mergify
* .pytool
Cc: Andrew Fish <afish@apple.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
---
Maintainers.txt | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/Maintainers.txt b/Maintainers.txt
index 922cc664dd..180ad35d09 100644
--- a/Maintainers.txt
+++ b/Maintainers.txt
@@ -98,6 +98,29 @@ F: */Arm/
M: Leif Lindholm <leif.lindholm@linaro.org>
M: Ard Biesheuvel <ard.biesheuvel@linaro.org>
+EDK II Continuous Integration:
+------------------------------
+.azurepipelines/
+F: .azurepipelines/
+M: Sean Brogan <sean.brogan@microsoft.com>
+M: Bret Barkelew <Bret.Barkelew@microsoft.com>
+R: Michael D Kinney <michael.d.kinney@intel.com>
+R: Liming Gao <liming.gao@intel.com>
+
+.mergify/
+F: .mergify/
+M: Michael D Kinney <michael.d.kinney@intel.com>
+M: Liming Gao <liming.gao@intel.com>
+R: Sean Brogan <sean.brogan@microsoft.com>
+R: Bret Barkelew <Bret.Barkelew@microsoft.com>
+
+.pytool/
+F: .pytool/
+M: Sean Brogan <sean.brogan@microsoft.com>
+M: Bret Barkelew <Bret.Barkelew@microsoft.com>
+R: Michael D Kinney <michael.d.kinney@intel.com>
+R: Liming Gao <liming.gao@intel.com>
+
EDK II Packages:
----------------
ArmPkg
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 02/22] .gitignore: Ignore python compiled files, extdeps, and vscode
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
2019-11-11 8:40 ` [Patch v5 01/22] Maintainers.txt: Add continuous integration(CI) directories Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file Michael D Kinney
` (19 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel
Cc: Sean Brogan, Andrew Fish, Laszlo Ersek, Leif Lindholm, Kinney,
Philippe Mathieu-Daude, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Update .gitignore to ignore .pyc files and __pycache__
directories. Python based plugins can be added to any
package or platform, so these files and directories may
be present outside of BaseTools.
Ignore _extdep directories that are generated by the
pytool external dependency feature.
Ignore .vscode directories generated by the VS Code
editor.
Cc: Andrew Fish <afish@apple.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Signed-off-by: Kinney <michael.d.kinney@intel.com>
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
.gitignore | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/.gitignore b/.gitignore
index 97f22c348c..1dd30c1410 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,7 @@
Build/
-tags/
.DS_Store
+*_extdep/
+*.pyc
+__pycache__/
+tags/
+.vscode/
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
2019-11-11 8:40 ` [Patch v5 01/22] Maintainers.txt: Add continuous integration(CI) directories Michael D Kinney
2019-11-11 8:40 ` [Patch v5 02/22] .gitignore: Ignore python compiled files, extdeps, and vscode Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 9:58 ` Laszlo Ersek
2019-11-11 8:40 ` [Patch v5 04/22] BaseTools: Add RC_PATH define for VS2017/2019 Michael D Kinney
` (18 subsequent siblings)
21 siblings, 1 reply; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Andrew Fish, Laszlo Ersek, Leif Lindholm, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Add pip requirements file that is used to install the
python pip modules build from the edk2-pytool-library and
edk2-pytool-extensions repositories.
These python modules provide the extensions required to
perform EDK II Continuous Integration(CI) builds.
Cc: Andrew Fish <afish@apple.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
pip_requirements.txt | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
create mode 100644 pip_requirements.txt
diff --git a/pip_requirements.txt b/pip_requirements.txt
new file mode 100644
index 0000000000..4ad72cfc98
--- /dev/null
+++ b/pip_requirements.txt
@@ -0,0 +1,17 @@
+## @file
+# EDK II Python PIP requirements file
+#
+# This file provides the list of python components to install using PIP.
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.<BR>
+#
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+# https://pypi.org/project/pip/
+# https://pip.pypa.io/en/stable/user_guide/#requirements-files
+# https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format
+#
+##
+
+edk2-pytool-library==0.10.*
+edk2-pytool-extensions==0.12.*
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 04/22] BaseTools: Add RC_PATH define for VS2017/2019
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (2 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 05/22] BaseTools: Add YAML files with path env and tool extdeps Michael D Kinney
` (17 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bob Feng, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Add use of RC_PATH define that provides the path to the resource
compiler that is typically provided in a Windows SDK. The path
changes with different Windows SDK releases. This define is set
to the WINSDK_PATH_FOR_RC_EXE environment variable. This
environment variable must be set to the path to the currently
installed resource compiler (rc.exe).
Update set_vsprefix_envs.bat to set WINSDK_PATH_FOR_RC_EXE
if a Windows SDK is detected.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
BaseTools/Conf/tools_def.template | 18 ++++++++++++++++--
BaseTools/set_vsprefix_envs.bat | 10 ++++++++++
2 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.template
index dce6c5875a..afa13a7cae 100755
--- a/BaseTools/Conf/tools_def.template
+++ b/BaseTools/Conf/tools_def.template
@@ -4,9 +4,18 @@
# Portions copyright (c) 2011 - 2014, ARM Ltd. All rights reserved.<BR>
# Copyright (c) 2015, Hewlett-Packard Development Company, L.P.<BR>
# (C) Copyright 2016 Hewlett Packard Enterprise Development LP<BR>
+# Copyright (c) Microsoft Corporation
#
# SPDX-License-Identifier: BSD-2-Clause-Patent
#
+#
+# Increase this version tag any time you want user to get warning about updating this
+# file in the Conf dir. By default it does not do update existing conf dirs.
+#
+# 2.00 - Initial version with changes for CI
+# - Change RC path to use plugin
+#
+#!VERSION=2.00
IDENTIFIER = Default TOOL_CHAIN_CONF
@@ -71,6 +80,11 @@ DEFINE VS2019_BIN_X64 = DEF(VS2019_BIN)\HostDEF(VS_HOST)\x64
DEFINE VS2019_BIN_ARM = DEF(VS2019_BIN)\HostDEF(VS_HOST)\arm
DEFINE VS2019_BIN_AARCH64 = DEF(VS2019_BIN)\HostDEF(VS_HOST)\arm64
+#
+# Resource compiler
+#
+DEFINE RC_PATH = ENV(WINSDK_PATH_FOR_RC_EXE)\rc.exe
+
DEFINE WINSDK_BIN = ENV(WINSDK_PREFIX)
DEFINE WINSDKx86_BIN = ENV(WINSDKx86_PREFIX)
@@ -1566,7 +1580,7 @@ NOOPT_VS2015x86_X64_DLINK_FLAGS = /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF
*_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe
*_VS2017_*_MAKE_FLAG = /nologo
-*_VS2017_*_RC_PATH = DEF(WINSDK10_BIN)\rc.exe
+*_VS2017_*_RC_PATH = DEF(RC_PATH)
*_VS2017_*_MAKE_FLAGS = /nologo
*_VS2017_*_SLINK_FLAGS = /NOLOGO /LTCG
@@ -1729,7 +1743,7 @@ NOOPT_VS2017_AARCH64_DLINK_FLAGS = /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF
*_VS2019_*_MAKE_PATH = DEF(VS2019_BIN_HOST)\nmake.exe
*_VS2019_*_MAKE_FLAG = /nologo
-*_VS2019_*_RC_PATH = DEF(WINSDK10_BIN)\rc.exe
+*_VS2019_*_RC_PATH = DEF(RC_PATH)
*_VS2019_*_MAKE_FLAGS = /nologo
*_VS2019_*_SLINK_FLAGS = /NOLOGO /LTCG
diff --git a/BaseTools/set_vsprefix_envs.bat b/BaseTools/set_vsprefix_envs.bat
index 6dff25d930..9966a5a4ed 100644
--- a/BaseTools/set_vsprefix_envs.bat
+++ b/BaseTools/set_vsprefix_envs.bat
@@ -158,6 +158,11 @@ if defined VCToolsInstallDir (
)
)
)
+if not defined WINSDK_PATH_FOR_RC_EXE (
+ if defined WINSDK10_PREFIX (
+ set "WINSDK_PATH_FOR_RC_EXE=%WINSDK10_PREFIX%x86"
+ )
+)
:SetVS2017
if not defined VS150COMNTOOLS (
@@ -209,6 +214,11 @@ if defined VCToolsInstallDir (
)
)
)
+if not defined WINSDK_PATH_FOR_RC_EXE (
+ if defined WINSDK10_PREFIX (
+ set "WINSDK_PATH_FOR_RC_EXE=%WINSDK10_PREFIX%x86"
+ )
+)
:SetWinDDK
if not defined WINDDK3790_PREFIX (
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 05/22] BaseTools: Add YAML files with path env and tool extdeps
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (3 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 04/22] BaseTools: Add RC_PATH define for VS2017/2019 Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 06/22] BaseTools: Add BaseTools plugins to support CI Michael D Kinney
` (16 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bob Feng, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml | 21 +++
BaseTools/Bin/gcc_arm_linux_ext_dep.yaml | 21 +++
BaseTools/Bin/iasl_ext_dep.yaml | 21 +++
BaseTools/Bin/nasm_ext_dep.yaml | 18 ++
.../BinWrappers/PosixLike/posix_path_env.yaml | 10 ++
.../WindowsLike/win_build_tools_path_env.yaml | 10 ++
BaseTools/Edk2ToolsBuild.py | 163 ++++++++++++++++++
.../basetool_tiano_python_path_env.yaml | 11 ++
BaseTools/basetools_calling_path_env.yaml | 11 ++
BaseTools/basetools_path_env.yaml | 11 ++
10 files changed, 297 insertions(+)
create mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
create mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
create mode 100644 BaseTools/Bin/iasl_ext_dep.yaml
create mode 100644 BaseTools/Bin/nasm_ext_dep.yaml
create mode 100644 BaseTools/BinWrappers/PosixLike/posix_path_env.yaml
create mode 100644 BaseTools/BinWrappers/WindowsLike/win_build_tools_path_env.yaml
create mode 100644 BaseTools/Edk2ToolsBuild.py
create mode 100644 BaseTools/Source/Python/basetool_tiano_python_path_env.yaml
create mode 100644 BaseTools/basetools_calling_path_env.yaml
create mode 100644 BaseTools/basetools_path_env.yaml
diff --git a/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
new file mode 100644
index 0000000000..bf31cbd8d1
--- /dev/null
+++ b/BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
@@ -0,0 +1,21 @@
+## @file
+# Download GCC AARCH64 compiler from Linaro's release site
+# Set shell variable GCC5_AARCH64_INSTALL to this folder
+#
+# This is only downloaded when a build activates scope gcc_aarch64_linux
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "gcc_aarch64_linux",
+ "type": "web",
+ "name": "gcc_aarch64_linux",
+ "source": "http://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz",
+ "version": "7.4.1",
+ "sha256": "27f1dc2c491ed61ae8f0d4b0c11de59cd2f7dd9c94761ee7153006fcac1bf9ab",
+ "compression_type": "tar",
+ "internal_path": "/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu/",
+ "flags": ["set_shell_var", ],
+ "var_name": "GCC5_AARCH64_INSTALL"
+}
diff --git a/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
new file mode 100644
index 0000000000..4105f5e4aa
--- /dev/null
+++ b/BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
@@ -0,0 +1,21 @@
+## @file
+# Download GCC ARM compiler from Linaro's release site
+# Set shell variable GCC5_ARM_INSTALL to this folder
+#
+# This is only downloaded when a build activates scope gcc_arm_linux
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "gcc_arm_linux",
+ "type": "web",
+ "name": "gcc_arm_linux",
+ "source": "https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/arm-linux-gnueabihf/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf.tar.xz",
+ "version": "7.4.1",
+ "sha256": "3C951CF1941D0FA06D64CC0D5E88612B209D8123B273FA26C16D70BD7BC6B163",
+ "compression_type": "tar",
+ "internal_path": "/gcc-linaro-7.4.1-2019.02-x86_64_arm-linux-gnueabihf/",
+ "flags": ["set_shell_var", ],
+ "var_name": "GCC5_ARM_INSTALL"
+}
diff --git a/BaseTools/Bin/iasl_ext_dep.yaml b/BaseTools/Bin/iasl_ext_dep.yaml
new file mode 100644
index 0000000000..7168eafe47
--- /dev/null
+++ b/BaseTools/Bin/iasl_ext_dep.yaml
@@ -0,0 +1,21 @@
+## @file
+# Download iasl executable tool from a nuget.org package
+# - package contains different binaries based on host
+# Add the folder with the tool to the path
+#
+# This is only downloaded for scope cibuild thus
+# should have no impact on the asl compiler used by any
+# given platform to build.
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "id": "iasl-ci-1",
+ "scope": "cibuild",
+ "type": "nuget",
+ "name": "iasl",
+ "source": "https://api.nuget.org/v3/index.json",
+ "version": "20190215.0.0",
+ "flags": ["set_path", "host_specific"]
+}
diff --git a/BaseTools/Bin/nasm_ext_dep.yaml b/BaseTools/Bin/nasm_ext_dep.yaml
new file mode 100644
index 0000000000..cc3156662e
--- /dev/null
+++ b/BaseTools/Bin/nasm_ext_dep.yaml
@@ -0,0 +1,18 @@
+## @file
+# Download nasm x86 assembler executable tool from a nuget.org package
+# - package contains different binaries based on host
+# Put on the tool on the path
+#
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "id": "nasm-1",
+ "scope": "edk2-build",
+ "type": "nuget",
+ "name": "mu_nasm",
+ "source": "https://api.nuget.org/v3/index.json",
+ "version": "2.14.02",
+ "flags": ["set_path", "host_specific"]
+}
diff --git a/BaseTools/BinWrappers/PosixLike/posix_path_env.yaml b/BaseTools/BinWrappers/PosixLike/posix_path_env.yaml
new file mode 100644
index 0000000000..ab5187efc8
--- /dev/null
+++ b/BaseTools/BinWrappers/PosixLike/posix_path_env.yaml
@@ -0,0 +1,10 @@
+## @file
+# Set this folder on the path for all linux builds
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global-nix",
+ "flags": ["set_path"]
+}
diff --git a/BaseTools/BinWrappers/WindowsLike/win_build_tools_path_env.yaml b/BaseTools/BinWrappers/WindowsLike/win_build_tools_path_env.yaml
new file mode 100644
index 0000000000..1e6dd21b70
--- /dev/null
+++ b/BaseTools/BinWrappers/WindowsLike/win_build_tools_path_env.yaml
@@ -0,0 +1,10 @@
+## @file
+# Add this folder to the path on Windows
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global-win",
+ "flags": ["set_path"]
+}
diff --git a/BaseTools/Edk2ToolsBuild.py b/BaseTools/Edk2ToolsBuild.py
new file mode 100644
index 0000000000..057d2e9e06
--- /dev/null
+++ b/BaseTools/Edk2ToolsBuild.py
@@ -0,0 +1,163 @@
+# @file Edk2ToolsBuild.py
+# Invocable class that builds the basetool c files.
+#
+# Supports VS2017, VS2019, and GCC5
+##
+# Copyright (c) Microsoft Corporation
+#
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import sys
+import logging
+import argparse
+from edk2toolext import edk2_logging
+from edk2toolext.environment import self_describing_environment
+from edk2toolext.base_abstract_invocable import BaseAbstractInvocable
+from edk2toollib.utility_functions import RunCmd
+from edk2toollib.windows.locate_tools import QueryVcVariables
+
+
+class Edk2ToolsBuild(BaseAbstractInvocable):
+
+ def ParseCommandLineOptions(self):
+ ''' parse arguments '''
+ ParserObj = argparse.ArgumentParser()
+ ParserObj.add_argument("-t", "--tool_chain_tag", dest="tct", default="VS2017",
+ help="Set the toolchain used to compile the build tools")
+ args = ParserObj.parse_args()
+ self.tool_chain_tag = args.tct
+
+ def GetWorkspaceRoot(self):
+ ''' Return the workspace root for initializing the SDE '''
+
+ # this is the bastools dir...not the traditional EDK2 workspace root
+ return os.path.dirname(os.path.abspath(__file__))
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+
+ # for now don't use scopes
+ return ('global',)
+
+ def GetLoggingLevel(self, loggerType):
+ ''' Get the logging level for a given type (return Logging.Level)
+ base == lowest logging level supported
+ con == Screen logging
+ txt == plain text file logging
+ md == markdown file logging
+ '''
+ if(loggerType == "con"):
+ return logging.ERROR
+ else:
+ return logging.DEBUG
+
+ def GetLoggingFolderRelativeToRoot(self):
+ ''' Return a path to folder for log files '''
+ return "BaseToolsBuild"
+
+ def GetVerifyCheckRequired(self):
+ ''' Will call self_describing_environment.VerifyEnvironment if this returns True '''
+ return True
+
+ def GetLoggingFileName(self, loggerType):
+ ''' Get the logging file name for the type.
+ Return None if the logger shouldn't be created
+
+ base == lowest logging level supported
+ con == Screen logging
+ txt == plain text file logging
+ md == markdown file logging
+ '''
+ return "BASETOOLS_BUILD"
+
+ def WritePathEnvFile(self, OutputDir):
+ ''' Write a PyTool path env file for future PyTool based edk2 builds'''
+ content = '''##
+# Set shell variable EDK_TOOLS_BIN to this folder
+#
+# Autogenerated by Edk2ToolsBuild.py
+#
+# Copyright (c), Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "id": "You-Built-BaseTools",
+ "scope": "edk2-build",
+ "flags": ["set_shell_var", "set_path"],
+ "var_name": "EDK_TOOLS_BIN"
+}
+'''
+ with open(os.path.join(OutputDir, "basetoolsbin_path_env.yaml"), "w") as f:
+ f.write(content)
+
+ def Go(self):
+ logging.info("Running Python version: " + str(sys.version_info))
+
+ (build_env, shell_env) = self_describing_environment.BootstrapEnvironment(
+ self.GetWorkspaceRoot(), self.GetActiveScopes())
+
+ # # Bind our current execution environment into the shell vars.
+ ph = os.path.dirname(sys.executable)
+ if " " in ph:
+ ph = '"' + ph + '"'
+ shell_env.set_shell_var("PYTHON_HOME", ph)
+ # PYTHON_COMMAND is required to be set for using edk2 python builds.
+ pc = sys.executable
+ if " " in pc:
+ pc = '"' + pc + '"'
+ shell_env.set_shell_var("PYTHON_COMMAND", pc)
+
+ if self.tool_chain_tag.lower().startswith("vs"):
+
+ # # Update environment with required VC vars.
+ interesting_keys = ["ExtensionSdkDir", "INCLUDE", "LIB"]
+ interesting_keys.extend(
+ ["LIBPATH", "Path", "UniversalCRTSdkDir", "UCRTVersion", "WindowsLibPath", "WindowsSdkBinPath"])
+ interesting_keys.extend(
+ ["WindowsSdkDir", "WindowsSdkVerBinPath", "WindowsSDKVersion", "VCToolsInstallDir"])
+ vc_vars = QueryVcVariables(
+ interesting_keys, 'x86', vs_version=self.tool_chain_tag.lower())
+ for key in vc_vars.keys():
+ logging.debug(f"Var - {key} = {vc_vars[key]}")
+ if key.lower() == 'path':
+ shell_env.insert_path(vc_vars[key])
+ else:
+ shell_env.set_shell_var(key, vc_vars[key])
+
+ self.OutputDir = os.path.join(
+ shell_env.get_shell_var("EDK_TOOLS_PATH"), "Bin", "Win32")
+
+ # compiled tools need to be added to path because antlr is referenced
+ shell_env.insert_path(self.OutputDir)
+
+ # Actually build the tools.
+ ret = RunCmd('nmake.exe', None,
+ workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
+ if ret != 0:
+ raise Exception("Failed to build.")
+
+ self.WritePathEnvFile(self.OutputDir)
+ return ret
+
+ elif self.tool_chain_tag.lower().startswith("gcc"):
+ ret = RunCmd("make", "-C .", workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
+ if ret != 0:
+ raise Exception("Failed to build.")
+
+ self.OutputDir = os.path.join(
+ shell_env.get_shell_var("EDK_TOOLS_PATH"), "Source", "C", "bin")
+
+ self.WritePathEnvFile(self.OutputDir)
+ return ret
+
+ logging.critical("Tool Chain not supported")
+ return -1
+
+
+def main():
+ Edk2ToolsBuild().Invoke()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/BaseTools/Source/Python/basetool_tiano_python_path_env.yaml b/BaseTools/Source/Python/basetool_tiano_python_path_env.yaml
new file mode 100644
index 0000000000..12d517d32c
--- /dev/null
+++ b/BaseTools/Source/Python/basetool_tiano_python_path_env.yaml
@@ -0,0 +1,11 @@
+## @file
+# Add this folder to the pypath so modules can be easily
+# loaded
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global",
+ "flags": ["set_pypath"]
+}
diff --git a/BaseTools/basetools_calling_path_env.yaml b/BaseTools/basetools_calling_path_env.yaml
new file mode 100644
index 0000000000..2e444cd480
--- /dev/null
+++ b/BaseTools/basetools_calling_path_env.yaml
@@ -0,0 +1,11 @@
+## @file
+# Set shell variable BASE_TOOL_PATH to this folder
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global",
+ "flags": ["set_shell_var"],
+ "var_name": "BASE_TOOLS_PATH"
+}
diff --git a/BaseTools/basetools_path_env.yaml b/BaseTools/basetools_path_env.yaml
new file mode 100644
index 0000000000..23ce5533ba
--- /dev/null
+++ b/BaseTools/basetools_path_env.yaml
@@ -0,0 +1,11 @@
+## @file
+# Set shell variable EDK_TOOL_PATH to this folder
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global",
+ "flags": ["set_shell_var", "set_build_var"],
+ "var_name": "EDK_TOOLS_PATH"
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 06/22] BaseTools: Add BaseTools plugins to support CI
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (4 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 05/22] BaseTools: Add YAML files with path env and tool extdeps Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 07/22] .pytool/Plugin: Add CI plugins Michael D Kinney
` (15 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bob Feng, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Add the following plugins that are required to support
EDK II Continuous Integration (CI) builds. These plugins
are added to BaseTools because that support EDK II BaseTools
features.
* BuildToolsReportGenerator
* LinuxGcc5ToolChain
* WindowsResourceCompiler
* WindowsVsToolChain
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
.../BuildToolsReportGenerator.py | 69 ++++++++++
.../BuildToolsReportGenerator_plug_in.yaml | 12 ++
.../BuildToolsReport_Template.html | 126 ++++++++++++++++++
.../LinuxGcc5ToolChain/LinuxGcc5ToolChain.py | 85 ++++++++++++
.../LinuxGcc5ToolChain_plug_in.yaml | 12 ++
.../WindowsResourceCompiler/WinRcPath.py | 29 ++++
.../WinRcPath_plug_in.yaml | 13 ++
.../WindowsVsToolChain/WindowsVsToolChain.py | 126 ++++++++++++++++++
.../WindowsVsToolChain_plug_in.yaml | 11 ++
9 files changed, 483 insertions(+)
create mode 100644 BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator.py
create mode 100644 BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator_plug_in.yaml
create mode 100644 BaseTools/Plugin/BuildToolsReport/BuildToolsReport_Template.html
create mode 100644 BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain.py
create mode 100644 BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain_plug_in.yaml
create mode 100644 BaseTools/Plugin/WindowsResourceCompiler/WinRcPath.py
create mode 100644 BaseTools/Plugin/WindowsResourceCompiler/WinRcPath_plug_in.yaml
create mode 100644 BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain.py
create mode 100644 BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain_plug_in.yaml
diff --git a/BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator.py b/BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator.py
new file mode 100644
index 0000000000..d5df9f2f9b
--- /dev/null
+++ b/BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator.py
@@ -0,0 +1,69 @@
+##
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+import json
+
+try:
+ from edk2toolext.environment.plugintypes.uefi_build_plugin import IUefiBuildPlugin
+
+ class BuildToolsReportGenerator(IUefiBuildPlugin):
+ def do_report(self, thebuilder):
+ try:
+ from edk2toolext.environment import version_aggregator
+ except ImportError:
+ logging.critical("Loading BuildToolsReportGenerator failed, please update your Edk2-PyTool-Extensions")
+ return 0
+
+ OutputReport = os.path.join(thebuilder.env.GetValue("BUILD_OUTPUT_BASE"), "BUILD_TOOLS_REPORT")
+ OutputReport = os.path.normpath(OutputReport)
+ if not os.path.isdir(os.path.dirname(OutputReport)):
+ os.makedirs(os.path.dirname(OutputReport))
+
+ Report = BuildToolsReport()
+ Report.MakeReport(version_aggregator.GetVersionAggregator().GetAggregatedVersionInformation(), OutputReport=OutputReport)
+
+ def do_pre_build(self, thebuilder):
+ self.do_report(thebuilder)
+ return 0
+
+ def do_post_build(self, thebuilder):
+ self.do_report(thebuilder)
+ return 0
+
+except ImportError:
+ pass
+
+
+class BuildToolsReport(object):
+ MY_FOLDER = os.path.dirname(os.path.realpath(__file__))
+ VERSION = "1.00"
+
+ def __init__(self):
+ pass
+
+ def MakeReport(self, BuildTools, OutputReport="BuildToolsReport"):
+ logging.info("Writing BuildToolsReports to {0}".format(OutputReport))
+ versions_list = []
+ for key, value in BuildTools.items():
+ versions_list.append(value)
+ versions_list = sorted(versions_list, key=lambda k: k['type'])
+ json_dict = {"modules": versions_list,
+ "PluginVersion": BuildToolsReport.VERSION}
+
+ htmlfile = open(OutputReport + ".html", "w")
+ jsonfile = open(OutputReport + ".json", "w")
+ template = open(os.path.join(BuildToolsReport.MY_FOLDER, "BuildToolsReport_Template.html"), "r")
+
+ for line in template.readlines():
+ if "%TO_BE_FILLED_IN_BY_PYTHON_SCRIPT%" in line:
+ line = line.replace("%TO_BE_FILLED_IN_BY_PYTHON_SCRIPT%", json.dumps(json_dict))
+ htmlfile.write(line)
+
+ jsonfile.write(json.dumps(versions_list, indent=4))
+
+ jsonfile.close()
+ template.close()
+ htmlfile.close()
diff --git a/BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator_plug_in.yaml b/BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator_plug_in.yaml
new file mode 100644
index 0000000000..9e07c4889b
--- /dev/null
+++ b/BaseTools/Plugin/BuildToolsReport/BuildToolsReportGenerator_plug_in.yaml
@@ -0,0 +1,12 @@
+## @file
+# Build Plugin used to output html report of all versions collected
+# during the build
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global",
+ "name": "Build Tools Report Generator",
+ "module": "BuildToolsReportGenerator"
+}
diff --git a/BaseTools/Plugin/BuildToolsReport/BuildToolsReport_Template.html b/BaseTools/Plugin/BuildToolsReport/BuildToolsReport_Template.html
new file mode 100644
index 0000000000..26350ab425
--- /dev/null
+++ b/BaseTools/Plugin/BuildToolsReport/BuildToolsReport_Template.html
@@ -0,0 +1,126 @@
+<!doctype html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible">
+ <title>Build Tools Report</title>
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link rel="stylesheet" type="text/css" href="https://ajax.aspnetcdn.com/ajax/bootstrap/3.3.7/css/bootstrap.min.css" />
+ <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.15/css/dataTables.bootstrap.min.css" />
+ <style>
+ div.attribution {
+ border: 1px solid #ddd;
+ background-color: #bbb;
+ padding-left: 20px;
+ }
+ </style>
+</head>
+<body>
+ <div class="container-fluid">
+ <h1>Build Tools Report</h1>
+ <ul class="nav nav-tabs">
+ <li class="active"><a data-toggle="tab" href="#tabs-1">Tools</a></li>
+ <li><a data-toggle="tab" href="#tabs-2">About</a></li>
+ </ul>
+ <div class="tab-content">
+ <div id="tabs-1" class="tab-pane fade in active">
+ <table id="modinfo" class="table table-striped table-bordered table-hover" cellspacing="0">
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Value</th>
+ <th>Type</th>
+ </tr>
+ </thead>
+ <tbody></tbody>
+ </table>
+ </div>
+ <div id="tabs-2" class="tab-pane">
+ <div class="row">
+ <div class="col-xs-7">
+ <p></p>
+ <p>
+ Build Tools Report Template Version: <span id="ReportTemplateVersion">1.00</span><br />
+ Build Tools Report Plugin Version: <span id='ReportToolVersion'></span><br />
+ </p>
+ <h3>License</h3>
+ <hr />
+ <div id="ToolLicenseContent">
+ <p>
+ <span class="copyright">Copyright (c) Microsoft Corporation. All rights reserved.</span><br />
+ <span class="license">
+ SPDX-License-Identifier: BSD-2-Clause-Patent
+ </span>
+ </p>
+ </div>
+ </div>
+ <div id="AttributionListWrapper" class="col-xs-5">
+ <h3>External Licenses</h3>
+ </div>
+ </div>
+ </div>
+ </div>
+ </div>
+
+ <!-- Javascript libraries -->
+ <script type="text/javascript" charset="utf8" src="https://ajax.aspnetcdn.com/ajax/jQuery/jquery-3.2.1.min.js"></script>
+ <script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.15/js/jquery.dataTables.min.js"></script>
+ <script type="text/javascript" charset="utf8" src="https://ajax.aspnetcdn.com/ajax/bootstrap/3.3.7/bootstrap.min.js"></script>
+ <script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.15/js/dataTables.bootstrap.min.js"></script>
+
+ <script>
+ var EmbeddedJd = %TO_BE_FILLED_IN_BY_PYTHON_SCRIPT%;
+ </script>
+ <!-- Add javascript here -->
+ <script>
+ var MODULE_TABLE_OFFSET = 350; //Space needed for other stuff besides the Table
+ $(document).ready(function () {
+ $('span#ReportToolVersion').text(EmbeddedJd.PluginVersion);
+ //To support tabs and correct column width we need this change
+ $('a[data-toggle="tab"][href="#tabs-1"]').on('shown.bs.tab', function (e) {
+ $.fn.dataTable.tables({ visible: true, api: true }).columns.adjust();
+ });
+ //table for modules
+ var mTable = $('table#modinfo').dataTable({
+ "aaData": EmbeddedJd.modules,
+ "paginate": false,
+ "autoWidth": false,
+ "scrollY": ($(window).height() - MODULE_TABLE_OFFSET) + "px",
+ "aaSorting": [[2, "asc"]],
+ "aoColumnDefs": [
+ {
+ "mData": "name",
+ "aTargets": [0]
+ },
+
+ {
+ "mData": "version",
+ "aTargets": [1]
+ },
+ {
+ "mData": "type",
+ "aTargets": [2],
+ }
+ ] //end of column def
+ }); //end of modules table
+
+ //
+ // Create Attribution List for all external libraries used
+ //
+ [
+ { Title: "JQuery", Copyright: "Copyright 2017 The jQuery Foundation", Version: $.fn.jquery, LicenseType: "MIT", LicenseLink: "https://jquery.org/license/" },
+ { Title: "DataTables", Copyright: "DataTables designed and created by SpryMedia Ltd Copyright 2007-2017", Version: $.fn.dataTable.version, LicenseType: "MIT", LicenseLink: "https://datatables.net/license/mit" },
+ { Title: "BootStrap", Copyright: "Code and documentation copyright 2011-2017 the Bootstrap Authors and Twitter, Inc.", Version: "3.3.7", LicenseType: "MIT", LicenseLink: "https://github.com/twbs/bootstrap/blob/master/LICENSE" }
+ ].forEach(function (element) {
+ $("<div class='attribution'><h4>" + element.Title + "</h4><p>Version: <span class='version'>" + element.Version + "</span><br /><span class='copyright'>" +
+ element.Copyright + "</span><br />License: <a class='license' href='" + element.LicenseLink + "'>" + element.LicenseType + "</a></p></div>").appendTo("div#AttributionListWrapper");
+ });
+ });
+ $(window).resize(function() {
+ $.fn.dataTable.tables({ visible: true, api: true }).columns.adjust();
+ });
+
+
+ </script>
+</body>
+</html>
diff --git a/BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain.py b/BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain.py
new file mode 100644
index 0000000000..c31641e931
--- /dev/null
+++ b/BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain.py
@@ -0,0 +1,85 @@
+# @file LinuxGcc5ToolChain.py
+# Plugin to configures paths for GCC5 ARM/AARCH64 Toolchain
+##
+# This plugin works in conjuncture with the tools_def
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+from edk2toolext.environment.plugintypes.uefi_build_plugin import IUefiBuildPlugin
+from edk2toolext.environment import shell_environment
+
+
+class LinuxGcc5ToolChain(IUefiBuildPlugin):
+
+ def do_post_build(self, thebuilder):
+ return 0
+
+ def do_pre_build(self, thebuilder):
+ self.Logger = logging.getLogger("LinuxGcc5ToolChain")
+
+ #
+ # GCC5 - The ARM and AARCH64 compilers need their paths set if available
+ if thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "GCC5":
+
+ # Start with AARACH64 compiler
+ ret = self._check_aarch64()
+ if ret != 0:
+ self.Logger.critical("Failed in check aarch64")
+ return ret
+
+ # Check arm compiler
+ ret = self._check_arm()
+ if ret != 0:
+ self.Logger.critical("Failed in check arm")
+ return ret
+
+ return 0
+
+ def _check_arm(self):
+ # check to see if full path already configured
+ if shell_environment.GetEnvironment().get_shell_var("GCC5_ARM_PREFIX") is not None:
+ self.Logger.info("GCC5_ARM_PREFIX is already set.")
+
+ else:
+ # now check for install dir. If set then set the Prefix
+ install_path = shell_environment.GetEnvironment().get_shell_var("GCC5_ARM_INSTALL")
+ if install_path is None:
+ return 0
+
+ # make GCC5_ARM_PREFIX to align with tools_def.txt
+ prefix = os.path.join(install_path, "bin", "arm-linux-gnueabihf-")
+ shell_environment.GetEnvironment().set_shell_var("GCC5_ARM_PREFIX", prefix)
+
+ # now confirm it exists
+ if not os.path.exists(shell_environment.GetEnvironment().get_shell_var("GCC5_ARM_PREFIX") + "gcc"):
+ self.Logger.error("Path for GCC5_ARM_PREFIX toolchain is invalid")
+ return -2
+
+ return 0
+
+ def _check_aarch64(self):
+ # check to see if full path already configured
+ if shell_environment.GetEnvironment().get_shell_var("GCC5_AARCH64_PREFIX") is not None:
+ self.Logger.info("GCC5_AARCH64_PREFIX is already set.")
+
+ else:
+ # now check for install dir. If set then set the Prefix
+ install_path = shell_environment.GetEnvironment(
+ ).get_shell_var("GCC5_AARCH64_INSTALL")
+ if install_path is None:
+ return 0
+
+ # make GCC5_AARCH64_PREFIX to align with tools_def.txt
+ prefix = os.path.join(install_path, "bin", "aarch64-linux-gnu-")
+ shell_environment.GetEnvironment().set_shell_var("GCC5_AARCH64_PREFIX", prefix)
+
+ # now confirm it exists
+ if not os.path.exists(shell_environment.GetEnvironment().get_shell_var("GCC5_AARCH64_PREFIX") + "gcc"):
+ self.Logger.error(
+ "Path for GCC5_AARCH64_PREFIX toolchain is invalid")
+ return -2
+
+ return 0
diff --git a/BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain_plug_in.yaml b/BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain_plug_in.yaml
new file mode 100644
index 0000000000..aee971606b
--- /dev/null
+++ b/BaseTools/Plugin/LinuxGcc5ToolChain/LinuxGcc5ToolChain_plug_in.yaml
@@ -0,0 +1,12 @@
+## @file
+# Build Plugin used to set the path
+# for the GCC5 ARM/AARCH64 downloaded compilers
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global-nix",
+ "name": "Linux GCC5 Tool Chain Support",
+ "module": "LinuxGcc5ToolChain"
+}
diff --git a/BaseTools/Plugin/WindowsResourceCompiler/WinRcPath.py b/BaseTools/Plugin/WindowsResourceCompiler/WinRcPath.py
new file mode 100644
index 0000000000..ec2f2d1298
--- /dev/null
+++ b/BaseTools/Plugin/WindowsResourceCompiler/WinRcPath.py
@@ -0,0 +1,29 @@
+## @file WinRcPath.py
+# Plugin to find Windows SDK Resource Compiler rc.exe
+##
+# This plugin works in conjuncture with the tools_def to support rc.exe
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+from edk2toolext.environment.plugintypes.uefi_build_plugin import IUefiBuildPlugin
+import edk2toollib.windows.locate_tools as locate_tools
+from edk2toolext.environment import shell_environment
+from edk2toolext.environment import version_aggregator
+
+class WinRcPath(IUefiBuildPlugin):
+
+ def do_post_build(self, thebuilder):
+ return 0
+
+ def do_pre_build(self, thebuilder):
+ #get the locate tools module
+ path = locate_tools.FindToolInWinSdk("rc.exe")
+ if path is None:
+ thebuilder.logging.warning("Failed to find rc.exe")
+ else:
+ p = os.path.abspath(os.path.dirname(path))
+ shell_environment.GetEnvironment().set_shell_var("WINSDK_PATH_FOR_RC_EXE", p)
+ version_aggregator.GetVersionAggregator().ReportVersion("WINSDK_PATH_FOR_RC_EXE", p, version_aggregator.VersionTypes.INFO)
+ return 0
diff --git a/BaseTools/Plugin/WindowsResourceCompiler/WinRcPath_plug_in.yaml b/BaseTools/Plugin/WindowsResourceCompiler/WinRcPath_plug_in.yaml
new file mode 100644
index 0000000000..253f22f833
--- /dev/null
+++ b/BaseTools/Plugin/WindowsResourceCompiler/WinRcPath_plug_in.yaml
@@ -0,0 +1,13 @@
+## @file
+# Build Plugin used to set the path to rc.exe on windows.
+# The plugin is able to use python to locate the tool as to avoid
+# hard-coding the path
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global-win",
+ "name": "Windows RC Path Support",
+ "module": "WinRcPath"
+}
diff --git a/BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain.py b/BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain.py
new file mode 100644
index 0000000000..a8202e5992
--- /dev/null
+++ b/BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain.py
@@ -0,0 +1,126 @@
+## @file WindowsVsToolChain.py
+# Plugin to configures paths for the VS2017 and VS2019 tool chain
+##
+# This plugin works in conjuncture with the tools_def
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+from edk2toolext.environment.plugintypes.uefi_build_plugin import IUefiBuildPlugin
+import edk2toollib.windows.locate_tools as locate_tools
+from edk2toollib.windows.locate_tools import FindWithVsWhere
+from edk2toolext.environment import shell_environment
+from edk2toolext.environment import version_aggregator
+
+class WindowsVsToolChain(IUefiBuildPlugin):
+
+ def do_post_build(self, thebuilder):
+ return 0
+
+ def do_pre_build(self, thebuilder):
+ self.Logger = logging.getLogger("WindowsVsToolChain")
+
+#
+ # VS2017 - Follow VS2017 where there is potential for many versions of the tools.
+ # If a specific version is required then the user must set both env variables:
+ ## VS150INSTALLPATH: base install path on system to VC install dir. Here you will find the VC folder, etc
+ ## VS150TOOLVER: version number for the VC compiler tools
+ ## VS2017_PREFIX: path to MSVC compiler folder with trailing slash (can be used instead of two vars above)
+ if thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "VS2017":
+
+ # check to see if full path already configured
+ if shell_environment.GetEnvironment().get_shell_var("VS2017_PREFIX") != None:
+ self.Logger.info("VS2017_PREFIX is already set.")
+
+ else:
+ install_path = self._get_vs_install_path("VS2017".lower(), "VS150INSTALLPATH")
+ vc_ver = self._get_vc_version(install_path, "VS150TOOLVER")
+
+ if install_path is None or vc_ver is None:
+ self.Logger.error("Failed to configure environment for VS2017")
+ return -1
+
+ version_aggregator.GetVersionAggregator().ReportVersion(
+ "Visual Studio Install Path", install_path, version_aggregator.VersionTypes.INFO)
+ version_aggregator.GetVersionAggregator().ReportVersion(
+ "VC Version", vc_ver, version_aggregator.VersionTypes.TOOL)
+
+ #make VS2017_PREFIX to align with tools_def.txt
+ prefix = os.path.join(install_path, "VC", "Tools", "MSVC", vc_ver)
+ prefix = prefix + os.path.sep
+ shell_environment.GetEnvironment().set_shell_var("VS2017_PREFIX", prefix)
+
+ # now confirm it exists
+ if not os.path.exists(shell_environment.GetEnvironment().get_shell_var("VS2017_PREFIX")):
+ self.Logger.error("Path for VS2017 toolchain is invalid")
+ return -2
+
+ #
+ # VS2019 - Follow VS2019 where there is potential for many versions of the tools.
+ # If a specific version is required then the user must set both env variables:
+ ## VS160INSTALLPATH: base install path on system to VC install dir. Here you will find the VC folder, etc
+ ## VS160TOOLVER: version number for the VC compiler tools
+ ## VS2019_PREFIX: path to MSVC compiler folder with trailing slash (can be used instead of two vars above)
+ elif thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "VS2019":
+
+ # check to see if full path already configured
+ if shell_environment.GetEnvironment().get_shell_var("VS2019_PREFIX") != None:
+ self.Logger.info("VS2019_PREFIX is already set.")
+
+ else:
+ install_path = self._get_vs_install_path("VS2019".lower(), "VS160INSTALLPATH")
+ vc_ver = self._get_vc_version(install_path, "VS160TOOLVER")
+
+ if install_path is None or vc_ver is None:
+ self.Logger.error("Failed to configure environment for VS2019")
+ return -1
+
+ version_aggregator.GetVersionAggregator().ReportVersion(
+ "Visual Studio Install Path", install_path, version_aggregator.VersionTypes.INFO)
+ version_aggregator.GetVersionAggregator().ReportVersion(
+ "VC Version", vc_ver, version_aggregator.VersionTypes.TOOL)
+
+ #make VS2019_PREFIX to align with tools_def.txt
+ prefix = os.path.join(install_path, "VC", "Tools", "MSVC", vc_ver)
+ prefix = prefix + os.path.sep
+ shell_environment.GetEnvironment().set_shell_var("VS2019_PREFIX", prefix)
+
+ # now confirm it exists
+ if not os.path.exists(shell_environment.GetEnvironment().get_shell_var("VS2019_PREFIX")):
+ self.Logger.error("Path for VS2019 toolchain is invalid")
+ return -2
+
+ return 0
+
+ def _get_vs_install_path(self, vs_version, varname):
+ # check if already specified
+ path = shell_environment.GetEnvironment().get_shell_var(varname)
+ if(path is None):
+ # Not specified...find latest
+ (rc, path) = FindWithVsWhere(vs_version=vs_version)
+ if rc == 0 and path is not None and os.path.exists(path):
+ self.Logger.debug("Found VS instance for %s", vs_version)
+ else:
+ self.Logger.error("Failed to find VS instance with VsWhere (%d)" % rc)
+ return path
+
+ def _get_vc_version(self, path, varname):
+ # check if already specified
+ vc_ver = shell_environment.GetEnvironment().get_shell_var(varname)
+ if (path is None):
+ self.Logger.critical("Failed to find Visual Studio tools. Might need to check for VS install")
+ return vc_ver
+ if(vc_ver is None):
+ # Not specified...find latest
+ p2 = os.path.join(path, "VC", "Tools", "MSVC")
+ if not os.path.isdir(p2):
+ self.Logger.critical(
+ "Failed to find VC tools. Might need to check for VS install")
+ return vc_ver
+ vc_ver = os.listdir(p2)[-1].strip() # get last in list
+ self.Logger.debug("Found VC Tool version is %s" % vc_ver)
+ return vc_ver
+
+
diff --git a/BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain_plug_in.yaml b/BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain_plug_in.yaml
new file mode 100644
index 0000000000..30b8ec8766
--- /dev/null
+++ b/BaseTools/Plugin/WindowsVsToolChain/WindowsVsToolChain_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# Build Plugin used to set the path to the visual studio tools chain
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "global-win",
+ "name": "Windows Visual Studio Tool Chain Support",
+ "module": "WindowsVsToolChain"
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 07/22] .pytool/Plugin: Add CI plugins
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (5 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 06/22] BaseTools: Add BaseTools plugins to support CI Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 08/22] CryptoPkg: Add YAML file for CI builds Michael D Kinney
` (14 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bret Barkelew, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Add .pytool directory to the edk2 repository with the
following plugins. These plugins are in a top level
directory because that can be used with all packages
and platforms.
* CharEncodingCheck
* CompilerPlugin
* DependencyCheck
* DscCompleteCheck
* GuidCheck
* LibraryClassCheck
* SpellCheck
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
.../CharEncodingCheck/CharEncodingCheck.py | 118 ++++++++
.../CharEncodingCheck_plug_in.yaml | 11 +
.pytool/Plugin/CharEncodingCheck/Readme.md | 18 ++
.../Plugin/CompilerPlugin/CompilerPlugin.py | 102 +++++++
.../CompilerPlugin/Compiler_plug_in.yaml | 11 +
.pytool/Plugin/CompilerPlugin/Readme.md | 17 ++
.../Plugin/DependencyCheck/DependencyCheck.py | 120 +++++++++
.../DependencyCheck_plug_in.yaml | 13 +
.pytool/Plugin/DependencyCheck/Readme.md | 31 +++
.../DscCompleteCheck/DscCompleteCheck.py | 118 ++++++++
.../DscCompleteCheck_plug_in.yaml | 12 +
.pytool/Plugin/DscCompleteCheck/Readme.md | 27 ++
.pytool/Plugin/GuidCheck/GuidCheck.py | 251 ++++++++++++++++++
.../Plugin/GuidCheck/GuidCheck_plug_in.yaml | 11 +
.pytool/Plugin/GuidCheck/Readme.md | 80 ++++++
.../LibraryClassCheck/LibraryClassCheck.py | 153 +++++++++++
.../LibraryClassCheck_plug_in.yaml | 11 +
.pytool/Plugin/LibraryClassCheck/Readme.md | 25 ++
.pytool/Plugin/SpellCheck/Readme.md | 127 +++++++++
.pytool/Plugin/SpellCheck/SpellCheck.py | 216 +++++++++++++++
.../Plugin/SpellCheck/SpellCheck_plug_in.yaml | 11 +
.pytool/Plugin/SpellCheck/cspell.base.yaml | 165 ++++++++++++
22 files changed, 1648 insertions(+)
create mode 100644 .pytool/Plugin/CharEncodingCheck/CharEncodingCheck.py
create mode 100644 .pytool/Plugin/CharEncodingCheck/CharEncodingCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/CharEncodingCheck/Readme.md
create mode 100644 .pytool/Plugin/CompilerPlugin/CompilerPlugin.py
create mode 100644 .pytool/Plugin/CompilerPlugin/Compiler_plug_in.yaml
create mode 100644 .pytool/Plugin/CompilerPlugin/Readme.md
create mode 100644 .pytool/Plugin/DependencyCheck/DependencyCheck.py
create mode 100644 .pytool/Plugin/DependencyCheck/DependencyCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/DependencyCheck/Readme.md
create mode 100644 .pytool/Plugin/DscCompleteCheck/DscCompleteCheck.py
create mode 100644 .pytool/Plugin/DscCompleteCheck/DscCompleteCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/DscCompleteCheck/Readme.md
create mode 100644 .pytool/Plugin/GuidCheck/GuidCheck.py
create mode 100644 .pytool/Plugin/GuidCheck/GuidCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/GuidCheck/Readme.md
create mode 100644 .pytool/Plugin/LibraryClassCheck/LibraryClassCheck.py
create mode 100644 .pytool/Plugin/LibraryClassCheck/LibraryClassCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/LibraryClassCheck/Readme.md
create mode 100644 .pytool/Plugin/SpellCheck/Readme.md
create mode 100644 .pytool/Plugin/SpellCheck/SpellCheck.py
create mode 100644 .pytool/Plugin/SpellCheck/SpellCheck_plug_in.yaml
create mode 100644 .pytool/Plugin/SpellCheck/cspell.base.yaml
diff --git a/.pytool/Plugin/CharEncodingCheck/CharEncodingCheck.py b/.pytool/Plugin/CharEncodingCheck/CharEncodingCheck.py
new file mode 100644
index 0000000000..54a2424875
--- /dev/null
+++ b/.pytool/Plugin/CharEncodingCheck/CharEncodingCheck.py
@@ -0,0 +1,118 @@
+# @file CharEncodingCheck.py
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+
+import os
+import logging
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toolext.environment.var_dict import VarDict
+
+##
+# map
+##
+EcodingMap = {
+ ".md": 'utf-8',
+ ".dsc": 'utf-8',
+ ".dec": 'utf-8',
+ ".c": 'utf-8',
+ ".h": 'utf-8',
+ ".asm": 'utf-8',
+ ".masm": 'utf-8',
+ ".nasm": 'utf-8',
+ ".s": 'utf-8',
+ ".inf": 'utf-8',
+ ".asl": 'utf-8',
+ ".uni": 'utf-8',
+ ".py": 'utf-8'
+}
+
+
+class CharEncodingCheck(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that scans each file in the code tree and confirms the encoding is correct.
+
+ Configuration options:
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ }
+ """
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+ testclassname: a descriptive string for the testcase can include whitespace
+ classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ """
+ return ("Check for valid file encoding for " + packagename, packagename + ".CharEncodingCheck")
+
+ ##
+ # External function of plugin. This function is used to perform the task of the ci_build_plugin Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - EnvConfig Object
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ overall_status = 0
+ files_tested = 0
+
+ abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(packagename)
+
+ if abs_pkg_path is None:
+ tc.SetSkipped()
+ tc.LogStdError("No Package folder {0}".format(abs_pkg_path))
+ return 0
+
+ for (ext, enc) in EcodingMap.items():
+ files = self.WalkDirectoryForExtension([ext], abs_pkg_path)
+ files = [Edk2pathObj.GetEdk2RelativePathFromAbsolutePath(x) for x in files] # make edk2relative path so can process ignores
+
+ if "IgnoreFiles" in pkgconfig:
+ for a in pkgconfig["IgnoreFiles"]:
+ a = a.replace(os.sep, "/")
+ try:
+ tc.LogStdOut("Ignoring File {0}".format(a))
+ files.remove(a)
+ except:
+ tc.LogStdError("CharEncodingCheck.IgnoreInf -> {0} not found in filesystem. Invalid ignore file".format(a))
+ logging.info("CharEncodingCheck.IgnoreInf -> {0} not found in filesystem. Invalid ignore file".format(a))
+
+ files = [Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(x) for x in files]
+ for a in files:
+ files_tested += 1
+ if(self.TestEncodingOk(a, enc)):
+ logging.debug("File {0} Passed Encoding Check {1}".format(a, enc))
+ else:
+ tc.LogStdError("Encoding Failure in {0}. Not {1}".format(a, enc))
+ overall_status += 1
+
+ tc.LogStdOut("Tested Encoding on {0} files".format(files_tested))
+ if overall_status is not 0:
+ tc.SetFailed("CharEncoding {0} Failed. Errors {1}".format(packagename, overall_status), "CHAR_ENCODING_CHECK_FAILED")
+ else:
+ tc.SetSuccess()
+ return overall_status
+
+ def TestEncodingOk(self, apath, encodingValue):
+ try:
+ with open(apath, "rb") as fobj:
+ fobj.read().decode(encodingValue)
+ except Exception as exp:
+ logging.error("Encoding failure: file: {0} type: {1}".format(apath, encodingValue))
+ logging.debug("EXCEPTION: while processing {1} - {0}".format(exp, apath))
+ return False
+
+ return True
diff --git a/.pytool/Plugin/CharEncodingCheck/CharEncodingCheck_plug_in.yaml b/.pytool/Plugin/CharEncodingCheck/CharEncodingCheck_plug_in.yaml
new file mode 100644
index 0000000000..f0d4c70c63
--- /dev/null
+++ b/.pytool/Plugin/CharEncodingCheck/CharEncodingCheck_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# CiBuildPlugin used to check char encoding
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Char Encoding Check Test",
+ "module": "CharEncodingCheck"
+}
diff --git a/.pytool/Plugin/CharEncodingCheck/Readme.md b/.pytool/Plugin/CharEncodingCheck/Readme.md
new file mode 100644
index 0000000000..47486e5c3c
--- /dev/null
+++ b/.pytool/Plugin/CharEncodingCheck/Readme.md
@@ -0,0 +1,18 @@
+# Character Encoding Check Plugin
+
+This CiBuildPlugin scans all the files in a package to make sure each file is
+correctly encoded and all characters can be read. Improper encoding causes
+tools to fail in some situations especially in different locals.
+
+## Configuration
+
+The plugin can be configured to ignore certain files.
+
+``` yaml
+"CharEncodingCheck": {
+ "IgnoreFiles": []
+}
+```
+### IgnoreFiles
+
+OPTIONAL List of file to ignore.
diff --git a/.pytool/Plugin/CompilerPlugin/CompilerPlugin.py b/.pytool/Plugin/CompilerPlugin/CompilerPlugin.py
new file mode 100644
index 0000000000..0a357309a4
--- /dev/null
+++ b/.pytool/Plugin/CompilerPlugin/CompilerPlugin.py
@@ -0,0 +1,102 @@
+# @file HostUnitTestCompiler_plugin.py
+##
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+import logging
+import os
+import re
+from edk2toollib.uefi.edk2.parsers.dsc_parser import DscParser
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toolext.environment.uefi_build import UefiBuilder
+from edk2toolext import edk2_logging
+from edk2toolext.environment.var_dict import VarDict
+
+
+class CompilerPlugin(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that compiles the package dsc
+ from the package being tested.
+
+ Configuration options:
+ "CompilerPlugin": {
+ "DscPath": "<path to dsc from root of pkg>"
+ }
+ """
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ """
+ target = environment.GetValue("TARGET")
+ return ("Compile " + packagename + " " + target, packagename + ".Compiler." + target)
+
+ def RunsOnTargetList(self):
+ return ["DEBUG", "RELEASE"]
+
+ ##
+ # External function of plugin. This function is used to perform the task of the MuBuild Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - EnvConfig Object
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ self._env = environment
+
+ # Parse the config for required DscPath element
+ if "DscPath" not in pkgconfig:
+ tc.SetSkipped()
+ tc.LogStdError("DscPath not found in config file. Nothing to compile.")
+ return -1
+
+ AP = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(packagename)
+
+ APDSC = os.path.join(AP, pkgconfig["DscPath"].strip())
+ AP_Path = Edk2pathObj.GetEdk2RelativePathFromAbsolutePath(APDSC)
+ if AP is None or AP_Path is None or not os.path.isfile(APDSC):
+ tc.SetSkipped()
+ tc.LogStdError("Package Dsc not found.")
+ return -1
+
+ logging.info("Building {0}".format(AP_Path))
+ self._env.SetValue("ACTIVE_PLATFORM", AP_Path, "Set in Compiler Plugin")
+
+ # Parse DSC to check for SUPPORTED_ARCHITECTURES
+ dp = DscParser()
+ dp.SetBaseAbsPath(Edk2pathObj.WorkspacePath)
+ dp.SetPackagePaths(Edk2pathObj.PackagePathList)
+ dp.ParseFile(AP_Path)
+ if "SUPPORTED_ARCHITECTURES" in dp.LocalVars:
+ SUPPORTED_ARCHITECTURES = dp.LocalVars["SUPPORTED_ARCHITECTURES"].split('|')
+ TARGET_ARCHITECTURES = environment.GetValue("TARGET_ARCH").split(' ')
+
+ # Skip if there is no intersection between SUPPORTED_ARCHITECTURES and TARGET_ARCHITECTURES
+ if len(set(SUPPORTED_ARCHITECTURES) & set(TARGET_ARCHITECTURES)) == 0:
+ tc.SetSkipped()
+ tc.LogStdError("No supported architecutres to build")
+ return -1
+
+ uefiBuilder = UefiBuilder()
+ # do all the steps
+ # WorkSpace, PackagesPath, PInHelper, PInManager
+ ret = uefiBuilder.Go(Edk2pathObj.WorkspacePath, os.pathsep.join(Edk2pathObj.PackagePathList), PLMHelper, PLM)
+ if ret != 0: # failure:
+ tc.SetFailed("Compile failed for {0}".format(packagename), "Compile_FAILED")
+ tc.LogStdError("{0} Compile failed with error code {1} ".format(AP_Path, ret))
+ return 1
+
+ else:
+ tc.SetSuccess()
+ return 0
diff --git a/.pytool/Plugin/CompilerPlugin/Compiler_plug_in.yaml b/.pytool/Plugin/CompilerPlugin/Compiler_plug_in.yaml
new file mode 100644
index 0000000000..3e5fb0b1df
--- /dev/null
+++ b/.pytool/Plugin/CompilerPlugin/Compiler_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# CiBuildPlugin used to compile each package
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Compiler Plugin",
+ "module": "CompilerPlugin"
+}
diff --git a/.pytool/Plugin/CompilerPlugin/Readme.md b/.pytool/Plugin/CompilerPlugin/Readme.md
new file mode 100644
index 0000000000..44e6a2c8d9
--- /dev/null
+++ b/.pytool/Plugin/CompilerPlugin/Readme.md
@@ -0,0 +1,17 @@
+# Compiler Plugin
+
+This CiBuildPlugin compiles the package DSC from the package being tested.
+
+## Configuration
+
+The package relative path of the DSC file to build.
+
+``` yaml
+"CompilerPlugin": {
+ "DscPath": "<path to dsc from root of pkg>"
+}
+```
+
+### DscPath
+
+Package relative path to the DSC file to build.
diff --git a/.pytool/Plugin/DependencyCheck/DependencyCheck.py b/.pytool/Plugin/DependencyCheck/DependencyCheck.py
new file mode 100644
index 0000000000..497914cf3a
--- /dev/null
+++ b/.pytool/Plugin/DependencyCheck/DependencyCheck.py
@@ -0,0 +1,120 @@
+# @file dependency_check.py
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+import logging
+import os
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser
+from edk2toolext.environment.var_dict import VarDict
+
+
+class DependencyCheck(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that finds all modules (inf files) in a package and reviews the packages used
+ to confirm they are acceptable. This is to help enforce layering and identify improper
+ dependencies between packages.
+
+ Configuration options:
+ "DependencyCheck": {
+ "AcceptableDependencies": [], # Package dec files that are allowed in all INFs. Example: MdePkg/MdePkg.dec
+ "AcceptableDependencies-<MODULE_TYPE>": [], # OPTIONAL Package dependencies for INFs that are HOST_APPLICATION
+ "AcceptableDependencies-HOST_APPLICATION": [], # EXAMPLE Package dependencies for INFs that are HOST_APPLICATION
+ "IgnoreInf": [] # Ignore INF if found in filesystem
+ }
+ """
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ testclassname: a descriptive string for the testcase can include whitespace
+ classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+ """
+ return ("Test Package Dependencies for modules in " + packagename, packagename + ".DependencyCheck")
+
+ ##
+ # External function of plugin. This function is used to perform the task of the MuBuild Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - EnvConfig Object
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ overall_status = 0
+
+ # Get current platform
+ abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(packagename)
+
+ # Get INF Files
+ INFFiles = self.WalkDirectoryForExtension([".inf"], abs_pkg_path)
+ INFFiles = [Edk2pathObj.GetEdk2RelativePathFromAbsolutePath(x) for x in INFFiles] # make edk2relative path so can compare with Ignore List
+
+ # Remove ignored INFs
+ if "IgnoreInf" in pkgconfig:
+ for a in pkgconfig["IgnoreInf"]:
+ a = a.replace(os.sep, "/") ## convert path sep in case ignore list is bad. Can't change case
+ try:
+ INFFiles.remove(a)
+ tc.LogStdOut("IgnoreInf {0}".format(a))
+ except:
+ logging.info("DependencyConfig.IgnoreInf -> {0} not found in filesystem. Invalid ignore file".format(a))
+ tc.LogStdError("DependencyConfig.IgnoreInf -> {0} not found in filesystem. Invalid ignore file".format(a))
+
+
+ # Get the AccpetableDependencies list
+ if "AcceptableDependencies" not in pkgconfig:
+ logging.info("DependencyCheck Skipped. No Acceptable Dependencies defined.")
+ tc.LogStdOut("DependencyCheck Skipped. No Acceptable Dependencies defined.")
+ tc.SetSkipped()
+ return -1
+
+ # Log dependencies
+ for k in pkgconfig.keys():
+ if k.startswith("AcceptableDependencies"):
+ pkgstring = "\n".join(pkgconfig[k])
+ if ("-" in k):
+ _, _, mod_type = k.partition("-")
+ tc.LogStdOut(f"Additional dependencies for MODULE_TYPE {mod_type}:\n {pkgstring}")
+ else:
+ tc.LogStdOut(f"Acceptable Dependencies:\n {pkgstring}")
+
+ # For each INF file
+ for file in INFFiles:
+ ip = InfParser()
+ logging.debug("Parsing " + file)
+ ip.SetBaseAbsPath(Edk2pathObj.WorkspacePath).SetPackagePaths(Edk2pathObj.PackagePathList).ParseFile(file)
+
+ if("MODULE_TYPE" not in ip.Dict):
+ tc.LogStdOut("Ignoring INF. Missing key for MODULE_TYPE {0}".format(file))
+ continue
+
+ mod_type = ip.Dict["MODULE_TYPE"].upper()
+ for p in ip.PackagesUsed:
+ if p not in pkgconfig["AcceptableDependencies"]:
+ # If not in the main acceptable dependencies list then check module specific
+ mod_specific_key = "AcceptableDependencies-" + mod_type
+ if mod_specific_key in pkgconfig and p in pkgconfig[mod_specific_key]:
+ continue
+
+ logging.error("Dependency Check: Invalid Dependency INF: {0} depends on pkg {1}".format(file, p))
+ tc.LogStdError("Dependency Check: Invalid Dependency INF: {0} depends on pkg {1}".format(file, p))
+ overall_status += 1
+
+ # If XML object exists, add results
+ if overall_status is not 0:
+ tc.SetFailed("Failed with {0} errors".format(overall_status), "DEPENDENCYCHECK_FAILED")
+ else:
+ tc.SetSuccess()
+ return overall_status
diff --git a/.pytool/Plugin/DependencyCheck/DependencyCheck_plug_in.yaml b/.pytool/Plugin/DependencyCheck/DependencyCheck_plug_in.yaml
new file mode 100644
index 0000000000..121c6889cf
--- /dev/null
+++ b/.pytool/Plugin/DependencyCheck/DependencyCheck_plug_in.yaml
@@ -0,0 +1,13 @@
+## @file
+# CiBuildPlugin used to check all infs within a package
+# to confirm the packagesdependency are on the configured list of acceptable
+# dependencies.
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Dependency Check Test",
+ "module": "DependencyCheck"
+}
diff --git a/.pytool/Plugin/DependencyCheck/Readme.md b/.pytool/Plugin/DependencyCheck/Readme.md
new file mode 100644
index 0000000000..62612f6965
--- /dev/null
+++ b/.pytool/Plugin/DependencyCheck/Readme.md
@@ -0,0 +1,31 @@
+# Depdendency Check Plugin
+
+A CiBuildPlugin that finds all modules (inf files) in a package and reviews the
+packages used to confirm they are acceptable. This is to help enforce layering
+and identify improper dependencies between packages.
+
+## Configuration
+
+The plugin must be configured with the acceptabe package dependencies for the
+package.
+
+``` yaml
+"DependencyCheck": {
+ "AcceptableDependencies": [],
+ "AcceptableDependencies-<MODULE_TYPE>": [],
+ "IgnoreInf": []
+}
+```
+
+### AcceptableDependencies
+
+Package dec files that are allowed in all INFs. Example: MdePkg/MdePkg.dec
+
+### AcceptableDependencies-<MODULE_TYPE>
+
+OPTIONAL Package dependencies for INFs that have module type <MODULE_TYPE>.
+Example: AcceptableDependencies-HOST_APPLICATION.
+
+### IgnoreInf
+
+OPTIONAL list of INFs to ignore for this dependency check.
diff --git a/.pytool/Plugin/DscCompleteCheck/DscCompleteCheck.py b/.pytool/Plugin/DscCompleteCheck/DscCompleteCheck.py
new file mode 100644
index 0000000000..dcd8946ca6
--- /dev/null
+++ b/.pytool/Plugin/DscCompleteCheck/DscCompleteCheck.py
@@ -0,0 +1,118 @@
+# @file DscCompleteCheck.py
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import logging
+import os
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toollib.uefi.edk2.parsers.dsc_parser import DscParser
+from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser
+from edk2toolext.environment.var_dict import VarDict
+
+
+class DscCompleteCheck(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that scans the package dsc file and confirms all modules (inf files) are
+ listed in the components sections.
+
+ Configuration options:
+ "DscCompleteCheck": {
+ "DscPath": "<path to dsc from root of pkg>"
+ "IgnoreInf": [] # Ignore INF if found in filesystem by not dsc
+ }
+ """
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ testclassname: a descriptive string for the testcase can include whitespace
+ classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+ """
+ return ("Check the " + packagename + " DSC for a being complete", packagename + ".DscCompleteCheck")
+
+ ##
+ # External function of plugin. This function is used to perform the task of the MuBuild Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - VarDict containing the shell environment Build Vars
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ overall_status = 0
+
+ # Parse the config for required DscPath element
+ if "DscPath" not in pkgconfig:
+ tc.SetSkipped()
+ tc.LogStdError("DscPath not found in config file. Nothing to check.")
+ return -1
+
+ abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(packagename)
+ abs_dsc_path = os.path.join(abs_pkg_path, pkgconfig["DscPath"].strip())
+ wsr_dsc_path = Edk2pathObj.GetEdk2RelativePathFromAbsolutePath(abs_dsc_path)
+
+ if abs_dsc_path is None or wsr_dsc_path is "" or not os.path.isfile(abs_dsc_path):
+ tc.SetSkipped()
+ tc.LogStdError("Package Dsc not found")
+ return 0
+
+ # Get INF Files
+ INFFiles = self.WalkDirectoryForExtension([".inf"], abs_pkg_path)
+ INFFiles = [Edk2pathObj.GetEdk2RelativePathFromAbsolutePath(x) for x in INFFiles] # make edk2relative path so can compare with DSC
+
+ # remove ignores
+
+ if "IgnoreInf" in pkgconfig:
+ for a in pkgconfig["IgnoreInf"]:
+ a = a.replace(os.sep, "/")
+ try:
+ tc.LogStdOut("Ignoring INF {0}".format(a))
+ INFFiles.remove(a)
+ except:
+ tc.LogStdError("DscCompleteCheck.IgnoreInf -> {0} not found in filesystem. Invalid ignore file".format(a))
+ logging.info("DscCompleteCheck.IgnoreInf -> {0} not found in filesystem. Invalid ignore file".format(a))
+
+ # DSC Parser
+ dp = DscParser()
+ dp.SetBaseAbsPath(Edk2pathObj.WorkspacePath)
+ dp.SetPackagePaths(Edk2pathObj.PackagePathList)
+ dp.SetInputVars(environment.GetAllBuildKeyValues())
+ dp.ParseFile(wsr_dsc_path)
+
+ # Check if INF in component section
+ for INF in INFFiles:
+ if not any(INF.strip() in x for x in dp.ThreeMods) and \
+ not any(INF.strip() in x for x in dp.SixMods) and \
+ not any(INF.strip() in x for x in dp.OtherMods):
+
+ infp = InfParser().SetBaseAbsPath(Edk2pathObj.WorkspacePath)
+ infp.SetPackagePaths(Edk2pathObj.PackagePathList)
+ infp.ParseFile(INF)
+ if("MODULE_TYPE" not in infp.Dict):
+ tc.LogStdOut("Ignoring INF. Missing key for MODULE_TYPE {0}".format(INF))
+ continue
+
+ if(infp.Dict["MODULE_TYPE"] == "HOST_APPLICATION"):
+ tc.LogStdOut("Ignoring INF. Module type is HOST_APPLICATION {0}".format(INF))
+ continue
+
+ logging.critical(INF + " not in " + wsr_dsc_path)
+ tc.LogStdError("{0} not in {1}".format(INF, wsr_dsc_path))
+ overall_status = overall_status + 1
+
+ # If XML object exists, add result
+ if overall_status is not 0:
+ tc.SetFailed("DscCompleteCheck {0} Failed. Errors {1}".format(wsr_dsc_path, overall_status), "CHECK_FAILED")
+ else:
+ tc.SetSuccess()
+ return overall_status
diff --git a/.pytool/Plugin/DscCompleteCheck/DscCompleteCheck_plug_in.yaml b/.pytool/Plugin/DscCompleteCheck/DscCompleteCheck_plug_in.yaml
new file mode 100644
index 0000000000..9e215d8bc6
--- /dev/null
+++ b/.pytool/Plugin/DscCompleteCheck/DscCompleteCheck_plug_in.yaml
@@ -0,0 +1,12 @@
+## @file
+# CiBuildPlugin used to confirm all INFs are listed in
+# the components section of package dsc
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Dsc Complete Check Test",
+ "module": "DscCompleteCheck"
+}
diff --git a/.pytool/Plugin/DscCompleteCheck/Readme.md b/.pytool/Plugin/DscCompleteCheck/Readme.md
new file mode 100644
index 0000000000..eefbb9894d
--- /dev/null
+++ b/.pytool/Plugin/DscCompleteCheck/Readme.md
@@ -0,0 +1,27 @@
+# Dsc Complete Check Plugin
+
+This CiBuildPlugin scans all INF files from a package and confirms they are
+listed in the package level DSC file. The test considers it an error if any INF
+does not appear in the `Components` section of the package-level DSC (indicating
+that it would not be built if the package were built). This is critical because
+much of the CI infrastructure assumes that all modules will be listed in the DSC
+and compiled.
+
+## Configuration
+
+The plugin has a few configuration options to support the UEFI codebase.
+
+``` yaml
+"DscCompleteCheck": {
+ "DscPath": "", # Path to dsc from root of package
+ "IgnoreInf": [] # Ignore INF if found in filesystem by not dsc
+ }
+```
+
+### DscPath
+
+Path to DSC to consider platform dsc
+
+### IgnoreInf
+
+Ignore error if Inf file is not listed in DSC file
diff --git a/.pytool/Plugin/GuidCheck/GuidCheck.py b/.pytool/Plugin/GuidCheck/GuidCheck.py
new file mode 100644
index 0000000000..467e17f3e8
--- /dev/null
+++ b/.pytool/Plugin/GuidCheck/GuidCheck.py
@@ -0,0 +1,251 @@
+# @file GuidCheck.py
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import logging
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toollib.uefi.edk2.guid_list import GuidList
+from edk2toolext.environment.var_dict import VarDict
+
+
+class GuidCheck(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that scans the code tree and looks for duplicate guids
+ from the package being tested.
+
+ Configuration options:
+ "GuidCheck": {
+ "IgnoreGuidName": [], # provide in format guidname=guidvalue or just guidname
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [] # Provide in format guidname=guidname=guidname...
+ }
+ """
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ testclassname: a descriptive string for the testcase can include whitespace
+ classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+ """
+ return ("Confirm GUIDs are unique in " + packagename, packagename + ".GuidCheck")
+
+ def _FindConflictingGuidValues(self, guidlist: list) -> list:
+ """ Find all duplicate guids by guid value and report them as errors
+ """
+ # Sort the list by guid
+ guidsorted = sorted(
+ guidlist, key=lambda x: x.guid.upper(), reverse=True)
+
+ previous = None # Store previous entry for comparison
+ error = None
+ errors = []
+ for index in range(len(guidsorted)):
+ i = guidsorted[index]
+ if(previous is not None):
+ if i.guid == previous.guid: # Error
+ if(error is None):
+ # Catch errors with more than 1 conflict
+ error = ErrorEntry("guid")
+ error.entries.append(previous)
+ errors.append(error)
+ error.entries.append(i)
+ else:
+ # no match. clear error
+ error = None
+ previous = i
+ return errors
+
+ def _FindConflictingGuidNames(self, guidlist: list) -> list:
+ """ Find all duplicate guids by name and if they are not all
+ from inf files report them as errors. It is ok to have
+ BASE_NAME duplication.
+
+ Is this useful? It would catch two same named guids in dec file
+ that resolve to different values.
+ """
+ # Sort the list by guid
+ namesorted = sorted(guidlist, key=lambda x: x.name.upper())
+
+ previous = None # Store previous entry for comparison
+ error = None
+ errors = []
+ for index in range(len(namesorted)):
+ i = namesorted[index]
+ if(previous is not None):
+ # If name matches
+ if i.name == previous.name:
+ if(error is None):
+ # Catch errors with more than 1 conflict
+ error = ErrorEntry("name")
+ error.entries.append(previous)
+ errors.append(error)
+ error.entries.append(i)
+ else:
+ # no match. clear error
+ error = None
+ previous = i
+
+ # Loop thru and remove any errors where all files are infs as it is ok if
+ # they have the same inf base name.
+ for e in errors[:]:
+ if len( [en for en in e.entries if not en.absfilepath.lower().endswith(".inf")]) == 0:
+ errors.remove(e)
+
+ return errors
+
+ ##
+ # External function of plugin. This function is used to perform the task of the MuBuild Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - EnvConfig Object
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ Errors = []
+
+ abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(
+ packagename)
+
+ if abs_pkg_path is None:
+ tc.SetSkipped()
+ tc.LogStdError("No package {0}".format(packagename))
+ return -1
+
+ All_Ignores = ["/Build", "/Conf"]
+ # Parse the config for other ignores
+ if "IgnoreFoldersAndFiles" in pkgconfig:
+ All_Ignores.extend(pkgconfig["IgnoreFoldersAndFiles"])
+
+ # Parse the workspace for all GUIDs
+ gs = GuidList.guidlist_from_filesystem(
+ Edk2pathObj.WorkspacePath, ignore_lines=All_Ignores)
+
+ # Remove ignored guidvalue
+ if "IgnoreGuidValue" in pkgconfig:
+ for a in pkgconfig["IgnoreGuidValue"]:
+ try:
+ tc.LogStdOut("Ignoring Guid {0}".format(a.upper()))
+ for b in gs[:]:
+ if b.guid == a.upper():
+ gs.remove(b)
+ except:
+ tc.LogStdError("GuidCheck.IgnoreGuid -> {0} not found. Invalid ignore guid".format(a.upper()))
+ logging.info("GuidCheck.IgnoreGuid -> {0} not found. Invalid ignore guid".format(a.upper()))
+
+ # Remove ignored guidname
+ if "IgnoreGuidName" in pkgconfig:
+ for a in pkgconfig["IgnoreGuidName"]:
+ entry = a.split("=")
+ if(len(entry) > 2):
+ tc.LogStdError("GuidCheck.IgnoreGuidName -> {0} Invalid Format.".format(a))
+ logging.info("GuidCheck.IgnoreGuidName -> {0} Invalid Format.".format(a))
+ continue
+ try:
+ tc.LogStdOut("Ignoring Guid {0}".format(a))
+ for b in gs[:]:
+ if b.name == entry[0]:
+ if(len(entry) == 1):
+ gs.remove(b)
+ elif(len(entry) == 2 and b.guid.upper() == entry[1].upper()):
+ gs.remove(b)
+ else:
+ c.LogStdError("GuidCheck.IgnoreGuidName -> {0} incomplete match. Invalid ignore guid".format(a))
+
+ except:
+ tc.LogStdError("GuidCheck.IgnoreGuidName -> {0} not found. Invalid ignore name".format(a))
+ logging.info("GuidCheck.IgnoreGuidName -> {0} not found. Invalid ignore name".format(a))
+
+ # Find conflicting Guid Values
+ Errors.extend(self._FindConflictingGuidValues(gs))
+
+ # Check if there are expected duplicates and remove it from the error list
+ if "IgnoreDuplicates" in pkgconfig:
+ for a in pkgconfig["IgnoreDuplicates"]:
+ names = a.split("=")
+ if len(names) < 2:
+ tc.LogStdError("GuidCheck.IgnoreDuplicates -> {0} invalid format".format(a))
+ logging.info("GuidCheck.IgnoreDuplicates -> {0} invalid format".format(a))
+ continue
+
+ for b in Errors[:]:
+ if b.type != "guid":
+ continue
+ ## Make a list of the names that are not in the names list. If there
+ ## are any in the list then this error should not be ignored.
+ t = [x for x in b.entries if x.name not in names]
+ if(len(t) == len(b.entries)):
+ ## did not apply to any entry
+ continue
+ elif(len(t) == 0):
+ ## full match - ignore duplicate
+ tc.LogStdOut("GuidCheck.IgnoreDuplicates -> {0}".format(a))
+ Errors.remove(b)
+ elif(len(t) < len(b.entries)):
+ ## partial match
+ tc.LogStdOut("GuidCheck.IgnoreDuplicates -> {0} incomplete match".format(a))
+ logging.info("GuidCheck.IgnoreDuplicates -> {0} incomplete match".format(a))
+ else:
+ tc.LogStdOut("GuidCheck.IgnoreDuplicates -> {0} unknown error.".format(a))
+ logging.info("GuidCheck.IgnoreDuplicates -> {0} unknown error".format(a))
+
+
+
+ # Find conflicting Guid Names
+ Errors.extend(self._FindConflictingGuidNames(gs))
+
+ # Log errors for anything within the package under test
+ for er in Errors[:]:
+ InMyPackage = False
+ for a in er.entries:
+ if abs_pkg_path in a.absfilepath:
+ InMyPackage = True
+ break
+ if(not InMyPackage):
+ Errors.remove(er)
+ else:
+ logging.error(str(er))
+ tc.LogStdError(str(er))
+
+ # add result to test case
+ overall_status = len(Errors)
+ if overall_status is not 0:
+ tc.SetFailed("GuidCheck {0} Failed. Errors {1}".format(
+ packagename, overall_status), "CHECK_FAILED")
+ else:
+ tc.SetSuccess()
+ return overall_status
+
+
+class ErrorEntry():
+ """ Custom/private class for reporting errors in the GuidList
+ """
+
+ def __init__(self, errortype):
+ self.type = errortype # 'guid' or 'name' depending on error type
+ self.entries = [] # GuidListEntry that are in error condition
+
+ def __str__(self):
+ a = f"Error Duplicate {self.type}: "
+ if(self.type == "guid"):
+ a += f" {self.entries[0].guid}"
+ elif(self.type == "name"):
+ a += f" {self.entries[0].name}"
+
+ a += f" ({len(self.entries)})\n"
+
+ for e in self.entries:
+ a += "\t" + str(e) + "\n"
+ return a
diff --git a/.pytool/Plugin/GuidCheck/GuidCheck_plug_in.yaml b/.pytool/Plugin/GuidCheck/GuidCheck_plug_in.yaml
new file mode 100644
index 0000000000..d76a58a1b9
--- /dev/null
+++ b/.pytool/Plugin/GuidCheck/GuidCheck_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# CiBuildPlugin used to check guid uniqueness
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Guid Check Test",
+ "module": "GuidCheck"
+}
diff --git a/.pytool/Plugin/GuidCheck/Readme.md b/.pytool/Plugin/GuidCheck/Readme.md
new file mode 100644
index 0000000000..4e2c353cc4
--- /dev/null
+++ b/.pytool/Plugin/GuidCheck/Readme.md
@@ -0,0 +1,80 @@
+# Guid Check Plugin
+
+This CiBuildPlugin scans all the files in a code tree to find all the GUID
+definitions. After collection it will then look for duplication in the package
+under test. Uniqueness of all GUIDs are critical within the UEFI environment.
+Duplication can cause numerous issues including locating the wrong data
+structure, calling the wrong function, or decoding the wrong data members.
+
+Currently Scanned:
+
+* INF files are scanned for there Module guid
+* DEC files are scanned for all of their Protocols, PPIs, and Guids as well as
+ the one package GUID.
+
+Any GUID value being equal to two names or even just defined in two files is
+considered an error unless in the ignore list.
+
+Any GUID name that is found more than once is an error unless all occurrences
+are Module GUIDs. Since the Module GUID is assigned to the Module name it is
+common to have numerous versions of the same module named the same.
+
+## Configuration
+
+The plugin has numerous configuration options to support the UEFI codebase.
+
+``` yaml
+"GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": []
+ }
+```
+
+### IgnoreGuidName
+
+This list allows strings in two formats.
+
+* _GuidName_
+ * This will remove any entry with this GuidName from the list of GUIDs
+ therefore ignoring any error associated with this name.
+* _GuidName=GuidValue_
+ * This will also ignore the GUID by name but only if the value equals the
+ GuidValue.
+ * GuidValue should be in registry format.
+ * This is the suggested format to use as it will limit the ignore to only the
+ defined case.
+
+### IgnoreGuidValue
+
+This list allows strings in guid registry format _GuidValue_.
+
+* This will remove any entry with this GuidValue from the list of GUIDs
+ therefore ignoring any error associated with this value.
+* GuidValue must be in registry format xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+
+### IgnoreFoldersAndFiles
+
+This supports .gitignore file and folder matching strings including wildcards
+
+* Any folder or file ignored will not be parsed and therefore any GUID defined
+ will be ignored.
+* The plugin will always ignores the following ["/Build", "/Conf"]
+
+### IgnoreDuplicates
+
+This supports strings in the format of _GuidName_=_GuidName_=_GuidName_
+
+* For the error with the GuidNames to be ignored the list must match completely
+ with what is found during the code scan.
+ * For example if there are two GUIDs that are by design equal within the code
+ tree then it should be _GuidName_=_GuidName_
+ * If instead there are three GUIDs then it must be
+ _GuidName_=_GuidName_=_GuidName_
+* This is the best ignore list to use because it is the most strict and will
+ catch new problems when new conflicts are introduced.
+* There are numerous places in the UEFI specification in which two GUID names
+ are assigned the same value. These names should be set in this ignore list so
+ that they don't cause an error but any additional duplication would still be
+ caught.
diff --git a/.pytool/Plugin/LibraryClassCheck/LibraryClassCheck.py b/.pytool/Plugin/LibraryClassCheck/LibraryClassCheck.py
new file mode 100644
index 0000000000..33745dff11
--- /dev/null
+++ b/.pytool/Plugin/LibraryClassCheck/LibraryClassCheck.py
@@ -0,0 +1,153 @@
+# @file LibraryClassCheck.py
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import logging
+import os
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toollib.uefi.edk2.parsers.dec_parser import DecParser
+from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser
+from edk2toolext.environment.var_dict import VarDict
+
+
+class LibraryClassCheck(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that scans the code tree and library classes for undeclared
+ files
+
+ Configuration options:
+ "LibraryClassCheck": {
+ IgnoreHeaderFile: [], # Ignore a file found on disk
+ IgnoreLibraryClass: [] # Ignore a declaration found in dec file
+ }
+ """
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+ testclassname: a descriptive string for the testcase can include whitespace
+ classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ """
+ return ("Check library class declarations in " + packagename, packagename + ".LibraryClassCheck")
+
+ def __GetPkgDec(self, rootpath):
+ try:
+ allEntries = os.listdir(rootpath)
+ for entry in allEntries:
+ if entry.lower().endswith(".dec"):
+ return(os.path.join(rootpath, entry))
+ except Exception:
+ logging.error("Unable to find DEC for package:{0}".format(rootpath))
+
+ return None
+
+ ##
+ # External function of plugin. This function is used to perform the task of the MuBuild Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - EnvConfig Object
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ overall_status = 0
+ LibraryClassIgnore = []
+
+ abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(packagename)
+ abs_dec_path = self.__GetPkgDec(abs_pkg_path)
+ wsr_dec_path = Edk2pathObj.GetEdk2RelativePathFromAbsolutePath(abs_dec_path)
+
+ if abs_dec_path is None or wsr_dec_path is "" or not os.path.isfile(abs_dec_path):
+ tc.SetSkipped()
+ tc.LogStdError("No DEC file {0} in package {1}".format(abs_dec_path, abs_pkg_path))
+ return -1
+
+ # Get all include folders
+ dec = DecParser()
+ dec.SetBaseAbsPath(Edk2pathObj.WorkspacePath).SetPackagePaths(Edk2pathObj.PackagePathList)
+ dec.ParseFile(wsr_dec_path)
+
+ AllHeaderFiles = []
+
+ for includepath in dec.IncludePaths:
+ ## Get all header files in the library folder
+ AbsLibraryIncludePath = os.path.join(abs_pkg_path, includepath, "Library")
+ if(not os.path.isdir(AbsLibraryIncludePath)):
+ continue
+
+ hfiles = self.WalkDirectoryForExtension([".h"], AbsLibraryIncludePath)
+ hfiles = [os.path.relpath(x,abs_pkg_path) for x in hfiles] # make package root relative path
+ hfiles = [x.replace("\\", "/") for x in hfiles] # make package relative path
+
+ AllHeaderFiles.extend(hfiles)
+
+ if len(AllHeaderFiles) == 0:
+ tc.SetSkipped()
+ tc.LogStdError(f"No Library include folder in any Include path")
+ return -1
+
+ # Remove ignored paths
+ if "IgnoreHeaderFile" in pkgconfig:
+ for a in pkgconfig["IgnoreHeaderFile"]:
+ try:
+ tc.LogStdOut("Ignoring Library Header File {0}".format(a))
+ AllHeaderFiles.remove(a)
+ except:
+ tc.LogStdError("LibraryClassCheck.IgnoreHeaderFile -> {0} not found. Invalid Header File".format(a))
+ logging.info("LibraryClassCheck.IgnoreHeaderFile -> {0} not found. Invalid Header File".format(a))
+
+ if "IgnoreLibraryClass" in pkgconfig:
+ LibraryClassIgnore = pkgconfig["IgnoreLibraryClass"]
+
+
+ ## Attempt to find library classes
+ for lcd in dec.LibraryClasses:
+ ## Check for correct file path separator
+ if "\\" in lcd.path:
+ tc.LogStdError("LibraryClassCheck.DecFilePathSeparator -> {0} invalid.".format(lcd.path))
+ logging.error("LibraryClassCheck.DecFilePathSeparator -> {0} invalid.".format(lcd.path))
+ overall_status += 1
+ continue
+
+ if lcd.name in LibraryClassIgnore:
+ tc.LogStdOut("Ignoring Library Class Name {0}".format(lcd.name))
+ LibraryClassIgnore.remove(lcd.name)
+ continue
+
+ logging.debug(f"Looking for Library Class {lcd.path}")
+ try:
+ AllHeaderFiles.remove(lcd.path)
+
+ except ValueError:
+ tc.LogStdError(f"Library {lcd.name} with path {lcd.path} not found in package filesystem")
+ logging.error(f"Library {lcd.name} with path {lcd.path} not found in package filesystem")
+ overall_status += 1
+
+ ## any remaining AllHeaderFiles are not described in DEC
+ for h in AllHeaderFiles:
+ tc.LogStdError(f"Library Header File {h} not declared in package DEC {wsr_dec_path}")
+ logging.error(f"Library Header File {h} not declared in package DEC {wsr_dec_path}")
+ overall_status += 1
+
+ ## Warn about any invalid library class names in the ignore list
+ for r in LibraryClassIgnore:
+ tc.LogStdError("LibraryClassCheck.IgnoreLibraryClass -> {0} not found. Library Class not found".format(r))
+ logging.info("LibraryClassCheck.IgnoreLibraryClass -> {0} not found. Library Class not found".format(r))
+
+
+ # If XML object exists, add result
+ if overall_status is not 0:
+ tc.SetFailed("LibraryClassCheck {0} Failed. Errors {1}".format(wsr_dec_path, overall_status), "CHECK_FAILED")
+ else:
+ tc.SetSuccess()
+ return overall_status
diff --git a/.pytool/Plugin/LibraryClassCheck/LibraryClassCheck_plug_in.yaml b/.pytool/Plugin/LibraryClassCheck/LibraryClassCheck_plug_in.yaml
new file mode 100644
index 0000000000..80cb6cc118
--- /dev/null
+++ b/.pytool/Plugin/LibraryClassCheck/LibraryClassCheck_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# CiBuildPlugin used to check that all library classes are declared correctly in dec file
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Library Class Check Test",
+ "module": "LibraryClassCheck"
+}
diff --git a/.pytool/Plugin/LibraryClassCheck/Readme.md b/.pytool/Plugin/LibraryClassCheck/Readme.md
new file mode 100644
index 0000000000..7fa2b202f2
--- /dev/null
+++ b/.pytool/Plugin/LibraryClassCheck/Readme.md
@@ -0,0 +1,25 @@
+# Library Class Check Plugin
+
+This CiBuildPlugin scans at all library header files found in the `Library`
+folders in all of the package's declared include directories and ensures that
+all files have a matching LibraryClass declaration in the DEC file for the
+package. Any missing declarations will cause a failure.
+
+## Configuration
+
+The plugin has a few configuration options to support the UEFI codebase.
+
+``` yaml
+"LibraryClassCheck": {
+ IgnoreHeaderFile: [], # Ignore a file found on disk
+ IgnoreLibraryClass: [] # Ignore a declaration found in dec file
+}
+```
+
+### IgnoreHeaderFile
+
+Ignore a file found on disk
+
+### IgnoreLibraryClass
+
+Ignore a declaration found in dec file
diff --git a/.pytool/Plugin/SpellCheck/Readme.md b/.pytool/Plugin/SpellCheck/Readme.md
new file mode 100644
index 0000000000..394bb8effc
--- /dev/null
+++ b/.pytool/Plugin/SpellCheck/Readme.md
@@ -0,0 +1,127 @@
+# Spell Check Plugin
+
+This CiBuildPlugin scans all the files in a given package and checks for
+spelling errors.
+
+This plugin requires NodeJs and cspell. If the plugin doesn't find its required
+tools then it will mark the test as skipped.
+
+* NodeJS: https://nodejs.org/en/
+* cspell: https://www.npmjs.com/package/cspell
+ * Src and doc available: https://github.com/streetsidesoftware/cspell
+
+## Configuration
+
+The plugin has a few configuration options to support the UEFI codebase.
+
+``` yaml
+ "SpellCheck": {
+ "AuditOnly": False, # If True, log all errors and then mark as skipped
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+```
+
+### AuditOnly
+
+Boolean - Default is False.
+If True run the test in an Audit only mode which will log all errors but instead
+of failing the build it will set the test as skipped. This allows visibility
+into the failures without breaking the build.
+
+### IgnoreFiles
+
+This supports .gitignore file and folder matching strings including wildcards
+
+* All files will be parsed regardless but then any spelling errors found within
+ ignored files will not be reported as an error.
+* Errors in ignored files will still be output to the test results as
+ informational comments.
+
+### ExtendWords
+
+This list allows words to be added to the dictionary for the spell checker when
+this package is tested. These follow the rules of the cspell config words field.
+
+### IgnoreStandardPaths
+
+This plugin by default will check the below standard paths. If the package
+would like to ignore any of them list that here.
+
+```python
+[
+# C source
+"*.c",
+"*.h",
+
+# Assembly files
+"*.nasm",
+"*.asm",
+"*.masm",
+"*.s",
+
+# ACPI source language
+"*.asl",
+
+# Edk2 build files
+"*.dsc", "*.dec", "*.fdf", "*.inf",
+
+# Documentation files
+"*.md", "*.txt"
+]
+```
+
+### AdditionalIncludePaths
+
+If the package would to add additional path patterns to be included in
+spellchecking they can be defined here.
+
+## Other configuration
+
+In the cspell.base.json there are numerous other settings configured. There is
+no support to override these on a per package basis but future features could
+make this available. One interesting configuration option is `minWordLength`.
+Currently it is set to _5_ which means all 2,3, and 4 letter words will be
+ignored. This helps minimize the number of technical acronyms, register names,
+and other UEFI specific values that must be ignored.
+
+## False positives
+
+The cspell dictionary is not perfect and there are cases where technical words
+or acronyms are not found in the dictionary. There are three ways to resolve
+false positives and the choice for which method should be based on how broadly
+the word should be accepted.
+
+### CSpell Base Config file
+
+If the change should apply to all UEFI code and documentation then it should be
+added to the base config file `words` section. The base config file is adjacent
+to this file and titled `cspell.base.json`. This is a list of accepted words
+for all spell checking operations on all packages.
+
+### Package Config
+
+In the package `*.ci.yaml` file there is a `SpellCheck` config section. This
+section allows files to be ignored as well as words that should be considered
+valid for all files within this package. Add the desired words to the
+"ExtendedWords" member.
+
+### In-line File
+
+CSpell supports numerous methods to annotate your files to ignore words,
+sections, etc. This can be found in CSpell documentation. Suggestion here is
+to use a c-style comment at the top of the file to add words that should be
+ignored just for this file. Obviously this has the highest maintenance cost so
+it should only be used for file unique words.
+
+``` c
+// spell-checker:ignore unenroll, word2, word3
+```
+
+or
+
+```ini
+# spell-checker:ignore unenroll, word2, word3
+```
diff --git a/.pytool/Plugin/SpellCheck/SpellCheck.py b/.pytool/Plugin/SpellCheck/SpellCheck.py
new file mode 100644
index 0000000000..94ca4cd071
--- /dev/null
+++ b/.pytool/Plugin/SpellCheck/SpellCheck.py
@@ -0,0 +1,216 @@
+# @file SpellCheck.py
+#
+# An edk2-pytool based plugin wrapper for cspell
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import logging
+import json
+import yaml
+from io import StringIO
+import os
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toollib.utility_functions import RunCmd
+from edk2toolext.environment.var_dict import VarDict
+from edk2toollib.gitignore_parser import parse_gitignore_lines
+from edk2toolext.environment import version_aggregator
+
+
+class SpellCheck(ICiBuildPlugin):
+ """
+ A CiBuildPlugin that uses the cspell node module to scan the files
+ from the package being tested for spelling errors. The plugin contains
+ the base cspell.json file then thru the configuration options other settings
+ can be changed or extended.
+
+ Configuration options:
+ "SpellCheck": {
+ "AuditOnly": False, # Don't fail the build if there are errors. Just log them
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+ """
+
+ #
+ # A package can remove any of these using IgnoreStandardPaths
+ #
+ STANDARD_PLUGIN_DEFINED_PATHS = ["*.c", "*.h",
+ "*.nasm", "*.asm", "*.masm", "*.s",
+ "*.asl",
+ "*.dsc", "*.dec", "*.fdf", "*.inf",
+ "*.md", "*.txt"
+ ]
+
+ def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+ """ Provide the testcase name and classname for use in reporting
+
+ Args:
+ packagename: string containing name of package to build
+ environment: The VarDict for the test to run in
+ Returns:
+ a tuple containing the testcase name and the classname
+ (testcasename, classname)
+ testclassname: a descriptive string for the testcase can include whitespace
+ classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+ """
+ return ("Spell check files in " + packagename, packagename + ".SpellCheck")
+
+ ##
+ # External function of plugin. This function is used to perform the task of the CiBuild Plugin
+ #
+ # - package is the edk2 path to package. This means workspace/packagepath relative.
+ # - edk2path object configured with workspace and packages path
+ # - PkgConfig Object (dict) for the pkg
+ # - EnvConfig Object
+ # - Plugin Manager Instance
+ # - Plugin Helper Obj Instance
+ # - Junit Logger
+ # - output_stream the StringIO output stream from this plugin via logging
+
+ def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+ Errors = []
+
+ abs_pkg_path = Edk2pathObj.GetAbsolutePathOnThisSytemFromEdk2RelativePath(
+ packagename)
+
+ if abs_pkg_path is None:
+ tc.SetSkipped()
+ tc.LogStdError("No package {0}".format(packagename))
+ return -1
+
+ # check for node
+ return_buffer = StringIO()
+ ret = RunCmd("node", "--version", outstream=return_buffer)
+ if (ret != 0):
+ tc.SetSkipped()
+ tc.LogStdError("NodeJs not installed. Test can't run")
+ logging.warning("NodeJs not installed. Test can't run")
+ return -1
+ node_version = return_buffer.getvalue().strip() # format vXX.XX.XX
+ tc.LogStdOut(f"Node version: {node_version}")
+ version_aggregator.GetVersionAggregator().ReportVersion(
+ "NodeJs", node_version, version_aggregator.VersionTypes.INFO)
+
+ # Check for cspell
+ return_buffer = StringIO()
+ ret = RunCmd("cspell", "--version", outstream=return_buffer)
+ if (ret != 0):
+ tc.SetSkipped()
+ tc.LogStdError("cspell not installed. Test can't run")
+ logging.warning("cspell not installed. Test can't run")
+ return -1
+ cspell_version = return_buffer.getvalue().strip() # format XX.XX.XX
+ tc.LogStdOut(f"CSpell version: {cspell_version}")
+ version_aggregator.GetVersionAggregator().ReportVersion(
+ "CSpell", cspell_version, version_aggregator.VersionTypes.INFO)
+
+ package_relative_paths_to_spell_check = SpellCheck.STANDARD_PLUGIN_DEFINED_PATHS
+
+ #
+ # Allow the ci.yaml to remove any of the above standard paths
+ #
+ if("IgnoreStandardPaths" in pkgconfig):
+ for a in pkgconfig["IgnoreStandardPaths"]:
+ if(a in package_relative_paths_to_spell_check):
+ tc.LogStdOut(
+ f"ignoring standard path due to ci.yaml ignore: {a}")
+ package_relative_paths_to_spell_check.remove(a)
+ else:
+ tc.LogStdOut(f"Invalid IgnoreStandardPaths value: {a}")
+
+ #
+ # check for any additional include paths defined by package config
+ #
+ if("AdditionalIncludePaths" in pkgconfig):
+ package_relative_paths_to_spell_check.extend(
+ pkgconfig["AdditionalIncludePaths"])
+
+ #
+ # Make the path string for cspell to check
+ #
+ relpath = os.path.relpath(abs_pkg_path)
+ cpsell_paths = " ".join(
+ [f"{relpath}/**/{x}" for x in package_relative_paths_to_spell_check])
+
+ # Make the config file
+ config_file_path = os.path.join(
+ Edk2pathObj.WorkspacePath, "Build", packagename, "cspell_actual_config.json")
+ mydir = os.path.dirname(os.path.abspath(__file__))
+ # load as yaml so it can have comments
+ base = os.path.join(mydir, "cspell.base.yaml")
+ with open(base, "r") as i:
+ config = yaml.safe_load(i)
+
+ if("ExtendWords" in pkgconfig):
+ config["words"].extend(pkgconfig["ExtendWords"])
+ with open(config_file_path, "w") as o:
+ json.dump(config, o) # output as json so compat with cspell
+
+ All_Ignores = []
+ # Parse the config for other ignores
+ if "IgnoreFiles" in pkgconfig:
+ All_Ignores.extend(pkgconfig["IgnoreFiles"])
+
+ # spell check all the files
+ ignore = parse_gitignore_lines(All_Ignores, os.path.join(
+ abs_pkg_path, "nofile.txt"), abs_pkg_path)
+
+ # result is a list of strings like this
+ # C:\src\sp-edk2\edk2\FmpDevicePkg\FmpDevicePkg.dec:53:9 - Unknown word (Capule)
+ EasyFix = []
+ results = self._check_spelling(cpsell_paths, config_file_path)
+ for r in results:
+ path, _, word = r.partition(" - Unknown word ")
+ if len(word) == 0:
+ # didn't find pattern
+ continue
+
+ pathinfo = path.rsplit(":", 2) # remove the line no info
+ if(ignore(pathinfo[0])): # check against ignore list
+ tc.LogStdOut(f"ignoring error due to ci.yaml ignore: {r}")
+ continue
+
+ # real error
+ EasyFix.append(word.strip().strip("()"))
+ Errors.append(r)
+
+ # Log all errors tc StdError
+ for l in Errors:
+ tc.LogStdError(l.strip())
+
+ # Helper - Log the syntax needed to add these words to dictionary
+ if len(EasyFix) > 0:
+ EasyFix = sorted(set(a.lower() for a in EasyFix))
+ tc.LogStdOut("\n Easy fix:")
+ OneString = "If these are not errors add this to your ci.yaml file.\n"
+ OneString += '"SpellCheck": {\n "ExtendWords": ['
+ for a in EasyFix:
+ tc.LogStdOut(f'\n"{a}",')
+ OneString += f'\n "{a}",'
+ logging.info(OneString.rstrip(",") + '\n ]\n}')
+
+ # add result to test case
+ overall_status = len(Errors)
+ if overall_status != 0:
+ if "AuditOnly" in pkgconfig and pkgconfig["AuditOnly"]:
+ # set as skipped if AuditOnly
+ tc.SetSkipped()
+ return -1
+ else:
+ tc.SetFailed("SpellCheck {0} Failed. Errors {1}".format(
+ packagename, overall_status), "CHECK_FAILED")
+ else:
+ tc.SetSuccess()
+ return overall_status
+
+ def _check_spelling(self, abs_file_to_check: str, abs_config_file_to_use: str) -> []:
+ output = StringIO()
+ ret = RunCmd(
+ "cspell", f"--config {abs_config_file_to_use} {abs_file_to_check}", outstream=output)
+ if ret == 0:
+ return []
+ else:
+ return output.getvalue().strip().splitlines()
diff --git a/.pytool/Plugin/SpellCheck/SpellCheck_plug_in.yaml b/.pytool/Plugin/SpellCheck/SpellCheck_plug_in.yaml
new file mode 100644
index 0000000000..103aff2fb1
--- /dev/null
+++ b/.pytool/Plugin/SpellCheck/SpellCheck_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# CiBuildPlugin used to check spelling
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "scope": "cibuild",
+ "name": "Spell Check Test",
+ "module": "SpellCheck"
+}
diff --git a/.pytool/Plugin/SpellCheck/cspell.base.yaml b/.pytool/Plugin/SpellCheck/cspell.base.yaml
new file mode 100644
index 0000000000..53000fc381
--- /dev/null
+++ b/.pytool/Plugin/SpellCheck/cspell.base.yaml
@@ -0,0 +1,165 @@
+## @file
+# CSpell configuration
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "version": "0.1",
+ "language": "en",
+ "dictionaries": [
+ "companies ",
+ "softwareTerms",
+ "python",
+ "cpp"
+ ],
+ "ignorePaths": [
+ "*.pdb",
+ "**/*_extdep/**",
+ "*.pdf",
+ "*.exe",
+ "*.jpg"
+ ],
+ "minWordLength": 5,
+ "allowCompoundWords": false,
+ "ignoreWords": [
+ "muchange"
+ ],
+ "words": [
+ "MTRRs",
+ "Microarchitecture",
+ "Goldmont",
+ "cpuid",
+ "mwait",
+ "cstate",
+ "smram",
+ "scrtm",
+ "smbus",
+ "selftest",
+ "socket",
+ "MMRAM",
+ "qword",
+ "ENDBR",
+ "SMBASE",
+ "FXSAVE",
+ "FXRSTOR",
+ "RDRAND",
+ "IOAPIC",
+ "ATAPI",
+ "movsb",
+ "iretw",
+ "XENSTORE",
+ "cdrom",
+ "oprom",
+ "oproms",
+ "varstore",
+ "EKU",
+ "ascii",
+ "nmake",
+ "NVDIMM",
+ "nasmb",
+ "Mtftp",
+ "Hypercall",
+ "hypercalls",
+ "IOMMU",
+ "QEMU",
+ "qemus",
+ "OVMF",
+ "tiano",
+ "tianocore",
+ "edkii",
+ "coreboot",
+ "uefipayload",
+ "bootloader",
+ "bootloaders",
+ "mdepkg",
+ "skuid",
+ "dxefv",
+ "toolchain",
+ "libraryclass",
+ "preboot",
+ "pythonpath",
+ "cygpath",
+ "nuget",
+ "basetools",
+ "prepi",
+ "OPTEE",
+ "stringid",
+ "peims",
+ "memmap",
+ "guids",
+ "uuids",
+ "smbios",
+ "certdb",
+ "certdbv",
+ "EfiSigList",
+ "depex",
+ "IHANDLE",
+ "Virtio",
+ "Mbytes",
+ "Citrix",
+ "initrd",
+ "semihost",
+ "Semihosting",
+ "Trustzone",
+ "Fastboot",
+ "framebuffer",
+ "genfw",
+ "TTYTERM",
+ "miniport",
+ "LFENCE",
+ "PCANSI",
+ "submodule",
+ "submodules",
+ "brotli",
+ "PCCTS",
+ "softfloat",
+ "whitepaper",
+ "ACPICA",
+ "plugfest",
+ "bringup",
+ "formset", #VFR
+ "ideqvallist",
+ "numberof",
+ "oneof",
+ "endformset",
+ "endnumeric",
+ "endoneof",
+ "disableif",
+ "guidid",
+ "classguid",
+ "efivarstore",
+ "formsetguid",
+ "formid",
+ "suppressif",
+ "grayoutif",
+ "ideqval",
+ "endform",
+ "endcheckbox",
+ "questionid",
+ "questionref",
+ "enddate",
+ "endstring",
+ "guidop",
+ "endguidop",
+ "langdef",
+ "dynamicex",
+ "tokenspace",
+ "tokenguid",
+ "pcd's", #seems like cspell bug
+ "peim's",
+ "autogen",
+ "Disasm",
+ "Torito",
+ "SRIOV",
+ "MRIOV",
+ "UARTs",
+ "Consplitter", # common module in UEFI
+ "FIFOs",
+ "ACPINVS",
+ "Endof", # due to of not being uppercase
+ "bootability",
+ "Sdhci",
+ "inmodule",
+ ]
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 08/22] CryptoPkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (6 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 07/22] .pytool/Plugin: Add CI plugins Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 09/22] FatPkg: " Michael D Kinney
` (13 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Jian J Wang, Xiaoyu Lu, Kinney, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Xiaoyu Lu <xiaoyux.lu@intel.com>
Signed-off-by: Kinney <michael.d.kinney@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
CryptoPkg/CryptoPkg.ci.yaml | 47 +++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
create mode 100644 CryptoPkg/CryptoPkg.ci.yaml
diff --git a/CryptoPkg/CryptoPkg.ci.yaml b/CryptoPkg/CryptoPkg.ci.yaml
new file mode 100644
index 0000000000..f54ebfb22e
--- /dev/null
+++ b/CryptoPkg/CryptoPkg.ci.yaml
@@ -0,0 +1,47 @@
+## @file
+# CI configuration for CryptoPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "CryptoPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "CryptoPkg/CryptoPkg.dec",
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "CryptoPkg.dsc",
+ "IgnoreInf": []
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": []
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "skip": True,
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 09/22] FatPkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (7 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 08/22] CryptoPkg: Add YAML file for CI builds Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 10/22] FmpDevicePkg: " Michael D Kinney
` (12 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Ray Ni, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
FatPkg/FatPkg.ci.yaml | 50 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
create mode 100644 FatPkg/FatPkg.ci.yaml
diff --git a/FatPkg/FatPkg.ci.yaml b/FatPkg/FatPkg.ci.yaml
new file mode 100644
index 0000000000..8b0fb1d4fc
--- /dev/null
+++ b/FatPkg/FatPkg.ci.yaml
@@ -0,0 +1,50 @@
+## @file
+# CI configuration for FatPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "FatPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "IgnoreInf": [],
+ "DscPath": "FatPkg.dsc"
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": []
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+ "SpellCheck": {
+ "ExtendWords": [
+ "ELTORITO",
+ "FHAND",
+ "IFILE",
+ "OFILE",
+ "FDISKed",
+ "Lfnbuffer",
+ "FFFFFFFFL",
+ "CDVOL",
+ "DMDEPKG"
+ ]
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 10/22] FmpDevicePkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (8 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 09/22] FatPkg: " Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 11/22] MdeModulePkg: " Michael D Kinney
` (11 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Use BaseCryptLibNull for package CI builds to reduce package
build times. Enabled with CONTINUOUS_INTEGRATION in YAML
files. By default CONTINUOUS_INTEGRATION is not defined,
and the original lib mappings are preserved.
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
FmpDevicePkg/FmpDevicePkg.ci.yaml | 43 +++++++++++++++++++++++++++++++
FmpDevicePkg/FmpDevicePkg.dsc | 6 ++++-
2 files changed, 48 insertions(+), 1 deletion(-)
create mode 100644 FmpDevicePkg/FmpDevicePkg.ci.yaml
diff --git a/FmpDevicePkg/FmpDevicePkg.ci.yaml b/FmpDevicePkg/FmpDevicePkg.ci.yaml
new file mode 100644
index 0000000000..c17b355d42
--- /dev/null
+++ b/FmpDevicePkg/FmpDevicePkg.ci.yaml
@@ -0,0 +1,43 @@
+## @file
+# CI configuration for FmpDevicePkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "FmpDevicePkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "FmpDevicePkg/FmpDevicePkg.dec",
+ "CryptoPkg/CryptoPkg.dec"
+ ],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "FmpDevicePkg.dsc",
+ "IgnoreInf": []
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": []
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+ "SpellCheck": {
+ "ExtendWords": [
+ "FMPSTATE",
+ ]
+ },
+ "Defines": {
+ "BLD_*_CONTINUOUS_INTEGRATION": "TRUE",
+ }
+}
diff --git a/FmpDevicePkg/FmpDevicePkg.dsc b/FmpDevicePkg/FmpDevicePkg.dsc
index 67b5aa7a32..f4093d3837 100644
--- a/FmpDevicePkg/FmpDevicePkg.dsc
+++ b/FmpDevicePkg/FmpDevicePkg.dsc
@@ -48,9 +48,13 @@ [LibraryClasses]
DebugLib|MdePkg/Library/UefiDebugLibStdErr/UefiDebugLibStdErr.inf
DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
PcdLib|MdePkg/Library/BasePcdLibNull/BasePcdLibNull.inf
- OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
+!endif
FmpAuthenticationLib|SecurityPkg/Library/FmpAuthenticationLibPkcs7/FmpAuthenticationLibPkcs7.inf
CapsuleUpdatePolicyLib|FmpDevicePkg/Library/CapsuleUpdatePolicyLibNull/CapsuleUpdatePolicyLibNull.inf
FmpPayloadHeaderLib|FmpDevicePkg/Library/FmpPayloadHeaderLibV1/FmpPayloadHeaderLibV1.inf
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 11/22] MdeModulePkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (9 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 10/22] FmpDevicePkg: " Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 12/22] MdePkg: " Michael D Kinney
` (10 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Jian J Wang, Hao A Wu, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
MdeModulePkg/MdeModulePkg.ci.yaml | 80 +++++++++++++++++++++++++++++++
1 file changed, 80 insertions(+)
create mode 100644 MdeModulePkg/MdeModulePkg.ci.yaml
diff --git a/MdeModulePkg/MdeModulePkg.ci.yaml b/MdeModulePkg/MdeModulePkg.ci.yaml
new file mode 100644
index 0000000000..0bf149f205
--- /dev/null
+++ b/MdeModulePkg/MdeModulePkg.ci.yaml
@@ -0,0 +1,80 @@
+## @file
+# CI configuration for MdeModulePkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ ## options defined ci/Plugin/CompilerPlugin
+ "CompilerPlugin": {
+ "DscPath": "MdeModulePkg.dsc"
+ },
+
+ ## options defined ci/Plugin/CharEncodingCheck
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+
+ ## options defined ci/Plugin/DependencyCheck
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "StandaloneMmPkg/StandaloneMmPkg.dec",
+ "ArmPkg/ArmPkg.dec" # this should be fixed by promoting an abstraction
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+
+ ## options defined ci/Plugin/DscCompleteCheck
+ "DscCompleteCheck": {
+ "IgnoreInf": [],
+ "DscPath": "MdeModulePkg.dsc"
+ },
+
+ ## options defined ci/Plugin/GuidCheck
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": ["00000000-0000-0000-0000-000000000000"],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [
+ "gEdkiiFormBrowserExProtocolGuid=gEfiFormBrowserExProtocolGuid",
+ "gEfiPeiMmAccessPpiGuid=gPeiSmmAccessPpiGuid",
+ "gPeiSmmControlPpiGuid=gEfiPeiMmControlPpiGuid",
+ ]
+ },
+
+ ## options defined ci/Plugin/LibraryClassCheck
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails test but run in AuditOnly mode to collect log
+ "IgnoreStandardPaths": [ # Standard Plugin defined paths that should be ignore
+ "*.c", "*.asm", "*.h", "*.nasm", "*.s", "*.asl", "*.inf"
+ ],
+ "IgnoreFiles": [ # use gitignore syntax to ignore errors in matching files
+ "Library/LzmaCustomDecompressLib/Sdk/DOC/*"
+ ],
+ "ExtendWords": [ # words to extend to the dictionary for this package
+ "LIGHTGRAY",
+ "DARKGRAY",
+ "LIGHTBLUE",
+ "LIGHTGREEN",
+ "LIGHTCYAN",
+ "LIGHTRED",
+ "LIGHTMAGENTA",
+ "FVMAIN",
+ "VARCHECKPCD",
+ "Getxx",
+ "lzturbo"
+ ],
+ "AdditionalIncludePaths": [] # Additional paths to spell check relative to package root (wildcards supported)
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 12/22] MdePkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (10 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 11/22] MdeModulePkg: " Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 13/22] NetworkPkg: " Michael D Kinney
` (9 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
MdePkg/MdePkg.ci.yaml | 90 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 90 insertions(+)
create mode 100644 MdePkg/MdePkg.ci.yaml
diff --git a/MdePkg/MdePkg.ci.yaml b/MdePkg/MdePkg.ci.yaml
new file mode 100644
index 0000000000..65c4ec3bec
--- /dev/null
+++ b/MdePkg/MdePkg.ci.yaml
@@ -0,0 +1,90 @@
+## @file
+# CI configuration for MdePkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ ## options defined ci/Plugin/CompilerPlugin
+ "CompilerPlugin": {
+ "DscPath": "MdePkg.dsc"
+ },
+
+ ## options defined ci/Plugin/CharEncodingCheck
+"CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+
+ ## options defined ci/Plugin/DependencyCheck
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+
+ ## options defined ci/Plugin/DscCompleteCheck
+ "DscCompleteCheck": {
+ "IgnoreInf": [],
+ "DscPath": "MdePkg.dsc"
+ },
+
+ ## options defined ci/Plugin/GuidCheck
+ "GuidCheck": {
+ "IgnoreGuidName": [
+ "gEfiFirmwareVolumeTopFileGuid" # sec modules must be set to this guid
+ ],
+ "IgnoreGuidValue": ["00000000-0000-0000-0000-000000000000"],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [
+ "gEfiFirmwareVolumeBlockProtocolGuid=gEfiFirmwareVolumeBlock2ProtocolGuid",
+ "gEfiMmAccessProtocolGuid=gEfiSmmAccess2ProtocolGuid",
+ "gEfiSmmCommunicationProtocolGuid=gEfiMmCommunicationProtocolGuid",
+ "gEfiMmPciRootBridgeIoProtocolGuid=gEfiSmmPciRootBridgeIoProtocolGuid",
+ "gEfiTrEEProtocolGuid=gEfiTcg2ProtocolGuid",
+ "gEfiSmmPowerButtonDispatch2ProtocolGuid=gEfiMmPowerButtonDispatchProtocolGuid",
+ "gEfiSmmBase2ProtocolGuid=gEfiMmBaseProtocolGuid",
+ "gEfiSmmUsbDispatch2ProtocolGuid=gEfiMmUsbDispatchProtocolGuid",
+ "gEfiSmmCpuProtocolGuid=gEfiMmCpuProtocolGuid",
+ "gEfiAcpiTableGuid=gEfiAcpi20TableGuid",
+ "gEfiSmmControl2ProtocolGuid=gEfiMmControlProtocolGuid",
+ "gEfiSmmStandbyButtonDispatch2ProtocolGuid=gEfiMmStandbyButtonDispatchProtocolGuid",
+ "gEfiSmmStatusCodeProtocolGuid=gEfiMmStatusCodeProtocolGuid",
+ "gEfiDxeSmmReadyToLockProtocolGuid=gEfiDxeMmReadyToLockProtocolGuid",
+ "gEfiSmmIoTrapDispatch2ProtocolGuid=gEfiMmIoTrapDispatchProtocolGuid",
+ "gEfiSmmReadyToLockProtocolGuid=gEfiMmReadyToLockProtocolGuid",
+ "gEfiSmmSxDispatch2ProtocolGuid=gEfiMmSxDispatchProtocolGuid",
+ "gEfiPeiCapsulePpiGuid=gPeiCapsulePpiGuid",
+ "gEfiSmmCpuIo2ProtocolGuid=gEfiMmCpuIoProtocolGuid",
+ "gEfiSmmRscHandlerProtocolGuid=gEfiMmRscHandlerProtocolGuid",
+ "gEfiSmmConfigurationProtocolGuid=gEfiMmConfigurationProtocolGuid",
+ "gEfiSmmGpiDispatch2ProtocolGuid=gEfiMmGpiDispatchProtocolGuid",
+ "gEfiSmmEndOfDxeProtocolGuid=gEfiMmEndOfDxeProtocolGuid",
+ "gEfiSmmSwDispatch2ProtocolGuid=gEfiMmSwDispatchProtocolGuid",
+ "gEfiDebugPortProtocolGuid=gEfiDebugPortVariableGuid=gEfiDebugPortDevicePathGuid", ## is this a bug
+ "gEfiProcessorSpecificErrorSectionGuid=gEfiIa32X64ProcessorErrorSectionGuid", ## is this a bug
+ "gEfiSmmPeriodicTimerDispatch2ProtocolGuid=gEfiMmPeriodicTimerDispatchProtocolGuid",
+ "gEdkiiFormBrowserExProtocolGuid=gEfiFormBrowserExProtocolGuid",
+ "gEfiPeiMmAccessPpiGuid=gPeiSmmAccessPpiGuid",
+ "gPeiSmmControlPpiGuid=gEfiPeiMmControlPpiGuid"
+ ]
+ },
+
+ ## options defined ci/Plugin/LibraryClassCheck
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails test but run in AuditOnly mode to collect log
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 13/22] NetworkPkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (11 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 12/22] MdePkg: " Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:40 ` [Patch v5 14/22] PcAtChipsetPkg: Add YAML files " Michael D Kinney
` (8 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Jiaxin Wu, Siyuan Fu, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Use BaseCryptLibNull and TlsLibNull for package CI
builds to reduce package build times. Enabled with
CONTINUOUS_INTEGRATION in YAML files. By default
CONTINUOUS_INTEGRATION is not defined, and the
original lib mappings are preserved.
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
NetworkPkg/NetworkPkg.ci.yaml | 62 +++++++++++++++++++++++++++++++++++
NetworkPkg/NetworkPkg.dsc | 5 +++
2 files changed, 67 insertions(+)
create mode 100644 NetworkPkg/NetworkPkg.ci.yaml
diff --git a/NetworkPkg/NetworkPkg.ci.yaml b/NetworkPkg/NetworkPkg.ci.yaml
new file mode 100644
index 0000000000..70f2e10147
--- /dev/null
+++ b/NetworkPkg/NetworkPkg.ci.yaml
@@ -0,0 +1,62 @@
+## @file
+# CI configuration for NetworkPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "NetworkPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "NetworkPkg/NetworkPkg.dec",
+ "CryptoPkg/CryptoPkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[
+ "ShellPkg/ShellPkg.dec"
+ ],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "NetworkPkg.dsc",
+ "IgnoreInf": []
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": []
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails test but run in AuditOnly mode to collect log
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ },
+
+ "Defines": {
+ "BLD_*_CONTINUOUS_INTEGRATION": "TRUE",
+ "BLD_*_NETWORK_ENABLE": "TRUE",
+ "BLD_*_NETWORK_SNP_ENABLE": "TRUE",
+ "BLD_*_NETWORK_VLAN_ENABLE": "TRUE",
+ "BLD_*_NETWORK_IP4_ENABLE": "TRUE",
+ "BLD_*_NETWORK_IP6_ENABLE": "TRUE",
+ "BLD_*_NETWORK_TLS_ENABLE": "TRUE",
+ "BLD_*_NETWORK_HTTP_BOOT_ENABLE": "TRUE",
+ "BLD_*_NETWORK_ISCSI_ENABLE": "TRUE",
+ }
+}
diff --git a/NetworkPkg/NetworkPkg.dsc b/NetworkPkg/NetworkPkg.dsc
index 11a2981261..b149453d26 100644
--- a/NetworkPkg/NetworkPkg.dsc
+++ b/NetworkPkg/NetworkPkg.dsc
@@ -41,10 +41,15 @@ [LibraryClasses]
DxeServicesTableLib|MdePkg/Library/DxeServicesTableLib/DxeServicesTableLib.inf
SafeIntLib|MdePkg/Library/BaseSafeIntLib/BaseSafeIntLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+ TlsLib|CryptoPkg/Library/TlsLibNull/TlsLibNull.inf
+!else
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
TlsLib|CryptoPkg/Library/TlsLib/TlsLib.inf
+!endif
DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
FileHandleLib|MdePkg/Library/UefiFileHandleLib/UefiFileHandleLib.inf
FileExplorerLib|MdeModulePkg/Library/FileExplorerLib/FileExplorerLib.inf
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 14/22] PcAtChipsetPkg: Add YAML files for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (12 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 13/22] NetworkPkg: " Michael D Kinney
@ 2019-11-11 8:40 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 15/22] SecurityPkg: " Michael D Kinney
` (7 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:40 UTC (permalink / raw)
To: devel; +Cc: Ray Ni, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml | 46 +++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
create mode 100644 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
diff --git a/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml b/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
new file mode 100644
index 0000000000..be470807bd
--- /dev/null
+++ b/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
@@ -0,0 +1,46 @@
+## @file
+# CI configuration for PcAtChipsetPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "PcAtChipsetPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "PcAtChipsetPkg/PcAtChipsetPkg.dec",
+ "UefiCpuPkg/UefiCpuPkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "PcAtChipsetPkg.dsc",
+ "IgnoreInf": []
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": []
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+ "SpellCheck": {
+ "ExtendWords": [
+ "ENUMER", # this is part of an IDE enum
+ "PCATCHIPSET",
+ "TXRDY"
+ ]
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 15/22] SecurityPkg: Add YAML files for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (13 preceding siblings ...)
2019-11-11 8:40 ` [Patch v5 14/22] PcAtChipsetPkg: Add YAML files " Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 16/22] ShellPkg: Add YAML file " Michael D Kinney
` (6 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Jiewen Yao, Jian J Wang, Chao Zhang, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Use BaseCryptLibNull for package CI builds to reduce package
build times. Enabled with CONTINUOUS_INTEGRATION in YAML
files. By default CONTINUOUS_INTEGRATION is not defined,
and the original lib mappings are preserved.
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
---
SecurityPkg/SecurityPkg.ci.yaml | 80 +++++++++++++++++++++++++++++++++
SecurityPkg/SecurityPkg.dsc | 32 ++++++++++++-
2 files changed, 110 insertions(+), 2 deletions(-)
create mode 100644 SecurityPkg/SecurityPkg.ci.yaml
diff --git a/SecurityPkg/SecurityPkg.ci.yaml b/SecurityPkg/SecurityPkg.ci.yaml
new file mode 100644
index 0000000000..9532190533
--- /dev/null
+++ b/SecurityPkg/SecurityPkg.ci.yaml
@@ -0,0 +1,80 @@
+## @file
+# CI configuration for SecurityPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "SecurityPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "SecurityPkg/SecurityPkg.dec",
+ "CryptoPkg/CryptoPkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "SecurityPkg.dsc",
+ "IgnoreInf": []
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": ["00000000-0000-0000-0000-000000000000"],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [
+ "Tpm2InstanceLibDTpm=gEfiTpmDeviceInstanceTpm20DtpmGuid", # by design
+ ]
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails test but run in AuditOnly mode to collect log
+ "ExtendWords": [ # words to extend to the dictionary for this package
+ "shortformed", # tpm acpi
+ "autodetect",
+ "blocksid",
+ "comid",
+ "cpinsidpin", #OpalSScV2
+ "ecdsa", # TPM
+ "ecschnorr", # TPM
+ "eisaid", # ACPI
+ "harddisk",
+ "hashall",
+ "hashto",
+ "kek's",
+ "lfanew", # PE/COFF
+ "pcrindex",
+ "pkglength",
+ "ppuser",
+ "preos",
+ "stclear",
+ "toctou",
+ "tpm's",
+ "tpmcmdbuflength",
+ "tpmcommlib",
+ "tpmnvvaluelength",
+ "wrlocked",
+ "xored"
+ ],
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ },
+
+ "Defines": {
+ "BLD_*_CONTINUOUS_INTEGRATION": "TRUE",
+ }
+}
diff --git a/SecurityPkg/SecurityPkg.dsc b/SecurityPkg/SecurityPkg.dsc
index 9a254087a3..a2eeadda7a 100644
--- a/SecurityPkg/SecurityPkg.dsc
+++ b/SecurityPkg/SecurityPkg.dsc
@@ -44,8 +44,6 @@ [LibraryClasses]
HiiLib|MdeModulePkg/Library/UefiHiiLib/UefiHiiLib.inf
UefiHiiServicesLib|MdeModulePkg/Library/UefiHiiServicesLib/UefiHiiServicesLib.inf
PcdLib|MdePkg/Library/BasePcdLibNull/BasePcdLibNull.inf
- IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
- OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsic.inf
TpmCommLib|SecurityPkg/Library/TpmCommLib/TpmCommLib.inf
PlatformSecureLib|SecurityPkg/Library/PlatformSecureLibNull/PlatformSecureLibNull.inf
@@ -84,7 +82,13 @@ [LibraryClasses.common.PEIM]
PeiServicesTablePointerLib|MdePkg/Library/PeiServicesTablePointerLib/PeiServicesTablePointerLib.inf
HobLib|MdePkg/Library/PeiHobLib/PeiHobLib.inf
MemoryAllocationLib|MdePkg/Library/PeiMemoryAllocationLib/PeiMemoryAllocationLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/PeiCryptLib.inf
+!endif
HashLib|SecurityPkg/Library/HashLibBaseCryptoRouter/HashLibBaseCryptoRouterPei.inf
ReportStatusCodeLib|MdeModulePkg/Library/PeiReportStatusCodeLib/PeiReportStatusCodeLib.inf
Tpm12DeviceLib|SecurityPkg/Library/Tpm12DeviceLibDTpm/Tpm12DeviceLibDTpm.inf
@@ -95,7 +99,13 @@ [LibraryClasses.common.PEIM]
[LibraryClasses.common.DXE_DRIVER]
HobLib|MdePkg/Library/DxeHobLib/DxeHobLib.inf
ReportStatusCodeLib|MdeModulePkg/Library/DxeReportStatusCodeLib/DxeReportStatusCodeLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
+!endif
HashLib|SecurityPkg/Library/HashLibBaseCryptoRouter/HashLibBaseCryptoRouterDxe.inf
Tpm12DeviceLib|SecurityPkg/Library/Tpm12DeviceLibTcg/Tpm12DeviceLibTcg.inf
Tpm2DeviceLib|SecurityPkg/Library/Tpm2DeviceLibTcg2/Tpm2DeviceLibTcg2.inf
@@ -110,13 +120,25 @@ [LibraryClasses.common.UEFI_DRIVER, LibraryClasses.common.DXE_RUNTIME_DRIVER, Li
[LibraryClasses.common.DXE_RUNTIME_DRIVER]
ReportStatusCodeLib|MdeModulePkg/Library/RuntimeDxeReportStatusCodeLib/RuntimeDxeReportStatusCodeLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf
+!endif
HashLib|SecurityPkg/Library/HashLibBaseCryptoRouter/HashLibBaseCryptoRouterDxe.inf
Tpm12DeviceLib|SecurityPkg/Library/Tpm12DeviceLibTcg/Tpm12DeviceLibTcg.inf
Tpm2DeviceLib|SecurityPkg/Library/Tpm2DeviceLibTcg2/Tpm2DeviceLibTcg2.inf
[LibraryClasses.common.UEFI_DRIVER, LibraryClasses.common.UEFI_APPLICATION]
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
+!endif
HashLib|SecurityPkg/Library/HashLibBaseCryptoRouter/HashLibBaseCryptoRouterDxe.inf
Tpm12DeviceLib|SecurityPkg/Library/Tpm12DeviceLibTcg/Tpm12DeviceLibTcg.inf
Tpm2DeviceLib|SecurityPkg/Library/Tpm2DeviceLibTcg2/Tpm2DeviceLibTcg2.inf
@@ -127,7 +149,13 @@ [LibraryClasses.common.DXE_SMM_DRIVER]
MemoryAllocationLib|MdePkg/Library/SmmMemoryAllocationLib/SmmMemoryAllocationLib.inf
ReportStatusCodeLib|MdeModulePkg/Library/SmmReportStatusCodeLib/SmmReportStatusCodeLib.inf
SmmMemLib|MdePkg/Library/SmmMemLib/SmmMemLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf
+!endif
Tpm12DeviceLib|SecurityPkg/Library/Tpm12DeviceLibTcg/Tpm12DeviceLibTcg.inf
Tpm2DeviceLib|SecurityPkg/Library/Tpm2DeviceLibTcg2/Tpm2DeviceLibTcg2.inf
Tcg2PhysicalPresenceLib|SecurityPkg/Library/SmmTcg2PhysicalPresenceLib/SmmTcg2PhysicalPresenceLib.inf
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 16/22] ShellPkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (14 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 15/22] SecurityPkg: " Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 17/22] UefiCpuPkg: " Michael D Kinney
` (5 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Ray Ni, Zhichao Gao, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Ray Ni <ray.ni@intel.com>
Cc: Zhichao Gao <zhichao.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Zhichao Gao <zhichao.gao@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
ShellPkg/ShellPkg.ci.yaml | 55 +++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
create mode 100644 ShellPkg/ShellPkg.ci.yaml
diff --git a/ShellPkg/ShellPkg.ci.yaml b/ShellPkg/ShellPkg.ci.yaml
new file mode 100644
index 0000000000..67de34a255
--- /dev/null
+++ b/ShellPkg/ShellPkg.ci.yaml
@@ -0,0 +1,55 @@
+## @file
+# CI configuration for ShellPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "ShellPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "ShellPkg/ShellPkg.dec",
+ "NetworkPkg/NetworkPkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "ShellPkg.dsc",
+ "IgnoreInf": [
+ "ShellPkg/Application/ShellCTestApp/ShellCTestApp.inf",
+ "ShellPkg/Application/ShellExecTestApp/SA.inf",
+ "ShellPkg/Application/ShellSortTestApp/ShellSortTestApp.inf"
+ ]
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [
+ "Shell=gUefiShellFileGuid", # by design
+ ]
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails test but run in AuditOnly mode to collect log
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 17/22] UefiCpuPkg: Add YAML file for CI builds
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (15 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 16/22] ShellPkg: Add YAML file " Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 18/22] SignedCapsulePkg: Use BaseCryptLibNull to reduce package CI time Michael D Kinney
` (4 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Eric Dong, Ray Ni, Laszlo Ersek, Liming Gao
Add YAML file to the package directory with the
configuration of the checks to perform during a
CI build.
Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Ray Ni <ray.ni@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
UefiCpuPkg/UefiCpuPkg.ci.yaml | 51 +++++++++++++++++++++++++++++++++++
1 file changed, 51 insertions(+)
create mode 100644 UefiCpuPkg/UefiCpuPkg.ci.yaml
diff --git a/UefiCpuPkg/UefiCpuPkg.ci.yaml b/UefiCpuPkg/UefiCpuPkg.ci.yaml
new file mode 100644
index 0000000000..99e460a8b0
--- /dev/null
+++ b/UefiCpuPkg/UefiCpuPkg.ci.yaml
@@ -0,0 +1,51 @@
+## @file
+# CI configuration for UefiCpuPkg
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "CompilerPlugin": {
+ "DscPath": "UefiCpuPkg.dsc"
+ },
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "UefiCpuPkg/UefiCpuPkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+ "DscCompleteCheck": {
+ "DscPath": "UefiCpuPkg.dsc",
+ "IgnoreInf": [
+ "UefiCpuPkg/ResetVector/FixupVtf/Vtf.inf",
+ "UefiCpuPkg/ResetVector/Vtf0/Vtf0.inf"
+ ]
+ },
+ "GuidCheck": {
+ "IgnoreGuidName": ["SecCore", "ResetVector"], # Expected duplication for gEfiFirmwareVolumeTopFileGuid
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": []
+ },
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined ci/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails test but run in AuditOnly mode to collect log
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 18/22] SignedCapsulePkg: Use BaseCryptLibNull to reduce package CI time
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (16 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 17/22] UefiCpuPkg: " Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 19/22] .pytool: Add CISettings.py and Readme.md Michael D Kinney
` (3 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Jiewen Yao, Chao Zhang, Liming Gao
Use BaseCryptLibNull for package CI builds to reduce package
build times. Enabled with PYTOOL_CONTINUOUS_INTEGRATION in YAML
files. By default PYTOOL_CONTINUOUS_INTEGRATION is not defined,
and the original lib mappings are preserved.
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
SignedCapsulePkg/SignedCapsulePkg.dsc | 38 +++++++++++++++++++++++++--
1 file changed, 36 insertions(+), 2 deletions(-)
diff --git a/SignedCapsulePkg/SignedCapsulePkg.dsc b/SignedCapsulePkg/SignedCapsulePkg.dsc
index 03f714f9a7..c5080ec1dd 100644
--- a/SignedCapsulePkg/SignedCapsulePkg.dsc
+++ b/SignedCapsulePkg/SignedCapsulePkg.dsc
@@ -87,8 +87,6 @@ [LibraryClasses]
AuthVariableLib|MdeModulePkg/Library/AuthVariableLibNull/AuthVariableLibNull.inf
VarCheckLib|MdeModulePkg/Library/VarCheckLib/VarCheckLib.inf
FileExplorerLib|MdeModulePkg/Library/FileExplorerLib/FileExplorerLib.inf
- IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
- OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
EdkiiSystemCapsuleLib|SignedCapsulePkg/Library/EdkiiSystemCapsuleLib/EdkiiSystemCapsuleLib.inf
IniParsingLib|SignedCapsulePkg/Library/IniParsingLib/IniParsingLib.inf
@@ -116,7 +114,13 @@ [LibraryClasses.common.PEIM]
HobLib|MdePkg/Library/PeiHobLib/PeiHobLib.inf
MemoryAllocationLib|MdePkg/Library/PeiMemoryAllocationLib/PeiMemoryAllocationLib.inf
LockBoxLib|MdeModulePkg/Library/SmmLockBoxLib/SmmLockBoxPeiLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/PeiCryptLib.inf
+!endif
[LibraryClasses.common.DXE_CORE]
HobLib|MdePkg/Library/DxeCoreHobLib/DxeCoreHobLib.inf
@@ -125,7 +129,13 @@ [LibraryClasses.common.DXE_CORE]
[LibraryClasses.common.DXE_DRIVER]
HobLib|MdePkg/Library/DxeHobLib/DxeHobLib.inf
MemoryAllocationLib|MdePkg/Library/UefiMemoryAllocationLib/UefiMemoryAllocationLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
+!endif
CapsuleLib|MdeModulePkg/Library/DxeCapsuleLibFmp/DxeCapsuleLib.inf
@@ -133,7 +143,13 @@ [LibraryClasses.common.DXE_RUNTIME_DRIVER]
HobLib|MdePkg/Library/DxeHobLib/DxeHobLib.inf
MemoryAllocationLib|MdePkg/Library/UefiMemoryAllocationLib/UefiMemoryAllocationLib.inf
DebugLib|MdePkg/Library/UefiDebugLibConOut/UefiDebugLibConOut.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf
+!endif
CapsuleLib|MdeModulePkg/Library/DxeCapsuleLibFmp/DxeRuntimeCapsuleLib.inf
@@ -150,19 +166,37 @@ [LibraryClasses.common.DXE_SMM_DRIVER]
MemoryAllocationLib|MdePkg/Library/SmmMemoryAllocationLib/SmmMemoryAllocationLib.inf
SmmServicesTableLib|MdePkg/Library/SmmServicesTableLib/SmmServicesTableLib.inf
SmmMemLib|MdePkg/Library/SmmMemLib/SmmMemLib.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf
+!endif
[LibraryClasses.common.UEFI_DRIVER]
HobLib|MdePkg/Library/DxeHobLib/DxeHobLib.inf
MemoryAllocationLib|MdePkg/Library/UefiMemoryAllocationLib/UefiMemoryAllocationLib.inf
DebugLib|MdePkg/Library/UefiDebugLibConOut/UefiDebugLibConOut.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
+!endif
[LibraryClasses.common.UEFI_APPLICATION]
HobLib|MdePkg/Library/DxeHobLib/DxeHobLib.inf
MemoryAllocationLib|MdePkg/Library/UefiMemoryAllocationLib/UefiMemoryAllocationLib.inf
DebugLib|MdePkg/Library/UefiDebugLibStdErr/UefiDebugLibStdErr.inf
+!ifdef CONTINUOUS_INTEGRATION
+ BaseCryptLib|CryptoPkg/Library/BaseCryptLibNull/BaseCryptLibNull.inf
+!else
+ IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+ OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
BaseCryptLib|CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
+!endif
[PcdsFixedAtBuild]
gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask|0x0f
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 19/22] .pytool: Add CISettings.py and Readme.md
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (17 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 18/22] SignedCapsulePkg: Use BaseCryptLibNull to reduce package CI time Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 20/22] .azurepipelines: Add Azure Pipelines YML configuration files Michael D Kinney
` (2 subsequent siblings)
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bret Barkelew, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Add main python script for EDK II Continuous Integration (CI)
builds along with a Readme.md that provides a summary of the
packages, platforms, and checks performs during a CI build.
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
.pytool/CISettings.py | 173 ++++++++++++++++++++++++++++++++
.pytool/Readme.md | 223 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 396 insertions(+)
create mode 100644 .pytool/CISettings.py
create mode 100644 .pytool/Readme.md
diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py
new file mode 100644
index 0000000000..a78e8b974c
--- /dev/null
+++ b/.pytool/CISettings.py
@@ -0,0 +1,173 @@
+# @file
+#
+# Copyright (c) 2018, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+from edk2toolext.environment import shell_environment
+from edk2toolext.invocables.edk2_ci_build import CiBuildSettingsManager
+from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule
+from edk2toolext.invocables.edk2_update import UpdateSettingsManager
+from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager
+from edk2toollib.utility_functions import GetHostInfo
+
+
+class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager):
+
+ def __init__(self):
+ self.ActualPackages = []
+ self.ActualTargets = []
+ self.ActualArchitectures = []
+ self.ActualToolChainTag = ""
+
+ # ####################################################################################### #
+ # Extra CmdLine configuration #
+ # ####################################################################################### #
+
+ def AddCommandLineOptions(self, parserObj):
+ pass
+
+ def RetrieveCommandLineOptions(self, args):
+ pass
+
+ # ####################################################################################### #
+ # Default Support for this Ci Build #
+ # ####################################################################################### #
+
+ def GetPackagesSupported(self):
+ ''' return iterable of edk2 packages supported by this build.
+ These should be edk2 workspace relative paths '''
+
+ return ("MdePkg",
+ "MdeModulePkg",
+ "NetworkPkg",
+ "PcAtChipsetPkg",
+ "SecurityPkg",
+ "UefiCpuPkg",
+ "FmpDevicePkg",
+ "ShellPkg",
+ "FatPkg",
+ "CryptoPkg"
+ )
+
+ def GetArchitecturesSupported(self):
+ ''' return iterable of edk2 architectures supported by this build '''
+ return ("IA32",
+ "X64",
+ "ARM",
+ "AARCH64")
+
+ def GetTargetsSupported(self):
+ ''' return iterable of edk2 target tags supported by this build '''
+ return ("DEBUG", "RELEASE", "NO-TARGET", "NOOPT")
+
+ # ####################################################################################### #
+ # Verify and Save requested Ci Build Config #
+ # ####################################################################################### #
+
+ def SetPackages(self, list_of_requested_packages):
+ ''' Confirm the requested package list is valid and configure SettingsManager
+ to build the requested packages.
+
+ Raise UnsupportedException if a requested_package is not supported
+ '''
+ unsupported = set(list_of_requested_packages) - \
+ set(self.GetPackagesSupported())
+ if(len(unsupported) > 0):
+ logging.critical(
+ "Unsupported Package Requested: " + " ".join(unsupported))
+ raise Exception("Unsupported Package Requested: " +
+ " ".join(unsupported))
+ self.ActualPackages = list_of_requested_packages
+
+ def SetArchitectures(self, list_of_requested_architectures):
+ ''' Confirm the requests architecture list is valid and configure SettingsManager
+ to run only the requested architectures.
+
+ Raise Exception if a list_of_requested_architectures is not supported
+ '''
+ unsupported = set(list_of_requested_architectures) - \
+ set(self.GetArchitecturesSupported())
+ if(len(unsupported) > 0):
+ logging.critical(
+ "Unsupported Architecture Requested: " + " ".join(unsupported))
+ raise Exception(
+ "Unsupported Architecture Requested: " + " ".join(unsupported))
+ self.ActualArchitectures = list_of_requested_architectures
+
+ def SetTargets(self, list_of_requested_target):
+ ''' Confirm the request target list is valid and configure SettingsManager
+ to run only the requested targets.
+
+ Raise UnsupportedException if a requested_target is not supported
+ '''
+ unsupported = set(list_of_requested_target) - \
+ set(self.GetTargetsSupported())
+ if(len(unsupported) > 0):
+ logging.critical(
+ "Unsupported Targets Requested: " + " ".join(unsupported))
+ raise Exception("Unsupported Targets Requested: " +
+ " ".join(unsupported))
+ self.ActualTargets = list_of_requested_target
+
+ # ####################################################################################### #
+ # Actual Configuration for Ci Build #
+ # ####################################################################################### #
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ scopes = ("cibuild","edk2-build")
+
+ self.ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "")
+
+ if GetHostInfo().os.upper() == "LINUX" and self.ActualToolChainTag.upper().startswith("GCC"):
+ if "AARCH64" in self.ActualArchitectures:
+ scopes += ("gcc_aarch64_linux",)
+ if "ARM" in self.ActualArchitectures:
+ scopes += ("gcc_arm_linux",)
+
+ return scopes
+
+ def GetRequiredSubmodules(self):
+ ''' return iterable containing RequiredSubmodule objects.
+ If no RequiredSubmodules return an empty iterable
+ '''
+ rs=[]
+ rs.append(RequiredSubmodule(
+ "ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3", False))
+ rs.append(RequiredSubmodule(
+ "CryptoPkg/Library/OpensslLib/openssl", False))
+ return rs
+
+ def GetName(self):
+ return "Edk2"
+
+ def GetDependencies(self):
+ return []
+
+ def GetPackagesPath(self):
+ return ()
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+ def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list:
+ ''' Filter potential packages to test based on changed files. '''
+ build_these_packages=[]
+ possible_packages=potentialPackagesList.copy()
+ for f in changedFilesList:
+ nodes=f.split("/") # split each part of path for comparison later
+
+ # python file change in .pytool folder causes building all
+ if f.endswith(".py") and ".pytool" in nodes:
+ build_these_packages = possible_packages
+ break
+
+ # BaseTools files that might change the build
+ if "BaseTools" in nodes:
+ if os.path.splitext(f) not in [".txt", ".md"]:
+ build_these_packages = possible_packages
+ break
+ return build_these_packages
diff --git a/.pytool/Readme.md b/.pytool/Readme.md
new file mode 100644
index 0000000000..34be0e7a24
--- /dev/null
+++ b/.pytool/Readme.md
@@ -0,0 +1,223 @@
+# Edk2 Continuous Integration
+
+## Basic Status
+
+| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |
+| :---- | :----- | :---- | :--- |
+| ArmPkg |
+| ArmPlatformPkg |
+| ArmVirtPkg |
+| CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
+| DynamicTablesPkg |
+| EmbeddedPkg |
+| EmulatorPkg |
+| FatPkg | :heavy_check_mark: | :heavy_check_mark: |
+| FmpDevicePkg | :heavy_check_mark: | :heavy_check_mark: |
+| IntelFsp2Pkg |
+| IntelFsp2WrapperPkg |
+| MdeModulePkg | :heavy_check_mark: | :heavy_check_mark: | DxeIpl dependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit mode
+| MdePkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
+| NetworkPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
+| OvmfPkg |
+| PcAtChipsetPkg | :heavy_check_mark: | :heavy_check_mark: |
+| SecurityPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
+| ShellPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode, 3 modules are not being built by DSC
+| SignedCapsulePkg |
+| SourceLevelDebugPkg |
+| StandaloneMmPkg |
+| UefiCpuPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode, 2 binary modules not being built by DSC
+| UefiPayloadPkg |
+
+For more detailed status look at the test results of the latest CI run on the
+repo readme.
+
+## Background
+
+This Continuous integration and testing infrastructure leverages the TianoCore EDKII Tools PIP modules:
+[library](https://pypi.org/project/edk2-pytool-library/) and
+[extensions](https://pypi.org/project/edk2-pytool-extensions/) (with repos
+located [here](https://github.com/tianocore/edk2-pytool-library) and
+[here](https://github.com/tianocore/edk2-pytool-extensions)).
+
+The primary execution flows can be found in the
+`.azurepipelines/Windows-VS2019.yml` and `.azurepipelines/Ubuntu-GCC5.yml`
+files. These YAML files are consumed by the Azure Dev Ops Build Pipeline and
+dictate what server resources should be used, how they should be configured, and
+what processes should be run on them. An overview of this schema can be found
+[here](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema).
+
+Inspection of these files reveals the EDKII Tools commands that make up the
+primary processes for the CI build: 'stuart_setup', 'stuart_update', and
+'stuart_ci_build'. These commands come from the EDKII Tools PIP modules and are
+configured as described below. More documentation on the tools can be
+found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md)
+and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_invocables.md).
+
+## Configuration
+
+Configuration of the CI process consists of (in order of precedence):
+
+* command-line arguments passed in via the Pipeline YAML
+* a per-package configuration file (e.g. `<package-name>.ci.yaml`) that is
+ detected by the CI system in EDKII Tools.
+* a global configuration Python module (e.g. `CISetting.py`) passed in via the
+ command-line
+
+The global configuration file is described in
+[this readme](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_settings_manager.md)
+from the EDKII Tools documentation. This configuration is written as a Python
+module so that decisions can be made dynamically based on command line
+parameters and codebase state.
+
+The per-package configuration file can override most settings in the global
+configuration file, but is not dynamic. This file can be used to skip or
+customize tests that may be incompatible with a specific package. Each test generally requires
+per package configuration which comes from this file.
+
+## Running CI locally
+
+The EDKII Tools environment (and by extension the ci) is designed to support
+easily and consistantly running locally and in a cloud ci environment. To do
+that a few steps should be followed. Details of EDKII Tools can be found in the
+[docs folder here](https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs)
+
+### Prerequisets
+
+1. A supported toolchain (others might work but this is what is tested and validated)
+ * Windows 10:
+ * VS 2017 or VS 2019
+ * Windows SDK (for rc)
+ * Windows WDK (for capsules)
+ * Ubuntu 16.04
+ * GCC5
+ * Easy to add more but this is the current state
+2. Python 3.7.x or newer on path
+3. git on path
+4. Recommended to setup and activate a python virtual environment
+5. Install the requirements `pip install --upgrade pip_requirements.txt`
+
+### Running CI
+
+1. clone your edk2 repo
+2. Activate your python virtual environment in cmd window
+3. Get code dependencies (done only when submodules change)
+ * `stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=<your tag here>`
+4. Update other dependencies (done more often)
+ * `stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=<your tag here>`
+5. Run CI build (--help will give you options)
+ * `stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=<your tag here>`
+ * -p <pkg1,pkg2,pkg3> : To build only certain packages use a CSV list
+ * -a <arch1,arch2,arch3>: To run only certain architectures use a CSV list
+ * -t <target1,target2>: To run only tests related to certain targets use a
+ CSV list
+ * By default all tests are opted in. Then given a package.ci.yaml file those
+ tests can be configured for a package. Finally setting the check to the
+ value `skip` will skip that plugin. Examples:
+ * `CompilerPlugin=skip` skip the build test
+ * `GuidCheck=skip` skip the Guid check
+ * `SpellCheck=skip` skip the spell checker
+ * etc
+6. Detailed reports and logs per package are captured in the `Build` directory
+
+## Current PyTool Test Capabilities
+
+All CI tests are instances of EDKII Tools plugins. Documentation on the plugin
+system can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_plugin_manager.md)
+and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_plugin_manager.md).
+Upon invocation, each plugin will be passed the path to the current package
+under test and a dictionary containing its targeted configuration, as assembled
+from the command line, per-package configuration, and global configuration.
+
+Note: CI plugins are considered unique from build plugins and helper plugins,
+even though some CI plugins may execute steps of a build.
+
+In the example, these plugins live alongside the code under test (in the
+`.pytool/Plugin` directory), but may be moved to the 'edk2-test' repo if that
+location makes more sense for the community.
+
+### Module Inclusion Test - DscCompleteCheck
+
+This test scans all available modules (via INF files) and compares them to the
+package-level DSC file for the package each module is contained within. The test
+considers it an error if any module does not appear in the `Components` section
+of at least one package-level DSC (indicating that it would not be built if the
+package were built).
+
+### Code Compilation Test - CompilerPlugin
+
+Once the Module Inclusion Test has verified that all modules would be built if
+all package-level DSCs were built, the Code Compilation Test simply runs through
+and builds every package-level DSC on every toolchain and for every architecture
+that is supported. Any module that fails to build is considered an error.
+
+### GUID Uniqueness Test - GuidCheck
+
+This test works on the collection of all packages rather than an individual
+package. It looks at all FILE_GUIDs and GUIDs declared in DEC files and ensures
+that they are unique for the codebase. This prevents, for example, accidental
+duplication of GUIDs when using an existing INF as a template for a new module.
+
+### Cross-Package Dependency Test - DependencyCheck
+
+This test compares the list of all packages used in INFs files for a given
+package against a list of "allowed dependencies" in plugin configuration for
+that package. Any module that depends on a disallowed package will cause a test
+failure.
+
+### Library Declaration Test - LibraryClassCheck
+
+This test scans at all library header files found in the `Library` folders in
+all of the package's declared include directories and ensures that all files
+have a matching LibraryClass declaration in the DEC file for the package. Any
+missing declarations will cause a failure.
+
+### Invalid Character Test - CharEncodingCheck
+
+This test scans all files in a package to make sure that there are no invalid
+Unicode characters that may cause build errors in some character
+sets/localizations.
+
+### Spell Checking - cspell
+
+This test runs a spell checker on all files within the package. This is done
+using the NodeJs cspell tool. For details check `.pytool/Plugin/SpellCheck`.
+For this plugin to run during ci you must install nodejs and cspell and have
+both available to the command line when running your CI.
+
+Install
+
+* Install nodejs from https://nodejs.org/en/
+* Install cspell
+ 1. Open cmd prompt with access to node and npm
+ 2. Run `npm install -g cspell`
+
+ More cspell info: https://github.com/streetsidesoftware/cspell
+
+## PyTool Scopes
+
+Scopes are how the PyTool ext_dep, path_env, and plugins are activated. Meaning
+that if an invocable process has a scope active then those ext_dep and path_env
+will be active. To allow easy integration of PyTools capabilities there are a
+few standard scopes.
+
+| Scope | Invocable | Description |
+| :---- | :----- | :---- |
+| global | edk2_invocable++ - should be base_abstract_invocable | Running an invocables |
+| global-win | edk2_invocable++ | Running on Microsoft Windows |
+| global-nix | edk2_invocable++ | Running on Linux based OS |
+| edk2-build | | This indicates that an invocable is building EDK2 based UEFI code |
+| cibuild | set in .pytool/CISettings.py | Suggested target for edk2 continuous integration builds. Tools used for CiBuilds can use this scope. Example: asl compiler |
+
+## Future investments
+
+* PatchCheck tests as plugins
+* MacOS/xcode support
+* Clang/LLVM support
+* Visual Studio AARCH64 and ARM support
+* BaseTools C tools CI/PR and binary release process
+* BaseTools Python tools CI/PR process
+* Host based unit testing
+* Extensible private/closed source platform reporting
+* Platform builds, validation
+* UEFI SCTs
+* Other automation
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 20/22] .azurepipelines: Add Azure Pipelines YML configuration files
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (18 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 19/22] .pytool: Add CISettings.py and Readme.md Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 8:41 ` [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file Michael D Kinney
2019-11-11 8:41 ` [Patch v5 22/22] Readme.md: Add CI build status badges Michael D Kinney
21 siblings, 0 replies; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bret Barkelew, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
Add YML configuration files used to run the EDK II Continuous
Integration (CI) checks on Azure Pipelines agents.
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
.azurepipelines/Ubuntu-GCC5.yml | 18 +++
.azurepipelines/Ubuntu-PatchCheck.yml | 35 +++++
.azurepipelines/Windows-VS2019.yml | 18 +++
.../templates/basetools-build-steps.yml | 37 +++++
.../templates/pr-gate-build-job.yml | 80 +++++++++++
.azurepipelines/templates/pr-gate-steps.yml | 130 ++++++++++++++++++
.../templates/spell-check-prereq-steps.yml | 22 +++
7 files changed, 340 insertions(+)
create mode 100644 .azurepipelines/Ubuntu-GCC5.yml
create mode 100644 .azurepipelines/Ubuntu-PatchCheck.yml
create mode 100644 .azurepipelines/Windows-VS2019.yml
create mode 100644 .azurepipelines/templates/basetools-build-steps.yml
create mode 100644 .azurepipelines/templates/pr-gate-build-job.yml
create mode 100644 .azurepipelines/templates/pr-gate-steps.yml
create mode 100644 .azurepipelines/templates/spell-check-prereq-steps.yml
diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
new file mode 100644
index 0000000000..16d4e74158
--- /dev/null
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -0,0 +1,18 @@
+## @file
+# Azure Pipeline build file for a build using ubuntu and GCC5
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+- master
+pr:
+- master
+
+jobs:
+- template: templates/pr-gate-build-job.yml
+ parameters:
+ tool_chain_tag: 'GCC5'
+ vm_image: 'ubuntu-latest'
+ arch_list: "IA32,X64,ARM,AARCH64"
+
diff --git a/.azurepipelines/Ubuntu-PatchCheck.yml b/.azurepipelines/Ubuntu-PatchCheck.yml
new file mode 100644
index 0000000000..dff8f579b6
--- /dev/null
+++ b/.azurepipelines/Ubuntu-PatchCheck.yml
@@ -0,0 +1,35 @@
+## @file
+# Azure Pipielines YML file that evalues the patch series in a PR using the
+# python script BaseTools/Scripts/PatchCheck.py.
+#
+# NOTE: This example monitors pull requests against the edk2-ci branch. Most
+# environments would replace 'edk2-ci' with 'master'.
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+# https://github.com/tianocore
+#
+##
+
+trigger: none
+
+pr:
+- master
+
+pool:
+ vmImage: 'ubuntu-latest'
+
+steps:
+- checkout: self
+ clean: true
+
+- task: UsePythonVersion@0
+ inputs:
+ versionSpec: '3.7.x'
+ architecture: 'x64'
+
+- script: |
+ git fetch origin $(System.PullRequest.TargetBranch):$(System.PullRequest.TargetBranch)
+ python BaseTools/Scripts/PatchCheck.py $(System.PullRequest.TargetBranch)..$(System.PullRequest.SourceCommitId)
+ displayName: 'Use PatchCheck.py to verify patch series in pull request'
diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml
new file mode 100644
index 0000000000..4a5262f3ff
--- /dev/null
+++ b/.azurepipelines/Windows-VS2019.yml
@@ -0,0 +1,18 @@
+## @file
+# Azure Pipeline build file for a build using Windows and VS2019
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+- master
+
+pr:
+- master
+
+jobs:
+- template: templates/pr-gate-build-job.yml
+ parameters:
+ tool_chain_tag: 'VS2019'
+ vm_image: 'windows-latest'
+ arch_list: "IA32,X64"
diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml
new file mode 100644
index 0000000000..05e12c9565
--- /dev/null
+++ b/.azurepipelines/templates/basetools-build-steps.yml
@@ -0,0 +1,37 @@
+## @file
+# File templates/basetools-build-job.yml
+#
+# template file to build basetools
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+parameters:
+ tool_chain_tag: ''
+
+steps:
+- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}:
+ - bash: sudo apt-get update
+ displayName: Update apt
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+ - bash: sudo apt-get install gcc g++ make uuid-dev
+ displayName: Install required tools
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+- task: CmdLine@1
+ displayName: Build Base Tools from source
+ inputs:
+ filename: python
+ arguments: BaseTools/Edk2ToolsBuild.py -t ${{ parameters.tool_chain_tag }}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+- task: CopyFiles@2
+ displayName: "Copy base tools build log"
+ inputs:
+ targetFolder: '$(Build.ArtifactStagingDirectory)'
+ SourceFolder: 'BaseTools/BaseToolsBuild'
+ contents: |
+ BASETOOLS_BUILD*.*
+ flattenFolders: true
+ condition: and(gt(variables.pkg_count, 0), succeededOrFailed())
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
new file mode 100644
index 0000000000..bc30628796
--- /dev/null
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -0,0 +1,80 @@
+## @file
+# File templates/pr-gate-build-job.yml
+#
+# template file used to build supported packages.
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+ tool_chain_tag: ''
+ vm_image: ''
+ arch_list: ''
+
+# Build step
+jobs:
+
+- job: Build_${{ parameters.tool_chain_tag }}
+
+ #Use matrix to speed up the build process
+ strategy:
+ matrix:
+ TARGET_MDE_CPU:
+ Build.Pkgs: 'MdePkg,UefiCpuPkg'
+ Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+ TARGET_MDEMODULE_DEBUG:
+ Build.Pkgs: 'MdeModulePkg'
+ Build.Targets: 'DEBUG'
+ TARGET_MDEMODULE_RELEASE:
+ Build.Pkgs: 'MdeModulePkg'
+ Build.Targets: 'RELEASE,NO-TARGET'
+ TARGET_NETWORK:
+ Build.Pkgs: 'NetworkPkg'
+ Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+ TARGET_OTHER:
+ Build.Pkgs: 'PcAtChipsetPkg,ShellPkg'
+ Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+ TARGET_FMP:
+ Build.Pkgs: 'FmpDevicePkg,FatPkg'
+ Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+ TARGET_CRYPTO:
+ Build.Pkgs: 'CryptoPkg'
+ Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+ TARGET_SECURITY:
+ Build.Pkgs: 'SecurityPkg'
+ Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+
+ workspace:
+ clean: all
+
+ pool:
+ vmImage: ${{ parameters.vm_image }}
+
+ steps:
+ - template: pr-gate-steps.yml
+ parameters:
+ tool_chain_tag: ${{ parameters.tool_chain_tag }}
+ build_pkgs: $(Build.Pkgs)
+ build_targets: $(Build.Targets)
+ build_archs: ${{ parameters.arch_list }}
+
+- job: FINISHED
+ dependsOn: Build_${{ parameters.tool_chain_tag }}
+ condition: succeeded()
+ steps:
+ - checkout: none
+ - script: |
+ echo FINISHED
+ sleep 10
+ displayName: FINISHED
+
+- job: FAILED
+ dependsOn: Build_${{ parameters.tool_chain_tag }}
+ condition: failed()
+ steps:
+ - checkout: none
+ - script: |
+ echo FAILED
+ sleep 10
+ displayName: FAILED
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
new file mode 100644
index 0000000000..945053c337
--- /dev/null
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -0,0 +1,130 @@
+## @file
+# File templates/pr-gate-steps.yml
+#
+# template file containing the steps to build
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+ tool_chain_tag: ''
+ build_pkgs: ''
+ build_targets: ''
+ build_archs: ''
+
+steps:
+- checkout: self
+ clean: true
+ fetchDepth: 1
+
+- task: UsePythonVersion@0
+ inputs:
+ versionSpec: '3.7.x'
+ architecture: 'x64'
+
+- script: pip install -r pip_requirements.txt --upgrade
+ displayName: 'Install/Upgrade pip modules'
+
+# Set default
+- bash: |
+ echo "##vso[task.setvariable variable=pkgs_to_build]${{ parameters.build_pkgs }}"
+ echo "##vso[task.setvariable variable=pkg_count]${{ 1 }}"
+
+# trim the package list if this is a PR
+- task: CmdLine@1
+ displayName: Check if ${{ parameters.build_pkgs }} need testing
+ inputs:
+ filename: stuart_pr_eval
+ arguments: -c .pytool/CISettings.py -p ${{ parameters.build_pkgs }} --pr-target origin/$(System.PullRequest.targetBranch) --output-csv-format-string "##vso[task.setvariable variable=pkgs_to_build;isOutpout=true]{pkgcsv}" --output-count-format-string "##vso[task.setvariable variable=pkg_count;isOutpout=true]{pkgcount}"
+ condition: eq(variables['Build.Reason'], 'PullRequest')
+
+# build basetools
+- template: basetools-build-steps.yml
+ parameters:
+ tool_chain_tag: ${{ parameters.tool_chain_tag }}
+
+# install spell check prereqs
+- template: spell-check-prereq-steps.yml
+
+# Build repo
+- task: CmdLine@1
+ displayName: Setup ${{ parameters.build_pkgs }} ${{ parameters.build_archs}}
+ inputs:
+ filename: stuart_setup
+ arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+- task: CmdLine@1
+ displayName: Update ${{ parameters.build_pkgs }} ${{ parameters.build_archs}}
+ inputs:
+ filename: stuart_update
+ arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+- task: CmdLine@1
+ displayName: Build and Test ${{ parameters.build_pkgs }} ${{ parameters.build_archs}}
+ inputs:
+ filename: stuart_ci_build
+ arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# Publish Test Results to Azure Pipelines/TFS
+- task: PublishTestResults@2
+ displayName: 'Publish junit test results'
+ continueOnError: true
+ condition: and( succeededOrFailed(),gt(variables.pkg_count, 0))
+ inputs:
+ testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
+ testResultsFiles: 'Build/TestSuites.xml'
+ #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
+ mergeTestResults: true # Optional
+ testRunTitle: $(System.JobName) # Optional
+ #buildPlatform: # Optional
+ #buildConfiguration: # Optional
+ publishRunAttachments: true # Optional
+
+# Publish Test Results to Azure Pipelines/TFS
+- task: PublishTestResults@2
+ displayName: 'Publish host based test results for $(System.JobName)'
+ continueOnError: true
+ condition: and( succeededOrFailed(), gt(variables.pkg_count, 0))
+ inputs:
+ testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
+ testResultsFiles: 'Build/**/*.result.xml'
+ #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
+ mergeTestResults: false # Optional
+ testRunTitle: ${{ parameters.build_pkgs }} # Optional
+ #buildPlatform: # Optional
+ #buildConfiguration: # Optional
+ publishRunAttachments: true # Optional
+
+# Copy the build logs to the artifact staging directory
+- task: CopyFiles@2
+ displayName: "Copy build logs"
+ inputs:
+ targetFolder: '$(Build.ArtifactStagingDirectory)'
+ SourceFolder: 'Build'
+ contents: |
+ BUILDLOG_*.txt
+ BUILDLOG_*.md
+ CI_*.txt
+ CI_*.md
+ CISETUP.txt
+ SETUPLOG.txt
+ UPDATE_LOG.txt
+ PREVALLOG.txt
+ TestSuites.xml
+ **/BUILD_TOOLS_REPORT.html
+ **/OVERRIDELOG.TXT
+ flattenFolders: true
+ condition: succeededOrFailed()
+
+# Publish build artifacts to Azure Artifacts/TFS or a file share
+- task: PublishBuildArtifacts@1
+ continueOnError: true
+ displayName: "Publish build logs"
+ inputs:
+ pathtoPublish: '$(Build.ArtifactStagingDirectory)'
+ artifactName: 'Build Logs $(System.JobName)'
+ condition: succeededOrFailed()
diff --git a/.azurepipelines/templates/spell-check-prereq-steps.yml b/.azurepipelines/templates/spell-check-prereq-steps.yml
new file mode 100644
index 0000000000..74d5eda2e7
--- /dev/null
+++ b/.azurepipelines/templates/spell-check-prereq-steps.yml
@@ -0,0 +1,22 @@
+## @file
+# File templates/spell-check-prereq-steps.yml
+#
+# template file used to install spell checking prerequisits
+#
+# Copyright (c) 2019, Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+ none: ''
+
+steps:
+- task: NodeTool@0
+ inputs:
+ versionSpec: '10.x'
+ #checkLatest: false # Optional
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+- script: npm install -g cspell
+ displayName: 'Install cspell npm'
+ condition: and(gt(variables.pkg_count, 0), succeeded())
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (19 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 20/22] .azurepipelines: Add Azure Pipelines YML configuration files Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2020-01-17 19:44 ` [edk2-devel] " Leif Lindholm
2019-11-11 8:41 ` [Patch v5 22/22] Readme.md: Add CI build status badges Michael D Kinney
21 siblings, 1 reply; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Liming Gao, Sean Brogan, Bret Barkelew
Add directory for the Mergify YML configuration files that
provides rules and actions used to process a pull request.
* Auto commit a PR from EDK II Maintainer with 'push' label
set and all CI checks pass
* Auto close a PR from any developers without 'push' label
set and all CI checks pass.
* Auto close a PR from a non EDK II Maintainer that has
the 'push' label set.
* Post a comment to a PR that has a merge conflict.
Submitter can resolved conflicts and reopen the PR.
* Post a comment to a PR that fails PatchCheck.py
Submitter can resolve PatchCheck.py issues and
reopen the PR.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
.mergify/config.yml | 97 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 97 insertions(+)
create mode 100644 .mergify/config.yml
diff --git a/.mergify/config.yml b/.mergify/config.yml
new file mode 100644
index 0000000000..26583de0f6
--- /dev/null
+++ b/.mergify/config.yml
@@ -0,0 +1,97 @@
+## @file
+# Mergify YML file that automatically merges a GitHub pull request against
+# edk2-ci if all of the GitHub branch protections have passed. It also
+# contains rules to:
+# * auto close branches that are not from an EDK II Maintainer
+# * post a comment on pull requests that have merge conflicts.
+# * post a comment on pull requests that have PatchCheck.py errors.
+#
+# Configuration Notes:
+# * Update the 'base=edk2-ci' statements with the name of the branch to merge
+# pull requests.
+#
+# * Update the 'status-failure' statement with the name of the name of the Azure
+# Pipelines Build that performs the EDK II Maintainer check.
+#
+# * This file must be checked into the 'default' branch of a repo. Copies
+# of this file on other branches of a repo are ignored by Mergify.
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+# https://github.com/apps/mergify
+# https://doc.mergify.io/
+#
+##
+
+pull_request_rules:
+
+ - name: Automatically merge a PR when all required checks pass and 'push' label is present
+ conditions:
+ - base=master
+ - label=push
+ - author=@tianocore/edk-ii-maintainers
+ - status-success=tianocore.PatchCheck
+ - status-success=Ubuntu GCC5 PR
+ - status-success=Windows VS2019 PR
+ actions:
+ merge:
+ strict: true
+ method: rebase
+
+ - name: Automatically close a PR when all required checks pass and 'push' label is not present
+ conditions:
+ - base=master
+ - -label=push
+ - -closed
+ - status-success=tianocore.PatchCheck
+ - status-success=Ubuntu GCC5 PR
+ - status-success=Windows VS2019 PR
+ - status-success=Ubuntu GCC5 PR (FINISHED)
+ - status-success=Windows VS2019 PR (FINISHED)
+ actions:
+ close:
+ message: All checks passed. Auto close personal build.
+
+ - name: Post a comment on a PR that can not be merged due to a merge conflict
+ conditions:
+ - base=master
+ - conflict
+ actions:
+ comment:
+ message: PR can not be merged due to conflict. Please rebase and resubmit
+
+ - name: Automatically close a PR that fails the EDK II Maintainers membership check and 'push' label is present
+ conditions:
+ - base=master
+ - label=push
+ - -author=@tianocore/edk-ii-maintainers
+ actions:
+ close:
+ message: PR submitter is not a member of the Tianocore EDK II Maintainers team
+
+ - name: Post a comment on a PR if PatchCheck fails
+ conditions:
+ - base=master
+ - status-failure=tianocore.PatchCheck
+ actions:
+ comment:
+ message: PR can not be merged due to a PatchCheck failure. Please resolve and resubmit
+
+ - name: Post a comment on a PR if Ubuntu GCC5 fails
+ conditions:
+ - base=master
+ - status-failure=Ubuntu GCC5 PR
+ - status-success=Ubuntu GCC5 PR (FAILED)
+ actions:
+ comment:
+ message: PR can not be merged due to an Ubuntu GCC5 failure. Please resolve and resubmit
+
+ - name: Post a comment on a PR if Windows VS2019 fails
+ conditions:
+ - base=master
+ - status-failure=Windows VS2019 PR
+ - status-success=Windows VS2019 PR (FAILED)
+ actions:
+ comment:
+ message: PR can not be merged due to a Windows VS2019 failure. Please resolve and resubmit
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Patch v5 22/22] Readme.md: Add CI build status badges
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
` (20 preceding siblings ...)
2019-11-11 8:41 ` [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file Michael D Kinney
@ 2019-11-11 8:41 ` Michael D Kinney
2019-11-11 17:14 ` Leif Lindholm
21 siblings, 1 reply; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 8:41 UTC (permalink / raw)
To: devel; +Cc: Andrew Fish, Laszlo Ersek, Leif Lindholm, Liming Gao
Update Readme.md with status badges from the most recent
EDK II Continuous Integration(CI) build.
Cc: Andrew Fish <afish@apple.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
Readme.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 51 insertions(+)
diff --git a/Readme.md b/Readme.md
index 27e4ce0771..c22cac9748 100644
--- a/Readme.md
+++ b/Readme.md
@@ -3,6 +3,57 @@
A modern, feature-rich, cross-platform firmware development environment
for the UEFI and PI specifications from www.uefi.org.
+# Build Status
+
+<table>
+ <tr>
+ <th>Host Type</th>
+ <th>Toolchain</th>
+ <th>Branch</th>
+ <th>Build Status</th>
+ <th>Test Status</th>
+ <th>Code Coverage</th>
+ </tr>
+ <tr>
+ <td>Windows</td>
+ <td>VS2019</td>
+ <td>edk2-ci</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=27&branchName=edk2-ci">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/edk2-staging/Windows%20VS2019%20CI?branchName=edk2-ci"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=27&branchName=edk2-ci">
+ <img src="https://img.shields.io/azure-devops/tests/tianocore/edk2-ci-play/27.svg"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=27&branchName=edk2-ci">
+ <img src="https://img.shields.io/badge/coverage-coming_soon-blue"/></a>
+ </td>
+ </tr>
+ <tr>
+ <td>Ubuntu</td>
+ <td>GCC</td>
+ <td>edk2-ci</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=26&branchName=edk2-ci">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/edk2-staging/Ubuntu%20GCC5%20CI?branchName=edk2-ci"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=26&branchName=edk2-ci">
+ <img src="https://img.shields.io/azure-devops/tests/tianocore/edk2-ci-play/26.svg"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=26&branchName=edk2-ci">
+ <img src="https://img.shields.io/badge/coverage-coming_soon-blue"/></a>
+ </td>
+ </tr>
+</table>
+
+[More CI Build information](.pytool/Readme.md)
+
+# License Details
+
The majority of the content in the EDK II open source project uses a
[BSD-2-Clause Plus Patent License](License.txt). The EDK II open source project
contains the following components that are covered by additional licenses:
--
2.21.0.windows.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file
2019-11-11 8:40 ` [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file Michael D Kinney
@ 2019-11-11 9:58 ` Laszlo Ersek
2019-11-11 16:35 ` Michael D Kinney
0 siblings, 1 reply; 30+ messages in thread
From: Laszlo Ersek @ 2019-11-11 9:58 UTC (permalink / raw)
To: Michael D Kinney, devel
Cc: Sean Brogan, Andrew Fish, Leif Lindholm, Liming Gao
Hi Mike,
On 11/11/19 09:40, Michael D Kinney wrote:
> From: Sean Brogan <sean.brogan@microsoft.com>
>
> Add pip requirements file that is used to install the
> python pip modules build from the edk2-pytool-library and
> edk2-pytool-extensions repositories.
>
> These python modules provide the extensions required to
> perform EDK II Continuous Integration(CI) builds.
>
> Cc: Andrew Fish <afish@apple.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Leif Lindholm <leif.lindholm@linaro.org>
> Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
> Reviewed-by: Liming Gao <liming.gao@intel.com>
> ---
> pip_requirements.txt | 17 +++++++++++++++++
> 1 file changed, 17 insertions(+)
> create mode 100644 pip_requirements.txt
>
> diff --git a/pip_requirements.txt b/pip_requirements.txt
> new file mode 100644
> index 0000000000..4ad72cfc98
> --- /dev/null
> +++ b/pip_requirements.txt
Is the underscore (ASCII 0x5F) a typo / misunderstanding, or is it
intentional?
I never tested "pip_requirements.txt"; I tested "pip-requirements.txt"
(hyphen, ASCII 0x2D).
Thanks
Laszlo
> @@ -0,0 +1,17 @@
> +## @file
> +# EDK II Python PIP requirements file
> +#
> +# This file provides the list of python components to install using PIP.
> +#
> +# Copyright (c) Microsoft Corporation. All rights reserved.<BR>
> +#
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +#
> +# https://pypi.org/project/pip/
> +# https://pip.pypa.io/en/stable/user_guide/#requirements-files
> +# https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format
> +#
> +##
> +
> +edk2-pytool-library==0.10.*
> +edk2-pytool-extensions==0.12.*
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file
2019-11-11 9:58 ` Laszlo Ersek
@ 2019-11-11 16:35 ` Michael D Kinney
2019-11-11 16:41 ` Laszlo Ersek
0 siblings, 1 reply; 30+ messages in thread
From: Michael D Kinney @ 2019-11-11 16:35 UTC (permalink / raw)
To: Laszlo Ersek, devel@edk2.groups.io, Kinney, Michael D
Cc: Sean Brogan, Andrew Fish, Leif Lindholm, Gao, Liming
Hi Laszlo,
I will change to pip-requirements.txt.
Mike
> -----Original Message-----
> From: Laszlo Ersek <lersek@redhat.com>
> Sent: Monday, November 11, 2019 1:58 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io
> Cc: Sean Brogan <sean.brogan@microsoft.com>; Andrew Fish
> <afish@apple.com>; Leif Lindholm
> <leif.lindholm@linaro.org>; Gao, Liming
> <liming.gao@intel.com>
> Subject: Re: [Patch v5 03/22] pip_requirements.txt: Add
> python pip requirements file
>
> Hi Mike,
>
> On 11/11/19 09:40, Michael D Kinney wrote:
> > From: Sean Brogan <sean.brogan@microsoft.com>
> >
> > Add pip requirements file that is used to install the
> python pip
> > modules build from the edk2-pytool-library and edk2-
> pytool-extensions
> > repositories.
> >
> > These python modules provide the extensions required
> to perform EDK II
> > Continuous Integration(CI) builds.
> >
> > Cc: Andrew Fish <afish@apple.com>
> > Cc: Laszlo Ersek <lersek@redhat.com>
> > Cc: Leif Lindholm <leif.lindholm@linaro.org>
> > Signed-off-by: Michael D Kinney
> <michael.d.kinney@intel.com>
> > Reviewed-by: Liming Gao <liming.gao@intel.com>
> > ---
> > pip_requirements.txt | 17 +++++++++++++++++
> > 1 file changed, 17 insertions(+)
> > create mode 100644 pip_requirements.txt
> >
> > diff --git a/pip_requirements.txt
> b/pip_requirements.txt new file mode
> > 100644 index 0000000000..4ad72cfc98
> > --- /dev/null
> > +++ b/pip_requirements.txt
>
> Is the underscore (ASCII 0x5F) a typo /
> misunderstanding, or is it
> intentional?
>
> I never tested "pip_requirements.txt"; I tested "pip-
> requirements.txt"
> (hyphen, ASCII 0x2D).
>
> Thanks
> Laszlo
>
> > @@ -0,0 +1,17 @@
> > +## @file
> > +# EDK II Python PIP requirements file
> > +#
> > +# This file provides the list of python components to
> install using PIP.
> > +#
> > +# Copyright (c) Microsoft Corporation. All rights
> reserved.<BR>
> > +#
> > +# SPDX-License-Identifier: BSD-2-Clause-Patent
> > +#
> > +# https://pypi.org/project/pip/
> > +#
> https://pip.pypa.io/en/stable/user_guide/#requirements-
> files
> > +#
> https://pip.pypa.io/en/stable/reference/pip_install/#req
> uirements-file-format
> > +#
> > +##
> > +
> > +edk2-pytool-library==0.10.*
> > +edk2-pytool-extensions==0.12.*
> >
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file
2019-11-11 16:35 ` Michael D Kinney
@ 2019-11-11 16:41 ` Laszlo Ersek
0 siblings, 0 replies; 30+ messages in thread
From: Laszlo Ersek @ 2019-11-11 16:41 UTC (permalink / raw)
To: Kinney, Michael D, devel@edk2.groups.io
Cc: Sean Brogan, Andrew Fish, Leif Lindholm, Gao, Liming
On 11/11/19 17:35, Kinney, Michael D wrote:
> Hi Laszlo,
>
> I will change to pip-requirements.txt.
Please update both the subject line and the actual filename.
With that:
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Thanks!
Laszlo
>
> Mike
>
>> -----Original Message-----
>> From: Laszlo Ersek <lersek@redhat.com>
>> Sent: Monday, November 11, 2019 1:58 AM
>> To: Kinney, Michael D <michael.d.kinney@intel.com>;
>> devel@edk2.groups.io
>> Cc: Sean Brogan <sean.brogan@microsoft.com>; Andrew Fish
>> <afish@apple.com>; Leif Lindholm
>> <leif.lindholm@linaro.org>; Gao, Liming
>> <liming.gao@intel.com>
>> Subject: Re: [Patch v5 03/22] pip_requirements.txt: Add
>> python pip requirements file
>>
>> Hi Mike,
>>
>> On 11/11/19 09:40, Michael D Kinney wrote:
>>> From: Sean Brogan <sean.brogan@microsoft.com>
>>>
>>> Add pip requirements file that is used to install the
>> python pip
>>> modules build from the edk2-pytool-library and edk2-
>> pytool-extensions
>>> repositories.
>>>
>>> These python modules provide the extensions required
>> to perform EDK II
>>> Continuous Integration(CI) builds.
>>>
>>> Cc: Andrew Fish <afish@apple.com>
>>> Cc: Laszlo Ersek <lersek@redhat.com>
>>> Cc: Leif Lindholm <leif.lindholm@linaro.org>
>>> Signed-off-by: Michael D Kinney
>> <michael.d.kinney@intel.com>
>>> Reviewed-by: Liming Gao <liming.gao@intel.com>
>>> ---
>>> pip_requirements.txt | 17 +++++++++++++++++
>>> 1 file changed, 17 insertions(+)
>>> create mode 100644 pip_requirements.txt
>>>
>>> diff --git a/pip_requirements.txt
>> b/pip_requirements.txt new file mode
>>> 100644 index 0000000000..4ad72cfc98
>>> --- /dev/null
>>> +++ b/pip_requirements.txt
>>
>> Is the underscore (ASCII 0x5F) a typo /
>> misunderstanding, or is it
>> intentional?
>>
>> I never tested "pip_requirements.txt"; I tested "pip-
>> requirements.txt"
>> (hyphen, ASCII 0x2D).
>>
>> Thanks
>> Laszlo
>>
>>> @@ -0,0 +1,17 @@
>>> +## @file
>>> +# EDK II Python PIP requirements file
>>> +#
>>> +# This file provides the list of python components to
>> install using PIP.
>>> +#
>>> +# Copyright (c) Microsoft Corporation. All rights
>> reserved.<BR>
>>> +#
>>> +# SPDX-License-Identifier: BSD-2-Clause-Patent
>>> +#
>>> +# https://pypi.org/project/pip/
>>> +#
>> https://pip.pypa.io/en/stable/user_guide/#requirements-
>> files
>>> +#
>> https://pip.pypa.io/en/stable/reference/pip_install/#req
>> uirements-file-format
>>> +#
>>> +##
>>> +
>>> +edk2-pytool-library==0.10.*
>>> +edk2-pytool-extensions==0.12.*
>>>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Patch v5 22/22] Readme.md: Add CI build status badges
2019-11-11 8:41 ` [Patch v5 22/22] Readme.md: Add CI build status badges Michael D Kinney
@ 2019-11-11 17:14 ` Leif Lindholm
0 siblings, 0 replies; 30+ messages in thread
From: Leif Lindholm @ 2019-11-11 17:14 UTC (permalink / raw)
To: Kinney, Michael D
Cc: edk2-devel-groups-io, Andrew Fish, Laszlo Ersek, Liming Gao
[-- Attachment #1: Type: text/plain, Size: 3234 bytes --]
(on a phone)
This looks ... OK. I'm fine with this as a compromise.
Reviewed-by: me
On Mon, 11 Nov 2019, 08:41 Michael D Kinney, <michael.d.kinney@intel.com>
wrote:
> Update Readme.md with status badges from the most recent
> EDK II Continuous Integration(CI) build.
>
> Cc: Andrew Fish <afish@apple.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Leif Lindholm <leif.lindholm@linaro.org>
> Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
> Acked-by: Laszlo Ersek <lersek@redhat.com>
> Reviewed-by: Liming Gao <liming.gao@intel.com>
> ---
> Readme.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 51 insertions(+)
>
> diff --git a/Readme.md b/Readme.md
> index 27e4ce0771..c22cac9748 100644
> --- a/Readme.md
> +++ b/Readme.md
> @@ -3,6 +3,57 @@
> A modern, feature-rich, cross-platform firmware development environment
> for the UEFI and PI specifications from www.uefi.org.
>
> +# Build Status
> +
> +<table>
> + <tr>
> + <th>Host Type</th>
> + <th>Toolchain</th>
> + <th>Branch</th>
> + <th>Build Status</th>
> + <th>Test Status</th>
> + <th>Code Coverage</th>
> + </tr>
> + <tr>
> + <td>Windows</td>
> + <td>VS2019</td>
> + <td>edk2-ci</td>
> + <td>
> + <a href="
> https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=27&branchName=edk2-ci
> ">
> + <img src="
> https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/edk2-staging/Windows%20VS2019%20CI?branchName=edk2-ci
> "/></a>
> + </td>
> + <td>
> + <a href="
> https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=27&branchName=edk2-ci
> ">
> + <img src="
> https://img.shields.io/azure-devops/tests/tianocore/edk2-ci-play/27.svg
> "/></a>
> + </td>
> + <td>
> + <a href="
> https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=27&branchName=edk2-ci
> ">
> + <img src="https://img.shields.io/badge/coverage-coming_soon-blue
> "/></a>
> + </td>
> + </tr>
> + <tr>
> + <td>Ubuntu</td>
> + <td>GCC</td>
> + <td>edk2-ci</td>
> + <td>
> + <a href="
> https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=26&branchName=edk2-ci
> ">
> + <img src="
> https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/edk2-staging/Ubuntu%20GCC5%20CI?branchName=edk2-ci
> "/></a>
> + </td>
> + <td>
> + <a href="
> https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=26&branchName=edk2-ci
> ">
> + <img src="
> https://img.shields.io/azure-devops/tests/tianocore/edk2-ci-play/26.svg
> "/></a>
> + </td>
> + <td>
> + <a href="
> https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=26&branchName=edk2-ci
> ">
> + <img src="https://img.shields.io/badge/coverage-coming_soon-blue
> "/></a>
> + </td>
> + </tr>
> +</table>
> +
> +[More CI Build information](.pytool/Readme.md)
> +
> +# License Details
> +
> The majority of the content in the EDK II open source project uses a
> [BSD-2-Clause Plus Patent License](License.txt). The EDK II open source
> project
> contains the following components that are covered by additional licenses:
> --
> 2.21.0.windows.1
>
>
[-- Attachment #2: Type: text/html, Size: 6889 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [edk2-devel] [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file
2019-11-11 8:41 ` [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file Michael D Kinney
@ 2020-01-17 19:44 ` Leif Lindholm
2020-01-17 20:22 ` Michael D Kinney
0 siblings, 1 reply; 30+ messages in thread
From: Leif Lindholm @ 2020-01-17 19:44 UTC (permalink / raw)
To: michael.d.kinney; +Cc: devel
Hi Mike (on-list),
Embarassing to admit, but I hadn't tried to do my first mergify commit
until the Maintainers.txt right now, and that isn't working as I expected.
I tried to force a retry on https://github.com/tianocore/edk2/pull/307
(since I added the 'push' tag *after* submitting the github pull
request), but as you can see mergify says "@leiflindholm is not allowed
to run commands".
Am I missing something obvious?
Best Regards,
Leif
On Mon, Nov 11, 2019 at 00:41:06 -0800, Michael D Kinney wrote:
> Add directory for the Mergify YML configuration files that
> provides rules and actions used to process a pull request.
>
> * Auto commit a PR from EDK II Maintainer with 'push' label
> set and all CI checks pass
> * Auto close a PR from any developers without 'push' label
> set and all CI checks pass.
> * Auto close a PR from a non EDK II Maintainer that has
> the 'push' label set.
> * Post a comment to a PR that has a merge conflict.
> Submitter can resolved conflicts and reopen the PR.
> * Post a comment to a PR that fails PatchCheck.py
> Submitter can resolve PatchCheck.py issues and
> reopen the PR.
>
> Cc: Liming Gao <liming.gao@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
> Reviewed-by: Liming Gao <liming.gao@intel.com>
> ---
> .mergify/config.yml | 97 +++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 97 insertions(+)
> create mode 100644 .mergify/config.yml
>
> diff --git a/.mergify/config.yml b/.mergify/config.yml
> new file mode 100644
> index 0000000000..26583de0f6
> --- /dev/null
> +++ b/.mergify/config.yml
> @@ -0,0 +1,97 @@
> +## @file
> +# Mergify YML file that automatically merges a GitHub pull request against
> +# edk2-ci if all of the GitHub branch protections have passed. It also
> +# contains rules to:
> +# * auto close branches that are not from an EDK II Maintainer
> +# * post a comment on pull requests that have merge conflicts.
> +# * post a comment on pull requests that have PatchCheck.py errors.
> +#
> +# Configuration Notes:
> +# * Update the 'base=edk2-ci' statements with the name of the branch to merge
> +# pull requests.
> +#
> +# * Update the 'status-failure' statement with the name of the name of the Azure
> +# Pipelines Build that performs the EDK II Maintainer check.
> +#
> +# * This file must be checked into the 'default' branch of a repo. Copies
> +# of this file on other branches of a repo are ignored by Mergify.
> +#
> +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +#
> +# https://github.com/apps/mergify
> +# https://doc.mergify.io/
> +#
> +##
> +
> +pull_request_rules:
> +
> + - name: Automatically merge a PR when all required checks pass and 'push' label is present
> + conditions:
> + - base=master
> + - label=push
> + - author=@tianocore/edk-ii-maintainers
> + - status-success=tianocore.PatchCheck
> + - status-success=Ubuntu GCC5 PR
> + - status-success=Windows VS2019 PR
> + actions:
> + merge:
> + strict: true
> + method: rebase
> +
> + - name: Automatically close a PR when all required checks pass and 'push' label is not present
> + conditions:
> + - base=master
> + - -label=push
> + - -closed
> + - status-success=tianocore.PatchCheck
> + - status-success=Ubuntu GCC5 PR
> + - status-success=Windows VS2019 PR
> + - status-success=Ubuntu GCC5 PR (FINISHED)
> + - status-success=Windows VS2019 PR (FINISHED)
> + actions:
> + close:
> + message: All checks passed. Auto close personal build.
> +
> + - name: Post a comment on a PR that can not be merged due to a merge conflict
> + conditions:
> + - base=master
> + - conflict
> + actions:
> + comment:
> + message: PR can not be merged due to conflict. Please rebase and resubmit
> +
> + - name: Automatically close a PR that fails the EDK II Maintainers membership check and 'push' label is present
> + conditions:
> + - base=master
> + - label=push
> + - -author=@tianocore/edk-ii-maintainers
> + actions:
> + close:
> + message: PR submitter is not a member of the Tianocore EDK II Maintainers team
> +
> + - name: Post a comment on a PR if PatchCheck fails
> + conditions:
> + - base=master
> + - status-failure=tianocore.PatchCheck
> + actions:
> + comment:
> + message: PR can not be merged due to a PatchCheck failure. Please resolve and resubmit
> +
> + - name: Post a comment on a PR if Ubuntu GCC5 fails
> + conditions:
> + - base=master
> + - status-failure=Ubuntu GCC5 PR
> + - status-success=Ubuntu GCC5 PR (FAILED)
> + actions:
> + comment:
> + message: PR can not be merged due to an Ubuntu GCC5 failure. Please resolve and resubmit
> +
> + - name: Post a comment on a PR if Windows VS2019 fails
> + conditions:
> + - base=master
> + - status-failure=Windows VS2019 PR
> + - status-success=Windows VS2019 PR (FAILED)
> + actions:
> + comment:
> + message: PR can not be merged due to a Windows VS2019 failure. Please resolve and resubmit
> --
> 2.21.0.windows.1
>
>
>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [edk2-devel] [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file
2020-01-17 19:44 ` [edk2-devel] " Leif Lindholm
@ 2020-01-17 20:22 ` Michael D Kinney
2020-01-17 20:29 ` Leif Lindholm
0 siblings, 1 reply; 30+ messages in thread
From: Michael D Kinney @ 2020-01-17 20:22 UTC (permalink / raw)
To: Leif Lindholm, Kinney, Michael D; +Cc: devel@edk2.groups.io
Hi Leif,
I have not tried the mergify commands like that. I can investigate
further if we think we need them.
I saw that some of the Azure pipelines checks did not run. That
Is what was blocking Mergify. Mergify was waiting for status.
I did a Close and Re-open to re-trigger Azure Pipelines and that
worked. You changed have been committed.
We have seen a very rare condition in communication between
Azure Pipelines and GitHub. This close/re-open is what we
can try first. The 2nd approach is to abandon the current
PR and create a new PR with the same content. That has only
had to be used once since enabling in early November.
Mike
> -----Original Message-----
> From: Leif Lindholm <leif.lindholm@linaro.org>
> Sent: Friday, January 17, 2020 11:44 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: devel@edk2.groups.io
> Subject: Re: [edk2-devel] [Patch v5 21/22] .mergify:
> Add Mergify YML pull request rules configuration file
>
> Hi Mike (on-list),
>
> Embarassing to admit, but I hadn't tried to do my first
> mergify commit
> until the Maintainers.txt right now, and that isn't
> working as I expected.
>
> I tried to force a retry on
> https://github.com/tianocore/edk2/pull/307
> (since I added the 'push' tag *after* submitting the
> github pull
> request), but as you can see mergify says
> "@leiflindholm is not allowed
> to run commands".
>
> Am I missing something obvious?
>
> Best Regards,
>
> Leif
>
> On Mon, Nov 11, 2019 at 00:41:06 -0800, Michael D
> Kinney wrote:
> > Add directory for the Mergify YML configuration files
> that
> > provides rules and actions used to process a pull
> request.
> >
> > * Auto commit a PR from EDK II Maintainer with 'push'
> label
> > set and all CI checks pass
> > * Auto close a PR from any developers without 'push'
> label
> > set and all CI checks pass.
> > * Auto close a PR from a non EDK II Maintainer that
> has
> > the 'push' label set.
> > * Post a comment to a PR that has a merge conflict.
> > Submitter can resolved conflicts and reopen the PR.
> > * Post a comment to a PR that fails PatchCheck.py
> > Submitter can resolve PatchCheck.py issues and
> > reopen the PR.
> >
> > Cc: Liming Gao <liming.gao@intel.com>
> > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Signed-off-by: Michael D Kinney
> <michael.d.kinney@intel.com>
> > Reviewed-by: Liming Gao <liming.gao@intel.com>
> > ---
> > .mergify/config.yml | 97
> +++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 97 insertions(+)
> > create mode 100644 .mergify/config.yml
> >
> > diff --git a/.mergify/config.yml
> b/.mergify/config.yml
> > new file mode 100644
> > index 0000000000..26583de0f6
> > --- /dev/null
> > +++ b/.mergify/config.yml
> > @@ -0,0 +1,97 @@
> > +## @file
> > +# Mergify YML file that automatically merges a
> GitHub pull request against
> > +# edk2-ci if all of the GitHub branch protections
> have passed. It also
> > +# contains rules to:
> > +# * auto close branches that are not from an EDK II
> Maintainer
> > +# * post a comment on pull requests that have merge
> conflicts.
> > +# * post a comment on pull requests that have
> PatchCheck.py errors.
> > +#
> > +# Configuration Notes:
> > +# * Update the 'base=edk2-ci' statements with the
> name of the branch to merge
> > +# pull requests.
> > +#
> > +# * Update the 'status-failure' statement with the
> name of the name of the Azure
> > +# Pipelines Build that performs the EDK II
> Maintainer check.
> > +#
> > +# * This file must be checked into the 'default'
> branch of a repo. Copies
> > +# of this file on other branches of a repo are
> ignored by Mergify.
> > +#
> > +# Copyright (c) 2019, Intel Corporation. All rights
> reserved.<BR>
> > +# SPDX-License-Identifier: BSD-2-Clause-Patent
> > +#
> > +# https://github.com/apps/mergify
> > +# https://doc.mergify.io/
> > +#
> > +##
> > +
> > +pull_request_rules:
> > +
> > + - name: Automatically merge a PR when all required
> checks pass and 'push' label is present
> > + conditions:
> > + - base=master
> > + - label=push
> > + - author=@tianocore/edk-ii-maintainers
> > + - status-success=tianocore.PatchCheck
> > + - status-success=Ubuntu GCC5 PR
> > + - status-success=Windows VS2019 PR
> > + actions:
> > + merge:
> > + strict: true
> > + method: rebase
> > +
> > + - name: Automatically close a PR when all required
> checks pass and 'push' label is not present
> > + conditions:
> > + - base=master
> > + - -label=push
> > + - -closed
> > + - status-success=tianocore.PatchCheck
> > + - status-success=Ubuntu GCC5 PR
> > + - status-success=Windows VS2019 PR
> > + - status-success=Ubuntu GCC5 PR (FINISHED)
> > + - status-success=Windows VS2019 PR (FINISHED)
> > + actions:
> > + close:
> > + message: All checks passed. Auto close
> personal build.
> > +
> > + - name: Post a comment on a PR that can not be
> merged due to a merge conflict
> > + conditions:
> > + - base=master
> > + - conflict
> > + actions:
> > + comment:
> > + message: PR can not be merged due to
> conflict. Please rebase and resubmit
> > +
> > + - name: Automatically close a PR that fails the
> EDK II Maintainers membership check and 'push' label is
> present
> > + conditions:
> > + - base=master
> > + - label=push
> > + - -author=@tianocore/edk-ii-maintainers
> > + actions:
> > + close:
> > + message: PR submitter is not a member of the
> Tianocore EDK II Maintainers team
> > +
> > + - name: Post a comment on a PR if PatchCheck fails
> > + conditions:
> > + - base=master
> > + - status-failure=tianocore.PatchCheck
> > + actions:
> > + comment:
> > + message: PR can not be merged due to a
> PatchCheck failure. Please resolve and resubmit
> > +
> > + - name: Post a comment on a PR if Ubuntu GCC5
> fails
> > + conditions:
> > + - base=master
> > + - status-failure=Ubuntu GCC5 PR
> > + - status-success=Ubuntu GCC5 PR (FAILED)
> > + actions:
> > + comment:
> > + message: PR can not be merged due to an
> Ubuntu GCC5 failure. Please resolve and resubmit
> > +
> > + - name: Post a comment on a PR if Windows VS2019
> fails
> > + conditions:
> > + - base=master
> > + - status-failure=Windows VS2019 PR
> > + - status-success=Windows VS2019 PR (FAILED)
> > + actions:
> > + comment:
> > + message: PR can not be merged due to a
> Windows VS2019 failure. Please resolve and resubmit
> > --
> > 2.21.0.windows.1
> >
> >
> >
> >
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [edk2-devel] [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file
2020-01-17 20:22 ` Michael D Kinney
@ 2020-01-17 20:29 ` Leif Lindholm
0 siblings, 0 replies; 30+ messages in thread
From: Leif Lindholm @ 2020-01-17 20:29 UTC (permalink / raw)
To: Kinney, Michael D; +Cc: devel@edk2.groups.io
Hi Mike,
Many thanks for resolving!
I'll try your other suggestions if this happens again in future.
Best Regards,
Leif
On Fri, Jan 17, 2020 at 20:22:03 +0000, Kinney, Michael D wrote:
> Hi Leif,
>
> I have not tried the mergify commands like that. I can investigate
> further if we think we need them.
>
> I saw that some of the Azure pipelines checks did not run. That
> Is what was blocking Mergify. Mergify was waiting for status.
>
> I did a Close and Re-open to re-trigger Azure Pipelines and that
> worked. You changed have been committed.
>
> We have seen a very rare condition in communication between
> Azure Pipelines and GitHub. This close/re-open is what we
> can try first. The 2nd approach is to abandon the current
> PR and create a new PR with the same content. That has only
> had to be used once since enabling in early November.
>
> Mike
>
> > -----Original Message-----
> > From: Leif Lindholm <leif.lindholm@linaro.org>
> > Sent: Friday, January 17, 2020 11:44 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>
> > Cc: devel@edk2.groups.io
> > Subject: Re: [edk2-devel] [Patch v5 21/22] .mergify:
> > Add Mergify YML pull request rules configuration file
> >
> > Hi Mike (on-list),
> >
> > Embarassing to admit, but I hadn't tried to do my first
> > mergify commit
> > until the Maintainers.txt right now, and that isn't
> > working as I expected.
> >
> > I tried to force a retry on
> > https://github.com/tianocore/edk2/pull/307
> > (since I added the 'push' tag *after* submitting the
> > github pull
> > request), but as you can see mergify says
> > "@leiflindholm is not allowed
> > to run commands".
> >
> > Am I missing something obvious?
> >
> > Best Regards,
> >
> > Leif
> >
> > On Mon, Nov 11, 2019 at 00:41:06 -0800, Michael D
> > Kinney wrote:
> > > Add directory for the Mergify YML configuration files
> > that
> > > provides rules and actions used to process a pull
> > request.
> > >
> > > * Auto commit a PR from EDK II Maintainer with 'push'
> > label
> > > set and all CI checks pass
> > > * Auto close a PR from any developers without 'push'
> > label
> > > set and all CI checks pass.
> > > * Auto close a PR from a non EDK II Maintainer that
> > has
> > > the 'push' label set.
> > > * Post a comment to a PR that has a merge conflict.
> > > Submitter can resolved conflicts and reopen the PR.
> > > * Post a comment to a PR that fails PatchCheck.py
> > > Submitter can resolve PatchCheck.py issues and
> > > reopen the PR.
> > >
> > > Cc: Liming Gao <liming.gao@intel.com>
> > > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > Signed-off-by: Michael D Kinney
> > <michael.d.kinney@intel.com>
> > > Reviewed-by: Liming Gao <liming.gao@intel.com>
> > > ---
> > > .mergify/config.yml | 97
> > +++++++++++++++++++++++++++++++++++++++++++++
> > > 1 file changed, 97 insertions(+)
> > > create mode 100644 .mergify/config.yml
> > >
> > > diff --git a/.mergify/config.yml
> > b/.mergify/config.yml
> > > new file mode 100644
> > > index 0000000000..26583de0f6
> > > --- /dev/null
> > > +++ b/.mergify/config.yml
> > > @@ -0,0 +1,97 @@
> > > +## @file
> > > +# Mergify YML file that automatically merges a
> > GitHub pull request against
> > > +# edk2-ci if all of the GitHub branch protections
> > have passed. It also
> > > +# contains rules to:
> > > +# * auto close branches that are not from an EDK II
> > Maintainer
> > > +# * post a comment on pull requests that have merge
> > conflicts.
> > > +# * post a comment on pull requests that have
> > PatchCheck.py errors.
> > > +#
> > > +# Configuration Notes:
> > > +# * Update the 'base=edk2-ci' statements with the
> > name of the branch to merge
> > > +# pull requests.
> > > +#
> > > +# * Update the 'status-failure' statement with the
> > name of the name of the Azure
> > > +# Pipelines Build that performs the EDK II
> > Maintainer check.
> > > +#
> > > +# * This file must be checked into the 'default'
> > branch of a repo. Copies
> > > +# of this file on other branches of a repo are
> > ignored by Mergify.
> > > +#
> > > +# Copyright (c) 2019, Intel Corporation. All rights
> > reserved.<BR>
> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent
> > > +#
> > > +# https://github.com/apps/mergify
> > > +# https://doc.mergify.io/
> > > +#
> > > +##
> > > +
> > > +pull_request_rules:
> > > +
> > > + - name: Automatically merge a PR when all required
> > checks pass and 'push' label is present
> > > + conditions:
> > > + - base=master
> > > + - label=push
> > > + - author=@tianocore/edk-ii-maintainers
> > > + - status-success=tianocore.PatchCheck
> > > + - status-success=Ubuntu GCC5 PR
> > > + - status-success=Windows VS2019 PR
> > > + actions:
> > > + merge:
> > > + strict: true
> > > + method: rebase
> > > +
> > > + - name: Automatically close a PR when all required
> > checks pass and 'push' label is not present
> > > + conditions:
> > > + - base=master
> > > + - -label=push
> > > + - -closed
> > > + - status-success=tianocore.PatchCheck
> > > + - status-success=Ubuntu GCC5 PR
> > > + - status-success=Windows VS2019 PR
> > > + - status-success=Ubuntu GCC5 PR (FINISHED)
> > > + - status-success=Windows VS2019 PR (FINISHED)
> > > + actions:
> > > + close:
> > > + message: All checks passed. Auto close
> > personal build.
> > > +
> > > + - name: Post a comment on a PR that can not be
> > merged due to a merge conflict
> > > + conditions:
> > > + - base=master
> > > + - conflict
> > > + actions:
> > > + comment:
> > > + message: PR can not be merged due to
> > conflict. Please rebase and resubmit
> > > +
> > > + - name: Automatically close a PR that fails the
> > EDK II Maintainers membership check and 'push' label is
> > present
> > > + conditions:
> > > + - base=master
> > > + - label=push
> > > + - -author=@tianocore/edk-ii-maintainers
> > > + actions:
> > > + close:
> > > + message: PR submitter is not a member of the
> > Tianocore EDK II Maintainers team
> > > +
> > > + - name: Post a comment on a PR if PatchCheck fails
> > > + conditions:
> > > + - base=master
> > > + - status-failure=tianocore.PatchCheck
> > > + actions:
> > > + comment:
> > > + message: PR can not be merged due to a
> > PatchCheck failure. Please resolve and resubmit
> > > +
> > > + - name: Post a comment on a PR if Ubuntu GCC5
> > fails
> > > + conditions:
> > > + - base=master
> > > + - status-failure=Ubuntu GCC5 PR
> > > + - status-success=Ubuntu GCC5 PR (FAILED)
> > > + actions:
> > > + comment:
> > > + message: PR can not be merged due to an
> > Ubuntu GCC5 failure. Please resolve and resubmit
> > > +
> > > + - name: Post a comment on a PR if Windows VS2019
> > fails
> > > + conditions:
> > > + - base=master
> > > + - status-failure=Windows VS2019 PR
> > > + - status-success=Windows VS2019 PR (FAILED)
> > > + actions:
> > > + comment:
> > > + message: PR can not be merged due to a
> > Windows VS2019 failure. Please resolve and resubmit
> > > --
> > > 2.21.0.windows.1
> > >
> > >
> > >
> > >
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2020-01-17 20:29 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-11-11 8:40 [Patch v5 00/22] Enable Phase 1 of EDK II CI Michael D Kinney
2019-11-11 8:40 ` [Patch v5 01/22] Maintainers.txt: Add continuous integration(CI) directories Michael D Kinney
2019-11-11 8:40 ` [Patch v5 02/22] .gitignore: Ignore python compiled files, extdeps, and vscode Michael D Kinney
2019-11-11 8:40 ` [Patch v5 03/22] pip_requirements.txt: Add python pip requirements file Michael D Kinney
2019-11-11 9:58 ` Laszlo Ersek
2019-11-11 16:35 ` Michael D Kinney
2019-11-11 16:41 ` Laszlo Ersek
2019-11-11 8:40 ` [Patch v5 04/22] BaseTools: Add RC_PATH define for VS2017/2019 Michael D Kinney
2019-11-11 8:40 ` [Patch v5 05/22] BaseTools: Add YAML files with path env and tool extdeps Michael D Kinney
2019-11-11 8:40 ` [Patch v5 06/22] BaseTools: Add BaseTools plugins to support CI Michael D Kinney
2019-11-11 8:40 ` [Patch v5 07/22] .pytool/Plugin: Add CI plugins Michael D Kinney
2019-11-11 8:40 ` [Patch v5 08/22] CryptoPkg: Add YAML file for CI builds Michael D Kinney
2019-11-11 8:40 ` [Patch v5 09/22] FatPkg: " Michael D Kinney
2019-11-11 8:40 ` [Patch v5 10/22] FmpDevicePkg: " Michael D Kinney
2019-11-11 8:40 ` [Patch v5 11/22] MdeModulePkg: " Michael D Kinney
2019-11-11 8:40 ` [Patch v5 12/22] MdePkg: " Michael D Kinney
2019-11-11 8:40 ` [Patch v5 13/22] NetworkPkg: " Michael D Kinney
2019-11-11 8:40 ` [Patch v5 14/22] PcAtChipsetPkg: Add YAML files " Michael D Kinney
2019-11-11 8:41 ` [Patch v5 15/22] SecurityPkg: " Michael D Kinney
2019-11-11 8:41 ` [Patch v5 16/22] ShellPkg: Add YAML file " Michael D Kinney
2019-11-11 8:41 ` [Patch v5 17/22] UefiCpuPkg: " Michael D Kinney
2019-11-11 8:41 ` [Patch v5 18/22] SignedCapsulePkg: Use BaseCryptLibNull to reduce package CI time Michael D Kinney
2019-11-11 8:41 ` [Patch v5 19/22] .pytool: Add CISettings.py and Readme.md Michael D Kinney
2019-11-11 8:41 ` [Patch v5 20/22] .azurepipelines: Add Azure Pipelines YML configuration files Michael D Kinney
2019-11-11 8:41 ` [Patch v5 21/22] .mergify: Add Mergify YML pull request rules configuration file Michael D Kinney
2020-01-17 19:44 ` [edk2-devel] " Leif Lindholm
2020-01-17 20:22 ` Michael D Kinney
2020-01-17 20:29 ` Leif Lindholm
2019-11-11 8:41 ` [Patch v5 22/22] Readme.md: Add CI build status badges Michael D Kinney
2019-11-11 17:14 ` Leif Lindholm
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox