From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web08.4553.1624435372141996748 for ; Wed, 23 Jun 2021 01:02:52 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: pierre.gondois@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C1630113E; Wed, 23 Jun 2021 01:02:51 -0700 (PDT) Received: from e120189.arm.com (unknown [10.57.78.245]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 07D413F719; Wed, 23 Jun 2021 01:02:49 -0700 (PDT) From: "PierreGondois" To: devel@edk2.groups.io, Sean Brogan , Bret Barkelew , Michael D Kinney , Liming Gao , Sami Mujawar Subject: [PATCH edk2-platforms v1 3/6] .pytool/Plugin: Add CI plugins Date: Wed, 23 Jun 2021 09:02:06 +0100 Message-Id: <20210623080209.28380-4-Pierre.Gondois@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210623080209.28380-1-Pierre.Gondois@arm.com> References: <20210623080209.28380-1-Pierre.Gondois@arm.com> From: Pierre Gondois To enable CI support of the tianocore/edk2-platforms repository, add a .pytool directory containing the following files: - .pytool/CISettings.py - .pytool/Readme.md These files are largely inspired from the same files available in the edk2 repository. The .pytool/Plugin/* files containing the CI tests to run are not copied. edk2-platforms will rely on the edk2basetools python package and on the edk2 python files, as edk2 is imported as a submodule of edk2-platforms. Cc: Sean Brogan Cc: Bret Barkelew Cc: Michael D Kinney Cc: Liming Gao Cc: Sami Mujawar Signed-off-by: Sami Mujawar Signed-off-by: Pierre Gondois --- .pytool/CISettings.py | 185 ++++++++++++++++++++++++++++ .pytool/Readme.md | 271 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 456 insertions(+) create mode 100644 .pytool/CISettings.py create mode 100644 .pytool/Readme.md diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py new file mode 100644 index 000000000000..18604646030f --- /dev/null +++ b/.pytool/CISettings.py @@ -0,0 +1,185 @@ +# @file +# +# Copyright (c) Microsoft Corporation. +# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All rights reserved.
+# Copyright (c) 2020 - 2021, ARM Limited. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +## +import os +import logging +import edk2basetools + +from edk2toolext.environment import shell_environment +from edk2toolext.invocables.edk2_ci_build import CiBuildSettingsManager +from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule +from edk2toolext.invocables.edk2_update import UpdateSettingsManager +from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager +from edk2toollib.utility_functions import GetHostInfo + + +class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager): + + def __init__(self): + self.ActualPackages = [] + self.ActualTargets = [] + self.ActualArchitectures = [] + self.ActualToolChainTag = "" + self.ActualScopes = None + + # ####################################################################################### # + # Extra CmdLine configuration # + # ####################################################################################### # + + def AddCommandLineOptions(self, parserObj): + pass + def RetrieveCommandLineOptions(self, args): + pass + + # ####################################################################################### # + # Default Support for this Ci Build # + # ####################################################################################### # + + def GetPackagesSupported(self): + ''' return iterable of edk2 packages supported by this build. + These should be edk2 workspace relative paths ''' + return ( + "JunoPkg", + "VExpressPkg" + ) + + def GetArchitecturesSupported(self): + ''' return iterable of edk2 architectures supported by this build ''' + return ( + "IA32", + "X64", + "ARM", + "AARCH64", + "RISCV64") + + def GetTargetsSupported(self): + ''' return iterable of edk2 target tags supported by this build ''' + return ("DEBUG", "RELEASE", "NO-TARGET", "NOOPT") + + # ####################################################################################### # + # Verify and Save requested Ci Build Config # + # ####################################################################################### # + + def SetPackages(self, list_of_requested_packages): + ''' Confirm the requested package list is valid and configure SettingsManager + to build the requested packages. + + Raise UnsupportedException if a requested_package is not supported + ''' + unsupported = set(list_of_requested_packages) - \ + set(self.GetPackagesSupported()) + if(len(unsupported) > 0): + logging.critical( + "Unsupported Package Requested: " + " ".join(unsupported)) + raise Exception("Unsupported Package Requested: " + + " ".join(unsupported)) + self.ActualPackages = list_of_requested_packages + + def SetArchitectures(self, list_of_requested_architectures): + ''' Confirm the requests architecture list is valid and configure SettingsManager + to run only the requested architectures. + + Raise Exception if a list_of_requested_architectures is not supported + ''' + unsupported = set(list_of_requested_architectures) - \ + set(self.GetArchitecturesSupported()) + if(len(unsupported) > 0): + logging.critical( + "Unsupported Architecture Requested: " + " ".join(unsupported)) + raise Exception( + "Unsupported Architecture Requested: " + " ".join(unsupported)) + self.ActualArchitectures = list_of_requested_architectures + + def SetTargets(self, list_of_requested_target): + ''' Confirm the request target list is valid and configure SettingsManager + to run only the requested targets. + + Raise UnsupportedException if a requested_target is not supported + ''' + unsupported = set(list_of_requested_target) - \ + set(self.GetTargetsSupported()) + if(len(unsupported) > 0): + logging.critical( + "Unsupported Targets Requested: " + " ".join(unsupported)) + raise Exception("Unsupported Targets Requested: " + + " ".join(unsupported)) + self.ActualTargets = list_of_requested_target + + # ####################################################################################### # + # Actual Configuration for Ci Build # + # ####################################################################################### # + + def GetActiveScopes(self): + ''' return tuple containing scopes that should be active for this process ''' + if self.ActualScopes is None: + scopes = ("cibuild", "edk2-build", "host-based-test") + + self.ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "") + + is_linux = GetHostInfo().os.upper() == "LINUX" + scopes += ('pipbuild-unix',) if is_linux else ('pipbuild-win',) + + if is_linux and self.ActualToolChainTag.upper().startswith("GCC"): + if "AARCH64" in self.ActualArchitectures: + scopes += ("gcc_aarch64_linux",) + if "ARM" in self.ActualArchitectures: + scopes += ("gcc_arm_linux",) + if "RISCV64" in self.ActualArchitectures: + scopes += ("gcc_riscv64_unknown",) + self.ActualScopes = scopes + return self.ActualScopes + + def GetRequiredSubmodules(self): + ''' return iterable containing RequiredSubmodule objects. + If no RequiredSubmodules return an empty iterable + ''' + rs = [] + rs.append(RequiredSubmodule( + "edk2", True)) + rs.append(RequiredSubmodule( + "Silicon/RISC-V/ProcessorPkg/Library/RiscVOpensbiLib/opensbi", False)) + return rs + + def GetName(self): + return "Edk2-platforms" + + def GetDependencies(self): + return [ + ] + + def GetPackagesPath(self): + ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath ''' + edk2_platforms_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + return [ + edk2_platforms_path, + os.path.join(edk2_platforms_path, "Platform", "ARM"), + os.path.join(edk2_platforms_path, "edk2") + ] + + def GetWorkspaceRoot(self): + ''' get WorkspacePath ''' + return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list: + ''' Filter potential packages to test based on changed files. ''' + build_these_packages = [] + possible_packages = potentialPackagesList.copy() + for f in changedFilesList: + # split each part of path for comparison later + nodes = f.split("/") + + # python file change in .pytool folder causes building all + if f.endswith(".py") and ".pytool" in nodes: + build_these_packages = possible_packages + break + + # BaseTools files that might change the build + if "BaseTools" in nodes: + if os.path.splitext(f) not in [".txt", ".md"]: + build_these_packages = possible_packages + break + return build_these_packages diff --git a/.pytool/Readme.md b/.pytool/Readme.md new file mode 100644 index 000000000000..5b3d9679d7f2 --- /dev/null +++ b/.pytool/Readme.md @@ -0,0 +1,271 @@ +# Edk2-platforms Continuous Integration + +## Basic Status + +| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues | +| :---- | :----- | :---- | :--- | +| Platfrom/ARM/JunoPkg | | :heavy_check_mark: | Spell checking in audit mode. CompilerCheck disabled (need a PlatformCI). + +For more detailed status look at the test results of the latest CI run on the +repo readme. + +## Background + +This Continuous integration and testing infrastructure leverages the TianoCore EDKII Tools PIP modules: +[library](https://pypi.org/project/edk2-pytool-library/) and +[extensions](https://pypi.org/project/edk2-pytool-extensions/) (with repos +located [here](https://github.com/tianocore/edk2-pytool-library) and +[here](https://github.com/tianocore/edk2-pytool-extensions)). + +The primary execution flows can be found in the +`.azurepipelines/Windows-VS2019.yml` and `.azurepipelines/Ubuntu-GCC5.yml` +files. These YAML files are consumed by the Azure Dev Ops Build Pipeline and +dictate what server resources should be used, how they should be configured, and +what processes should be run on them. An overview of this schema can be found +[here](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema). + +Inspection of these files reveals the EDKII Tools commands that make up the +primary processes for the CI build: 'stuart_setup', 'stuart_update', and +'stuart_ci_build'. These commands come from the EDKII Tools PIP modules and are +configured as described below. More documentation on the tools can be +found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md) +and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_invocables.md). + +## Configuration + +Configuration of the CI process consists of (in order of precedence): + +* command-line arguments passed in via the Pipeline YAML +* a per-package configuration file (e.g. `.ci.yaml`) that is + detected by the CI system in EDKII Tools. +* a global configuration Python module (e.g. `CISetting.py`) passed in via the + command-line + +The global configuration file is described in +[this readme](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_settings_manager.md) +from the EDKII Tools documentation. This configuration is written as a Python +module so that decisions can be made dynamically based on command line +parameters and codebase state. + +The per-package configuration file can override most settings in the global +configuration file, but is not dynamic. This file can be used to skip or +customize tests that may be incompatible with a specific package. Each test generally requires +per package configuration which comes from this file. + +## Running CI locally + +The EDKII Tools environment (and by extension the ci) is designed to support +easily and consistently running locally and in a cloud ci environment. To do +that a few steps should be followed. Details of EDKII Tools can be found in the +[docs folder here](https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs) + +### Prerequisets + +1. A supported toolchain (others might work but this is what is tested and validated) + * Windows 10: + * VS 2017 or VS 2019 + * Windows SDK (for rc) + * Windows WDK (for capsules) + * Ubuntu 18.04 or Fedora + * GCC5 + * Easy to add more but this is the current state +2. Python 3.7.x or newer on path +3. git on path +4. Recommended to setup and activate a python virtual environment +5. Install the requirements `pip install --upgrade pip-requirements.txt` + +### Running CI + +1. clone your edk2-platforms repo +2. Activate your python virtual environment in cmd window +3. Get code dependencies (done only when submodules change) + * `stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=` +4. Update other dependencies (done more often) + * `stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=` +5. Run CI build (--help will give you options) + * `stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=` + * -p : To build only certain packages use a CSV list + * -a : To run only certain architectures use a CSV list + * -t : To run only tests related to certain targets use a + CSV list + * By default all tests are opted in. Then given a package.ci.yaml file those + tests can be configured for a package. Finally setting the check to the + value `skip` will skip that plugin. Examples: + * `CompilerPlugin=skip` skip the build test + * `GuidCheck=skip` skip the Guid check + * `SpellCheck=skip` skip the spell checker + * etc +6. Detailed reports and logs per package are captured in the `Build` directory + +## Current PyTool Test Capabilities + +All CI tests are instances of EDKII Tools plugins. Documentation on the plugin +system can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_plugin_manager.md) +and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_plugin_manager.md). +Upon invocation, each plugin will be passed the path to the current package +under test and a dictionary containing its targeted configuration, as assembled +from the command line, per-package configuration, and global configuration. + +Note: CI plugins are considered unique from build plugins and helper plugins, +even though some CI plugins may execute steps of a build. + +In the example, these plugins live alongside the code under test (in the +`.pytool/Plugin` directory), but may be moved to the 'edk2-test' repo if that +location makes more sense for the community. + +### Module Inclusion Test - DscCompleteCheck + +This scans all INF files from a package and confirms they are +listed in the package level DSC file. The test considers it an error if any INF +does not appear in the `Components` section of the package-level DSC (indicating +that it would not be built if the package were built). This is critical because +much of the CI infrastructure assumes that all modules will be listed in the DSC +and compiled. + +This test will ignore INFs in the following cases: + +1. When `MODULE_TYPE` = `HOST_APPLICATION` +2. When a Library instance **only** supports the `HOST_APPLICATION` environment + +### Host Module Inclusion Test - HostUnitTestDscCompleteCheck + +This test scans all INF files from a package for those related to host +based unit tests and confirms they are listed in the unit test DSC file for the package. +The test considers it an error if any INF meeting the requirements does not appear +in the `Components` section of the unit test DSC. This is critical because +much of the CI infrastructure assumes that modules will be listed in the DSC +and compiled. + +This test will only require INFs in the following cases: + +1. When `MODULE_TYPE` = `HOST_APPLICATION` +2. When a Library instance explicitly supports the `HOST_APPLICATION` environment + +### Code Compilation Test - CompilerPlugin + +Once the Module Inclusion Test has verified that all modules would be built if +all package-level DSCs were built, the Code Compilation Test simply runs through +and builds every package-level DSC on every toolchain and for every architecture +that is supported. Any module that fails to build is considered an error. + +### Host Unit Test Compilation and Run Test - HostUnitTestCompilerPlugin + +A test that compiles the dsc for host based unit test apps. +On Windows this will also enable a build plugin to execute that will run the unit tests and verify the results. + +These tools will be invoked on any CI +pass that includes the NOOPT target. In order for these tools to do their job, +the package and tests must be configured in a particular way... + +#### Including Host-Based Tests in the Package YAML + +For example, looking at the `MdeModulePkg.ci.yaml` config file, there are two +config options that control HostBased test behavior: + +```json + ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin + "HostUnitTestCompilerPlugin": { + "DscPath": "Test/MdeModulePkgHostTest.dsc" + }, +``` + +This option tell the test builder to run. The test builder needs to know which +modules in this package are host-based tests, so that DSC path is provided. + +#### Configuring the HostBased DSC + +The HostBased DSC for `MdeModulePkg` is located at +`MdeModulePkg/Test/MdeModulePkgHostTest.dsc`. + +To add automated host-based unit test building to a new package, create a +similar DSC. The new DSC should make sure to have the `NOOPT` BUILD_TARGET +and should include the line: + +``` +!include UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc +``` + +All of the modules that are included in the `Components` section of this +DSC should be of type HOST_APPLICATION. + +### GUID Uniqueness Test - GuidCheck + +This test works on the collection of all packages rather than an individual +package. It looks at all FILE_GUIDs and GUIDs declared in DEC files and ensures +that they are unique for the codebase. This prevents, for example, accidental +duplication of GUIDs when using an existing INF as a template for a new module. + +### Cross-Package Dependency Test - DependencyCheck + +This test compares the list of all packages used in INFs files for a given +package against a list of "allowed dependencies" in plugin configuration for +that package. Any module that depends on a disallowed package will cause a test +failure. + +### Library Declaration Test - LibraryClassCheck + +This test scans at all library header files found in the `Library` folders in +all of the package's declared include directories and ensures that all files +have a matching LibraryClass declaration in the DEC file for the package. Any +missing declarations will cause a failure. + +### Invalid Character Test - CharEncodingCheck + +This test scans all files in a package to make sure that there are no invalid +Unicode characters that may cause build errors in some character +sets/localizations. + +### Spell Checking - cspell + +This test runs a spell checker on all files within the package. This is done +using the NodeJs cspell tool. For details check `.pytool/Plugin/SpellCheck`. +For this plugin to run during ci you must install nodejs and cspell and have +both available to the command line when running your CI. + +Install + +* Install nodejs from https://nodejs.org/en/ +* Install cspell + 1. Open cmd prompt with access to node and npm + 2. Run `npm install -g cspell` + + More cspell info: https://github.com/streetsidesoftware/cspell + +### License Checking - LicenseCheck + +Scans all new added files in a package to make sure code is contributed under +BSD-2-Clause-Patent. + +### Ecc tool - EccCheck + +Run the Ecc tool on the package. The Ecc tool is available in the BaseTools +package. It checks that the code complies to the EDKII coding standard. + +## PyTool Scopes + +Scopes are how the PyTool ext_dep, path_env, and plugins are activated. Meaning +that if an invocable process has a scope active then those ext_dep and path_env +will be active. To allow easy integration of PyTools capabilities there are a +few standard scopes. + +| Scope | Invocable | Description | +| :---- | :----- | :---- | +| global | edk2_invocable++ - should be base_abstract_invocable | Running an invocables | +| global-win | edk2_invocable++ | Running on Microsoft Windows | +| global-nix | edk2_invocable++ | Running on Linux based OS | +| edk2-build | | This indicates that an invocable is building EDK2 based UEFI code | +| cibuild | set in .pytool/CISettings.py | Suggested target for edk2 continuous integration builds. Tools used for CiBuilds can use this scope. Example: asl compiler | +| host-based-test | set in .pytool/CISettings.py | Turns on the host based tests and plugin | +| host-test-win | set in .pytool/CISettings.py | Enables the host based test runner for Windows | + +## Future investments + +* PatchCheck tests as plugins +* MacOS/xcode support +* Clang/LLVM support +* Visual Studio AARCH64 and ARM support +* BaseTools C tools CI/PR and binary release process +* BaseTools Python tools CI/PR process +* Extensible private/closed source platform reporting +* UEFI SCTs +* Other automation -- 2.17.1