* [PATCH v1 1/6] .azurepipelines: Add Platform CI template
[not found] <20200408181327.4324-1-michael.kubacki@outlook.com>
@ 2020-04-08 18:13 ` Michael Kubacki
2020-04-08 18:13 ` [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI Michael Kubacki
` (4 subsequent siblings)
5 siblings, 0 replies; 29+ messages in thread
From: Michael Kubacki @ 2020-04-08 18:13 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bret Barkelew, Michael D Kinney, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
Add template for Platform CI steps for a Pytools based build.
Add README to describe the template and how to use it.
Add helpful information for working with azurepipelines, templates, and
lessons learned.
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
---
.azurepipelines/ReadMe.md | 50 ++++++++
.azurepipelines/templates/ReadMe.md | 59 +++++++++
.azurepipelines/templates/platform-build-run-steps.yml | 134 ++++++++++++++++++++
3 files changed, 243 insertions(+)
diff --git a/.azurepipelines/ReadMe.md b/.azurepipelines/ReadMe.md
new file mode 100644
index 000000000000..cf57d282c197
--- /dev/null
+++ b/.azurepipelines/ReadMe.md
@@ -0,0 +1,50 @@
+# Azure DevOps Pipelines
+
+These yml files are used to provide CI builds using the Azure DevOps Pipeline Service.
+Most of the CI leverages edk2-pytools to support cross platform building and execution.
+
+## Core CI
+
+Focused on building and testing all packages in Edk2 without an actual target platform.
+
+See `.pytools/ReadMe.py` for more details
+
+## Platform CI
+
+Focused on building a single target platform and confirming functionality on that platform.
+
+## Conventions
+
+* Files extension should be *.yml. *.yaml is also supported but in Edk2 we use those for our package configuration.
+* Platform CI files should be in the `<PlatformPkg>/.azurepipelines` folder.
+* Core CI files are in the root folder.
+* Shared templates are in the `templates` folder.
+* Top level CI files should be named `<host os>-<tool_chain_tag>.yml`
+
+## Links
+
+* Basic Azure Landing Site - https://docs.microsoft.com/en-us/azure/devops/pipelines/?view=azure-devops
+* Pipeline jobs - https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
+* Pipeline yml scheme - https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
+* Pipeline expression - https://docs.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
+* PyTools - https://github.com/tianocore/edk2-pytool-extensions and https://github.com/tianocore/edk2-pytool-library
+
+## Lessons Learned
+
+### Templates and parameters
+
+They are great but evil. If they are used as part of determining the steps of a build they must resolve before the build starts. They can not use variables set in a yml or determined as part of a matrix. If they are used in a step then they can be bound late.
+
+### File matching patterns
+
+On Linux this can hang if there are too many files in the search list.
+
+### Templates and file splitting
+
+Suggestion is to do one big yaml file that does what you want for one of your targets. Then do the second one and find the deltas. From that you can start to figure out the right split of files, steps, jobs.
+
+### Conditional steps
+
+If you want the step to show up in the log but not run, use a step conditional. This is great when a platform doesn't currently support a feature but you want the builders to know that the features exists and maybe someday it will.
+
+If you want the step to not show up use a template step conditional wrapper. Beware this will be evaluated early (at build start). This can hide things not needed on a given OS for example.
diff --git a/.azurepipelines/templates/ReadMe.md b/.azurepipelines/templates/ReadMe.md
new file mode 100644
index 000000000000..fa433e3ef597
--- /dev/null
+++ b/.azurepipelines/templates/ReadMe.md
@@ -0,0 +1,59 @@
+# CI Templates
+
+This folder contains azure pipeline yml templates for "Core" and "Platform" Continuous Integration and PR validation.
+
+## Common CI templates
+
+### basetools-build-steps.yml
+
+This template compiles the Edk2 basetools from source. The steps in this template are
+conditional and will only run if variable `pkg_count` is greater than 0.
+
+It also has two conditional steps only used when the toolchain contains GCC. These two steps
+use `apt` to update the system packages and add those necessary for Edk2 builds.
+
+## Core CI templates
+
+### pr-gate-build-job.yml
+
+This templates contains the jobs and most importantly the matrix of which packages and
+targets to run for Core CI.
+
+### pr-gate-steps.yml
+
+This template is the main Core CI template. It controls all the steps run and is responsible for most functionality of the Core CI process. This template sets
+the `pkg_count` variable using the `stuart_pr_eval` tool when the
+build type is "pull request"
+
+### spell-check-prereq-steps.yml
+
+This template installs the node based tools used by the spell checker plugin. The steps
+in this template are conditional and will only run if variable `pkg_count` is greater than 0.
+
+## Platform CI templates
+
+### platform-build-run-steps.yml
+
+This template makes heavy use of pytools to build and run a platform in the Edk2 repo
+
+Also uses basetools-build-steps.yml to compile basetools
+
+#### Special Notes
+
+* For a build type of pull request it will conditionally build if the patches change files that impact the platform.
+ * uses `stuart_pr_eval` to determine impact
+* For manual builds or CI builds it will always build the platform
+* It compiles basetools from source
+* Will use `stuart_build --FlashOnly` to attempt to run the built image if the `Run` parameter is set.
+* See the parameters block for expected configuration options
+* Parameter `extra_install_step` allows the caller to insert extra steps. This is useful if additional dependencies, tools, or other things need to be installed. Here is an example of installing qemu on Windows.
+
+ ``` yaml
+ steps:
+ - template: ../../.azurepipelines/templates/build-run-steps.yml
+ parameters:
+ extra_install_step:
+ - powershell: choco install qemu; Write-Host "##vso[task.prependpath]c:\Program Files\qemu"
+ displayName: Install QEMU and Set QEMU on path # friendly name displayed in the UI
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+ ```
diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml
new file mode 100644
index 000000000000..ebf674bfc7d8
--- /dev/null
+++ b/.azurepipelines/templates/platform-build-run-steps.yml
@@ -0,0 +1,134 @@
+
+## @file
+# File steps.yml
+#
+# template file containing the steps to build
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+- name: tool_chain_tag
+ type: string
+ default: ''
+- name: build_pkg
+ type: string
+ default: ''
+- name: build_target
+ type: string
+ default: ''
+- name: build_arch
+ type: string
+ default: ''
+- name: build_file
+ type: string
+ default: ''
+- name: build_flags
+ type: string
+ default: ''
+- name: run_flags
+ type: string
+ default: ''
+
+- name: extra_install_step
+ type: stepList
+ default: []
+
+steps:
+- checkout: self
+ clean: true
+ fetchDepth: 1
+
+- task: UsePythonVersion@0
+ inputs:
+ versionSpec: "3.8.x"
+ architecture: "x64"
+
+- script: pip install -r pip-requirements.txt --upgrade
+ displayName: 'Install/Upgrade pip modules'
+
+# Set default
+- bash: echo "##vso[task.setvariable variable=pkg_count]${{ 1 }}"
+
+# trim the package list if this is a PR
+- task: CmdLine@1
+ displayName: Check if ${{ parameters.build_pkg }} need testing
+ inputs:
+ filename: stuart_pr_eval
+ arguments: -c ${{ parameters.build_file }} -t ${{ parameters.build_target}} -a ${{ parameters.build_arch}} --pr-target origin/$(System.PullRequest.targetBranch) --output-count-format-string "##vso[task.setvariable variable=pkg_count;isOutpout=true]{pkgcount}"
+ condition: eq(variables['Build.Reason'], 'PullRequest')
+
+ # Setup repo
+- task: CmdLine@1
+ displayName: Setup
+ inputs:
+ filename: stuart_setup
+ arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} -t ${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# Stuart Update
+- task: CmdLine@1
+ displayName: Update
+ inputs:
+ filename: stuart_update
+ arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} -t ${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# build basetools
+# do this after setup and update so that code base dependencies
+# are all resolved.
+- template: basetools-build-steps.yml
+ parameters:
+ tool_chain_tag: ${{ parameters.tool_chain_tag }}
+
+# Potential Extra steps
+- ${{ parameters.extra_install_step }}
+
+# Build
+- task: CmdLine@1
+ displayName: Build
+ inputs:
+ filename: stuart_build
+ arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} TARGET=${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}}
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# Run
+- task: CmdLine@1
+ displayName: Run to shell
+ inputs:
+ filename: stuart_build
+ arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} TARGET=${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}} ${{ parameters.run_flags }} --FlashOnly
+ condition: and(and(gt(variables.pkg_count, 0), succeeded()), eq(variables['Run'], true))
+ timeoutInMinutes: 1
+
+# Copy the build logs to the artifact staging directory
+- task: CopyFiles@2
+ displayName: "Copy build logs"
+ inputs:
+ targetFolder: "$(Build.ArtifactStagingDirectory)"
+ SourceFolder: "Build"
+ contents: |
+ BUILDLOG_*.txt
+ BUILDLOG_*.md
+ CI_*.txt
+ CI_*.md
+ CISETUP.txt
+ SETUPLOG.txt
+ UPDATE_LOG.txt
+ PREVALLOG.txt
+ TestSuites.xml
+ **/BUILD_TOOLS_REPORT.html
+ **/OVERRIDELOG.TXT
+ BASETOOLS_BUILD*.*
+ flattenFolders: true
+ condition: succeededOrFailed()
+
+# Publish build artifacts to Azure Artifacts/TFS or a file share
+- task: PublishBuildArtifacts@1
+ continueOnError: true
+ displayName: "Publish build logs"
+ inputs:
+ pathtoPublish: "$(Build.ArtifactStagingDirectory)"
+ artifactName: "Build Logs $(System.JobName)"
+ condition: succeededOrFailed()
--
2.16.3.windows.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
[not found] <20200408181327.4324-1-michael.kubacki@outlook.com>
2020-04-08 18:13 ` [PATCH v1 1/6] .azurepipelines: Add Platform CI template Michael Kubacki
@ 2020-04-08 18:13 ` Michael Kubacki
2020-04-09 6:47 ` Laszlo Ersek
` (2 more replies)
2020-04-08 18:13 ` [PATCH v1 3/6] EmulatorPkg: " Michael Kubacki
` (3 subsequent siblings)
5 siblings, 3 replies; 29+ messages in thread
From: Michael Kubacki @ 2020-04-08 18:13 UTC (permalink / raw)
To: devel; +Cc: Laszlo Ersek, Ard Biesheuvel, Leif Lindholm
From: Sean Brogan <sean.brogan@microsoft.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
* Ubuntu GCC5
Add PyTool based build of ArmVirtPkg
Add extdep for managing the iasl dependency
Add ArmVirtPkg.ci.yaml for Core CI
Add README-pytools for details and instructions
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
---
ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml | 89 +++++++
ArmVirtPkg/ArmVirtPkg.ci.yaml | 103 ++++++++
ArmVirtPkg/PlatformBuild.py | 263 ++++++++++++++++++++
ArmVirtPkg/README-pytools.md | 123 +++++++++
ArmVirtPkg/iasl_ext_dep.yaml | 21 ++
5 files changed, 599 insertions(+)
diff --git a/ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml
new file mode 100644
index 000000000000..ef5b61a0d4df
--- /dev/null
+++ b/ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml
@@ -0,0 +1,89 @@
+## @file
+# Azure Pipeline build file for building a platform.
+#
+# Platform: ArmVirtQemu
+# OS: Ubuntu
+# Toolchain: GCC5
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+ - master
+pr:
+ - master
+
+jobs:
+ - job: Platform_CI
+ variables:
+ package: 'ArmVirtPkg'
+ vm_image: 'ubuntu-latest'
+ should_run: true
+ run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
+
+ #Use matrix to speed up the build process
+ strategy:
+ matrix:
+ QEMU_AARCH64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "AARCH64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ QEMU_AARCH64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "AARCH64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ QEMU_AARCH64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "AARCH64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ QEMU_ARM_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "ARM"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ QEMU_ARM_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "ARM"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ QEMU_ARM_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "ARM"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ workspace:
+ clean: all
+
+ pool:
+ vmImage: $(vm_image)
+
+ steps:
+ - template: ../../.azurepipelines/templates/platform-build-run-steps.yml
+ parameters:
+ tool_chain_tag: GCC5
+ build_pkg: $(package)
+ build_target: $(Build.Target)
+ build_arch: $(Build.Arch)
+ build_file: $(Build.File)
+ build_flags: $(Build.Flags)
+ run_flags: $(Run.Flags)
+ extra_install_step:
+ - bash: sudo apt-get install qemu
+ displayName: Install qemu
+ condition: and(gt(variables.pkg_count, 0), succeeded())
diff --git a/ArmVirtPkg/ArmVirtPkg.ci.yaml b/ArmVirtPkg/ArmVirtPkg.ci.yaml
new file mode 100644
index 000000000000..4553725ee528
--- /dev/null
+++ b/ArmVirtPkg/ArmVirtPkg.ci.yaml
@@ -0,0 +1,103 @@
+## @file
+# Core CI configuration for ArmVirtPkg
+#
+# ArmVirtPkg is part of Platform Ci for builds so this is only
+# used for code analysis.
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ ## options defined .pytool/Plugin/CompilerPlugin
+ "CompilerPlugin": {
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin
+ "HostUnitTestCompilerPlugin": {
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/CharEncodingCheck
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+
+ ## options defined .pytool/Plugin/DependencyCheck
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "ArmVirtPkg/ArmVirtPkg.dec",
+ "NetworkPkg/NetworkPkg.dec",
+ "ArmPkg/ArmPkg.dec",
+ "OvmfPkg/OvmfPkg.dec",
+ "EmbeddedPkg/EmbeddedPkg.dec",
+ "ArmPlatformPkg/ArmPlatformPkg.dec",
+ "SecurityPkg/SecurityPkg.dec",
+ "ShellPkg/ShellPkg.dec" #Is this ok?
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[
+ "UnitTestFrameworkPkg/UnitTestFrameworkPkg.dec"
+ ],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[
+
+ ],
+ "IgnoreInf": []
+ },
+
+ ## options defined .pytool/Plugin/DscCompleteCheck
+ "DscCompleteCheck": {
+ "IgnoreInf": [""],
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/HostUnitTestDscCompleteCheck
+ "HostUnitTestDscCompleteCheck": {
+ "IgnoreInf": [""],
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/GuidCheck
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [],
+ },
+
+ ## options defined .pytool/Plugin/LibraryClassCheck
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined .pytool/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": False, # Fails right now with over 270 errors
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [
+ "setjump",
+ "plong",
+ "lparam",
+ "lpdword",
+ "lpthread",
+ "lresult",
+ "bootable",
+ "bsymbolic",
+ "endiannness",
+ "fvmain",
+ "multiboot",
+ "qemu's",
+ "ramdisk",
+ "ramfb",
+ "unbootable",
+ "virt's",
+ "werror",
+ "xenio"
+ ], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
diff --git a/ArmVirtPkg/PlatformBuild.py b/ArmVirtPkg/PlatformBuild.py
new file mode 100644
index 000000000000..c2c5cd098e7e
--- /dev/null
+++ b/ArmVirtPkg/PlatformBuild.py
@@ -0,0 +1,263 @@
+# @file
+# Script to Build ArmVirtPkg UEFI firmware
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+
+from edk2toolext.environment import shell_environment
+from edk2toolext.environment.uefi_build import UefiBuilder
+from edk2toolext.invocables.edk2_platform_build import BuildSettingsManager
+from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule
+from edk2toolext.invocables.edk2_update import UpdateSettingsManager
+from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager
+from edk2toollib.utility_functions import RunCmd
+from edk2toollib.utility_functions import GetHostInfo
+
+# ####################################################################################### #
+# Common Configuration #
+# ####################################################################################### #
+
+
+class CommonPlatform():
+ ''' Common settings for this platform. Define static data here and use
+ for the different parts of stuart
+ '''
+ PackagesSupported = ("ArmVirtPkg",)
+ ArchSupported = ("AARCH64", "ARM")
+ TargetsSupported = ("DEBUG", "RELEASE", "NOOPT")
+ Scopes = ('armvirt', 'edk2-build')
+ WorkspaceRoot = os.path.realpath(os.path.join(
+ os.path.dirname(os.path.abspath(__file__)), ".."))
+
+ # ####################################################################################### #
+ # Configuration for Update & Setup #
+ # ####################################################################################### #
+
+
+class SettingsManager(UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager):
+
+ def GetPackagesSupported(self):
+ ''' return iterable of edk2 packages supported by this build.
+ These should be edk2 workspace relative paths '''
+ return CommonPlatform.PackagesSupported
+
+ def GetArchitecturesSupported(self):
+ ''' return iterable of edk2 architectures supported by this build '''
+ return CommonPlatform.ArchSupported
+
+ def GetTargetsSupported(self):
+ ''' return iterable of edk2 target tags supported by this build '''
+ return CommonPlatform.TargetsSupported
+
+ def GetRequiredSubmodules(self):
+ ''' return iterable containing RequiredSubmodule objects.
+ If no RequiredSubmodules return an empty iterable
+ '''
+ rs = []
+ rs.append(RequiredSubmodule(
+ "ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3", False))
+ rs.append(RequiredSubmodule(
+ "CryptoPkg/Library/OpensslLib/openssl", False))
+ return rs
+
+ def SetArchitectures(self, list_of_requested_architectures):
+ ''' Confirm the requests architecture list is valid and configure SettingsManager
+ to run only the requested architectures.
+
+ Raise Exception if a list_of_requested_architectures is not supported
+ '''
+ unsupported = set(list_of_requested_architectures) - \
+ set(self.GetArchitecturesSupported())
+ if(len(unsupported) > 0):
+ errorString = (
+ "Unsupported Architecture Requested: " + " ".join(unsupported))
+ logging.critical(errorString)
+ raise Exception(errorString)
+ self.ActualArchitectures = list_of_requested_architectures
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return CommonPlatform.WorkspaceRoot
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+
+ scopes = CommonPlatform.Scopes
+ ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "")
+
+ if GetHostInfo().os.upper() == "LINUX" and ActualToolChainTag.upper().startswith("GCC"):
+ if "AARCH64" in self.ActualArchitectures:
+ scopes += ("gcc_aarch64_linux",)
+ if "ARM" in self.ActualArchitectures:
+ scopes += ("gcc_arm_linux",)
+ return scopes
+
+ def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list:
+ ''' Filter other cases that this package should be built
+ based on changed files. This should cover things that can't
+ be detected as dependencies. '''
+ build_these_packages = []
+ possible_packages = potentialPackagesList.copy()
+ for f in changedFilesList:
+ # BaseTools files that might change the build
+ if "BaseTools" in f:
+ if os.path.splitext(f) not in [".txt", ".md"]:
+ build_these_packages = possible_packages
+ break
+
+ # if the azure pipeline file changed
+ if ".azurepipelines" in f and "ArmVirt" in f:
+ build_these_packages = possible_packages
+ break
+
+ # if the azure pipeline platform template file changed
+ if ".azurepipelines" in f and "platforms" in f and "templates" in f:
+ build_these_packages = possible_packages
+ break
+
+ return build_these_packages
+
+ def GetPlatformDscAndConfig(self) -> tuple:
+ ''' If a platform desires to provide its DSC then Policy 4 will evaluate if
+ any of the changes will be built in the dsc.
+
+ The tuple should be (<workspace relative path to dsc file>, <input dictionary of dsc key value pairs>)
+ '''
+ return (os.path.join("ArmVirtPkg", "ArmVirtQemu.dsc"), {})
+
+
+ # ####################################################################################### #
+ # Actual Configuration for Platform Build #
+ # ####################################################################################### #
+
+
+class PlatformBuilder(UefiBuilder, BuildSettingsManager):
+ def __init__(self):
+ UefiBuilder.__init__(self)
+
+ def AddCommandLineOptions(self, parserObj):
+ ''' Add command line options to the argparser '''
+ parserObj.add_argument('-a', "--arch", dest="build_arch", type=str, default="AARCH64",
+ help="Optional - Architecture to build. Default = AARCH64")
+
+ def RetrieveCommandLineOptions(self, args):
+ ''' Retrieve command line options from the argparser '''
+
+ shell_environment.GetBuildVars().SetValue(
+ "TARGET_ARCH", args.build_arch.upper(), "From CmdLine")
+
+ shell_environment.GetBuildVars().SetValue(
+ "ACTIVE_PLATFORM", "ArmVirtPkg/ArmVirtQemu.dsc", "From CmdLine")
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return CommonPlatform.WorkspaceRoot
+
+ def GetPackagesPath(self):
+ ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath '''
+ return ()
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ scopes = CommonPlatform.Scopes
+ ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "")
+ Arch = shell_environment.GetBuildVars().GetValue("TARGET_ARCH", "")
+
+ if GetHostInfo().os.upper() == "LINUX" and ActualToolChainTag.upper().startswith("GCC"):
+ if "AARCH64" == Arch:
+ scopes += ("gcc_aarch64_linux",)
+ elif "ARM" == Arch:
+ scopes += ("gcc_arm_linux",)
+ return scopes
+
+ def GetName(self):
+ ''' Get the name of the repo, platform, or product being build '''
+ ''' Used for naming the log file, among others '''
+ # check the startup nsh flag and if set then rename the log file.
+ # this helps in CI so we don't overwrite the build log since running
+ # uses the stuart_build command.
+ if(shell_environment.GetBuildVars().GetValue("MAKE_STARTUP_NSH", "FALSE") == "TRUE"):
+ return "ArmVirtPkg_With_Run"
+ return "ArmVirtPkg"
+
+ def GetLoggingLevel(self, loggerType):
+ ''' Get the logging level for a given type
+ base == lowest logging level supported
+ con == Screen logging
+ txt == plain text file logging
+ md == markdown file logging
+ '''
+ return logging.DEBUG
+
+ def SetPlatformEnv(self):
+ logging.debug("PlatformBuilder SetPlatformEnv")
+ self.env.SetValue("PRODUCT_NAME", "ArmVirtQemu", "Platform Hardcoded")
+ self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
+ self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
+ return 0
+
+ def PlatformPreBuild(self):
+ return 0
+
+ def PlatformPostBuild(self):
+ return 0
+
+ def FlashRomImage(self):
+ VirtualDrive = os.path.join(self.env.GetValue(
+ "BUILD_OUTPUT_BASE"), "VirtualDrive")
+ os.makedirs(VirtualDrive, exist_ok=True)
+ OutputPath_FV = os.path.join(
+ self.env.GetValue("BUILD_OUTPUT_BASE"), "FV")
+ Built_FV = os.path.join(OutputPath_FV, "QEMU_EFI.fd")
+
+ # pad fd to 64mb
+ with open(Built_FV, "ab") as fvfile:
+ fvfile.seek(0, os.SEEK_END)
+ additional = b'\0' * ((64 * 1024 * 1024)-fvfile.tell())
+ fvfile.write(additional)
+
+ # QEMU must be on that path
+
+ # Unique Command and Args parameters per ARCH
+ if (self.env.GetValue("TARGET_ARCH").upper() == "AARCH64"):
+ cmd = "qemu-system-aarch64"
+ args = "-M virt"
+ args += " -cpu cortex-a57" # emulate cpu
+ elif(self.env.GetValue("TARGET_ARCH").upper() == "ARM"):
+ cmd = "qemu-system-arm"
+ args = "-M virt"
+ args += " -cpu cortex-a15" # emulate cpu
+ else:
+ raise NotImplementedError()
+
+ # Common Args
+ args += " -pflash " + Built_FV # path to fw
+ args += " -m 1024" # 1gb memory
+ # turn off network
+ args += " -net none"
+ # Serial messages out
+ args += " -serial stdio"
+ # Mount disk with startup.nsh
+ args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk"
+
+ # Conditional Args
+ if (self.env.GetValue("QEMU_HEADLESS").upper() == "TRUE"):
+ args += " -display none" # no graphics
+
+ if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
+ f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
+ f.write("BOOT SUCCESS !!! \n")
+ # add commands here
+ f.write("reset -s\n")
+ f.close()
+
+ ret = RunCmd(cmd, args)
+
+ if ret == 0xc0000005:
+ # for some reason getting a c0000005 on successful return
+ return 0
+
+ return ret
diff --git a/ArmVirtPkg/README-pytools.md b/ArmVirtPkg/README-pytools.md
new file mode 100644
index 000000000000..ea70018e510a
--- /dev/null
+++ b/ArmVirtPkg/README-pytools.md
@@ -0,0 +1,123 @@
+# ArmVirtPkg
+
+This README-pytools.md summarizes the current state of Platform CI for ArmVirtPkg. It also describes how to _build_ ArmVirtPkg using the Pytools build system.
+
+## Platform CI Current Status
+
+<table>
+ <tr>
+ <th>Config</th>
+ <th colspan="3">Build & Run</th>
+ <th>Notes</th>
+ </tr>
+ <tr>
+ <th></th>
+ <th>DEBUG</th>
+ <th>RELEASE</th>
+ <th>NOOPT</th>
+ <th></th>
+ </tr>
+ <tr>
+ <th colspan="5" align="left">
+ Ubuntu 18.04 GCC5
+ </th>
+ </tr>
+ <tr>
+ <td>AARCH64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>ARM</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+</table>
+
+### Config Details
+
+| Config | Architectures |Additional Flags |
+| :---- | :----- | :---- |
+| AARCH64 | AARCH64 | None |
+| ARM | ARM | None |
+
+## Setup
+
+### The Usual EDK2 Build Setup
+
+- [Python 3.8.x - Download & Install](https://www.python.org/downloads/)
+- [GIT - Download & Install](https://git-scm.com/download/)
+- [GIT - Configure for EDK II](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#github-help)
+- [QEMU - Download, Install, and add to your path](https://www.qemu.org/download/)
+- [EDKII Source - Download/Checkout from Github](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#download)
+ - **NOTE:** Do _not_ follow the EDK II Compile Tools and Build instructions, see below...
+
+### Differences from EDK Classic Build Setup
+
+- Build BaseTools using "`C:\git\edk2>python BaseTools\Edk2ToolsBuild.py [-t <ToolChainTag>]`"
+ - This replaces "`edksetup Rebuild`" from the classic build system
+ - For Windows `<ToolChainTag>` examples, refer to [Windows ToolChain Matrix](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems-ToolChain-Matrix), defaults to `VS2017` if not specified
+- **No Action:** Submodule initialization and manual installation/setup of NASM and iASL is **not** required, it is handled by the PyTools build system
+
+### Building with Pytools for ArmVirtPkg
+
+- Install Pytools
+ - `pip install --upgrade -r pip-requirements.txt`
+- Initialize & Update Submodules
+ - `stuart_setup -c ArmVirt\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
+- Initialize & Update Dependencies (e.g. iASL, NASM & GCC Arm/Aarch64 Compilers)
+ - `stuart_update -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
+- Compile (AARCH64 supported / ARM support coming soon)
+ - `stuart_build -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
+- Running Emulator
+ - You can add `--FlashRom` to the end of your build command and the emulator will run after the build is complete.
+ - or use the FlashOnly feature like `stuart_build -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH> --FlashOnly` to just run the emulator.
+
+### Notes
+
+1. Including the expected build architecture and toolchain to the _stuart_update_ command is critical. This is because there are extra scopes and tools that will be resolved during the update step that need to match your build step.
+2. Configuring _ACTIVE_PLATFORM_ and _TARGET_ARCH_ in Conf/target.txt is _not_ required. This environment is set by PlatformBuild.py based upon the "`[-a <TARGET_ARCH>]`" parameter.
+3. QEMU must be on your path. On Windows this is a manual process and not part of the QEMU installer.
+
+**NOTE:** Logging the execution output will be in the normal stuart log as well as to your console.
+
+## Custom Build Options
+
+**MAKE_STARTUP_NSH=TRUE** will output a _startup.nsh_ file to the location mapped as fs0. This is used in CI in combination with the --FlashOnly feature to run QEMU to the UEFI shell and then execute the contents of startup.nsh.
+
+**QEMU_HEADLESS=TRUE** Since CI servers run headless QEMU must be told to run with no display otherwise an error occurs. Locally you don't need to set this.
+
+## Passing Build Defines
+
+To pass build defines through _stuart_build_, prepend `BLD_*_`to the define name and pass it on the command-line. _stuart_build_ currently requires values to be assigned, so add an`=1` suffix for bare defines.
+For example, to enable the TPM2 support, instead of the traditional "-D TPM2_ENABLE=TRUE", the stuart_build command-line would be:
+
+`stuart_build -c ArmVirtPkg/PlatformBuild.py BLD_*_TPM2_ENABLE=TRUE`
+
+## References
+
+- [Installing Pytools](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md#installing)
+- For each workspace, consider creating & using a [Python Virtual Environment](https://docs.python.org/3/library/venv.html). For example <https://microsoft.github.io/mu/CodeDevelopment/prerequisites/#workspace-virtual-environment-setup-process>
+- [stuart_build command-line parser](https://github.com/tianocore/edk2-pytool-extensions/blob/56f6a7aee09995c2f22da4765e8b0a29c1cbf5de/edk2toolext/edk2_invocable.py#L109)
diff --git a/ArmVirtPkg/iasl_ext_dep.yaml b/ArmVirtPkg/iasl_ext_dep.yaml
new file mode 100644
index 000000000000..8869ed3ecef1
--- /dev/null
+++ b/ArmVirtPkg/iasl_ext_dep.yaml
@@ -0,0 +1,21 @@
+## @file
+# Download iasl executable tool from a nuget.org package
+# - package contains different binaries based on host
+# Add the folder with the tool to the path
+#
+# This is only downloaded for scope armvirt thus
+# should have no impact on the asl compiler used by any
+# other platform build
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "id": "iasl-armvirt-1",
+ "scope": "armvirt",
+ "type": "nuget",
+ "name": "iasl",
+ "source": "https://api.nuget.org/v3/index.json",
+ "version": "20190215.0.0",
+ "flags": ["set_path", "host_specific"],
+}
--
2.16.3.windows.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-08 18:13 ` [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI Michael Kubacki
@ 2020-04-09 6:47 ` Laszlo Ersek
2020-04-09 8:05 ` Ard Biesheuvel
2020-04-09 9:17 ` Laszlo Ersek
2 siblings, 0 replies; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-09 6:47 UTC (permalink / raw)
To: michael.kubacki, devel; +Cc: Ard Biesheuvel, Leif Lindholm
On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
> From: Sean Brogan <sean.brogan@microsoft.com>
>
> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
>
> Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
> * Ubuntu GCC5
> Add PyTool based build of ArmVirtPkg
> Add extdep for managing the iasl dependency
> Add ArmVirtPkg.ci.yaml for Core CI
> Add README-pytools for details and instructions
>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Cc: Leif Lindholm <leif@nuviainc.com>
> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
> ---
> ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml | 89 +++++++
> ArmVirtPkg/ArmVirtPkg.ci.yaml | 103 ++++++++
> ArmVirtPkg/PlatformBuild.py | 263 ++++++++++++++++++++
> ArmVirtPkg/README-pytools.md | 123 +++++++++
> ArmVirtPkg/iasl_ext_dep.yaml | 21 ++
> 5 files changed, 599 insertions(+)
Can we move as many as possible from these files under a separate
subdirectory? For example, "PlatformBuild.py" seems to be referenced
explicitly by the other files, so I guess at least "PlatformBuild.py"
could be "tucked away".
My intent with this request is two-fold:
- collect CI-related files ("metafiles") in their own dedicated space,
- when someone runs "ls -l" in the package root dir, prevent them from
thinking that e.g. "PlatformBuild.py" or "iasl_ext_dep.yaml" are somehow
necessary (or useful) for building locally.
Otherwise I'm ready to ACK this patch!
Thanks,
Laszlo
>
> diff --git a/ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml
> new file mode 100644
> index 000000000000..ef5b61a0d4df
> --- /dev/null
> +++ b/ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml
> @@ -0,0 +1,89 @@
> +## @file
> +# Azure Pipeline build file for building a platform.
> +#
> +# Platform: ArmVirtQemu
> +# OS: Ubuntu
> +# Toolchain: GCC5
> +#
> +# Copyright (c) Microsoft Corporation.
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +##
> +trigger:
> + - master
> +pr:
> + - master
> +
> +jobs:
> + - job: Platform_CI
> + variables:
> + package: 'ArmVirtPkg'
> + vm_image: 'ubuntu-latest'
> + should_run: true
> + run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
> +
> + #Use matrix to speed up the build process
> + strategy:
> + matrix:
> + QEMU_AARCH64_DEBUG:
> + Build.File: "$(package)/PlatformBuild.py"
> + Build.Arch: "AARCH64"
> + Build.Flags: ""
> + Build.Target: "DEBUG"
> + Run.Flags: $(run_flags)
> + Run: $(should_run)
> + QEMU_AARCH64_RELEASE:
> + Build.File: "$(package)/PlatformBuild.py"
> + Build.Arch: "AARCH64"
> + Build.Flags: ""
> + Build.Target: "RELEASE"
> + Run.Flags: $(run_flags)
> + Run: $(should_run)
> + QEMU_AARCH64_NOOPT:
> + Build.File: "$(package)/PlatformBuild.py"
> + Build.Arch: "AARCH64"
> + Build.Flags: ""
> + Build.Target: "NOOPT"
> + Run.Flags: $(run_flags)
> + Run: $(should_run)
> + QEMU_ARM_DEBUG:
> + Build.File: "$(package)/PlatformBuild.py"
> + Build.Arch: "ARM"
> + Build.Flags: ""
> + Build.Target: "DEBUG"
> + Run.Flags: $(run_flags)
> + Run: $(should_run)
> + QEMU_ARM_RELEASE:
> + Build.File: "$(package)/PlatformBuild.py"
> + Build.Arch: "ARM"
> + Build.Flags: ""
> + Build.Target: "RELEASE"
> + Run.Flags: $(run_flags)
> + Run: $(should_run)
> + QEMU_ARM_NOOPT:
> + Build.File: "$(package)/PlatformBuild.py"
> + Build.Arch: "ARM"
> + Build.Flags: ""
> + Build.Target: "NOOPT"
> + Run.Flags: $(run_flags)
> + Run: $(should_run)
> +
> + workspace:
> + clean: all
> +
> + pool:
> + vmImage: $(vm_image)
> +
> + steps:
> + - template: ../../.azurepipelines/templates/platform-build-run-steps.yml
> + parameters:
> + tool_chain_tag: GCC5
> + build_pkg: $(package)
> + build_target: $(Build.Target)
> + build_arch: $(Build.Arch)
> + build_file: $(Build.File)
> + build_flags: $(Build.Flags)
> + run_flags: $(Run.Flags)
> + extra_install_step:
> + - bash: sudo apt-get install qemu
> + displayName: Install qemu
> + condition: and(gt(variables.pkg_count, 0), succeeded())
> diff --git a/ArmVirtPkg/ArmVirtPkg.ci.yaml b/ArmVirtPkg/ArmVirtPkg.ci.yaml
> new file mode 100644
> index 000000000000..4553725ee528
> --- /dev/null
> +++ b/ArmVirtPkg/ArmVirtPkg.ci.yaml
> @@ -0,0 +1,103 @@
> +## @file
> +# Core CI configuration for ArmVirtPkg
> +#
> +# ArmVirtPkg is part of Platform Ci for builds so this is only
> +# used for code analysis.
> +#
> +# Copyright (c) Microsoft Corporation
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +##
> +{
> + ## options defined .pytool/Plugin/CompilerPlugin
> + "CompilerPlugin": {
> + "DscPath": "" # Don't support this test
> + },
> +
> + ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin
> + "HostUnitTestCompilerPlugin": {
> + "DscPath": "" # Don't support this test
> + },
> +
> + ## options defined .pytool/Plugin/CharEncodingCheck
> + "CharEncodingCheck": {
> + "IgnoreFiles": []
> + },
> +
> + ## options defined .pytool/Plugin/DependencyCheck
> + "DependencyCheck": {
> + "AcceptableDependencies": [
> + "MdePkg/MdePkg.dec",
> + "MdeModulePkg/MdeModulePkg.dec",
> + "ArmVirtPkg/ArmVirtPkg.dec",
> + "NetworkPkg/NetworkPkg.dec",
> + "ArmPkg/ArmPkg.dec",
> + "OvmfPkg/OvmfPkg.dec",
> + "EmbeddedPkg/EmbeddedPkg.dec",
> + "ArmPlatformPkg/ArmPlatformPkg.dec",
> + "SecurityPkg/SecurityPkg.dec",
> + "ShellPkg/ShellPkg.dec" #Is this ok?
> + ],
> + # For host based unit tests
> + "AcceptableDependencies-HOST_APPLICATION":[
> + "UnitTestFrameworkPkg/UnitTestFrameworkPkg.dec"
> + ],
> + # For UEFI shell based apps
> + "AcceptableDependencies-UEFI_APPLICATION":[
> +
> + ],
> + "IgnoreInf": []
> + },
> +
> + ## options defined .pytool/Plugin/DscCompleteCheck
> + "DscCompleteCheck": {
> + "IgnoreInf": [""],
> + "DscPath": "" # Don't support this test
> + },
> +
> + ## options defined .pytool/Plugin/HostUnitTestDscCompleteCheck
> + "HostUnitTestDscCompleteCheck": {
> + "IgnoreInf": [""],
> + "DscPath": "" # Don't support this test
> + },
> +
> + ## options defined .pytool/Plugin/GuidCheck
> + "GuidCheck": {
> + "IgnoreGuidName": [],
> + "IgnoreGuidValue": [],
> + "IgnoreFoldersAndFiles": [],
> + "IgnoreDuplicates": [],
> + },
> +
> + ## options defined .pytool/Plugin/LibraryClassCheck
> + "LibraryClassCheck": {
> + "IgnoreHeaderFile": []
> + },
> +
> + ## options defined .pytool/Plugin/SpellCheck
> + "SpellCheck": {
> + "AuditOnly": False, # Fails right now with over 270 errors
> + "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
> + "ExtendWords": [
> + "setjump",
> + "plong",
> + "lparam",
> + "lpdword",
> + "lpthread",
> + "lresult",
> + "bootable",
> + "bsymbolic",
> + "endiannness",
> + "fvmain",
> + "multiboot",
> + "qemu's",
> + "ramdisk",
> + "ramfb",
> + "unbootable",
> + "virt's",
> + "werror",
> + "xenio"
> + ], # words to extend to the dictionary for this package
> + "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
> + "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
> + }
> +}
> diff --git a/ArmVirtPkg/PlatformBuild.py b/ArmVirtPkg/PlatformBuild.py
> new file mode 100644
> index 000000000000..c2c5cd098e7e
> --- /dev/null
> +++ b/ArmVirtPkg/PlatformBuild.py
> @@ -0,0 +1,263 @@
> +# @file
> +# Script to Build ArmVirtPkg UEFI firmware
> +#
> +# Copyright (c) Microsoft Corporation.
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +##
> +import os
> +import logging
> +
> +from edk2toolext.environment import shell_environment
> +from edk2toolext.environment.uefi_build import UefiBuilder
> +from edk2toolext.invocables.edk2_platform_build import BuildSettingsManager
> +from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule
> +from edk2toolext.invocables.edk2_update import UpdateSettingsManager
> +from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager
> +from edk2toollib.utility_functions import RunCmd
> +from edk2toollib.utility_functions import GetHostInfo
> +
> +# ####################################################################################### #
> +# Common Configuration #
> +# ####################################################################################### #
> +
> +
> +class CommonPlatform():
> + ''' Common settings for this platform. Define static data here and use
> + for the different parts of stuart
> + '''
> + PackagesSupported = ("ArmVirtPkg",)
> + ArchSupported = ("AARCH64", "ARM")
> + TargetsSupported = ("DEBUG", "RELEASE", "NOOPT")
> + Scopes = ('armvirt', 'edk2-build')
> + WorkspaceRoot = os.path.realpath(os.path.join(
> + os.path.dirname(os.path.abspath(__file__)), ".."))
> +
> + # ####################################################################################### #
> + # Configuration for Update & Setup #
> + # ####################################################################################### #
> +
> +
> +class SettingsManager(UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager):
> +
> + def GetPackagesSupported(self):
> + ''' return iterable of edk2 packages supported by this build.
> + These should be edk2 workspace relative paths '''
> + return CommonPlatform.PackagesSupported
> +
> + def GetArchitecturesSupported(self):
> + ''' return iterable of edk2 architectures supported by this build '''
> + return CommonPlatform.ArchSupported
> +
> + def GetTargetsSupported(self):
> + ''' return iterable of edk2 target tags supported by this build '''
> + return CommonPlatform.TargetsSupported
> +
> + def GetRequiredSubmodules(self):
> + ''' return iterable containing RequiredSubmodule objects.
> + If no RequiredSubmodules return an empty iterable
> + '''
> + rs = []
> + rs.append(RequiredSubmodule(
> + "ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3", False))
> + rs.append(RequiredSubmodule(
> + "CryptoPkg/Library/OpensslLib/openssl", False))
> + return rs
> +
> + def SetArchitectures(self, list_of_requested_architectures):
> + ''' Confirm the requests architecture list is valid and configure SettingsManager
> + to run only the requested architectures.
> +
> + Raise Exception if a list_of_requested_architectures is not supported
> + '''
> + unsupported = set(list_of_requested_architectures) - \
> + set(self.GetArchitecturesSupported())
> + if(len(unsupported) > 0):
> + errorString = (
> + "Unsupported Architecture Requested: " + " ".join(unsupported))
> + logging.critical(errorString)
> + raise Exception(errorString)
> + self.ActualArchitectures = list_of_requested_architectures
> +
> + def GetWorkspaceRoot(self):
> + ''' get WorkspacePath '''
> + return CommonPlatform.WorkspaceRoot
> +
> + def GetActiveScopes(self):
> + ''' return tuple containing scopes that should be active for this process '''
> +
> + scopes = CommonPlatform.Scopes
> + ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "")
> +
> + if GetHostInfo().os.upper() == "LINUX" and ActualToolChainTag.upper().startswith("GCC"):
> + if "AARCH64" in self.ActualArchitectures:
> + scopes += ("gcc_aarch64_linux",)
> + if "ARM" in self.ActualArchitectures:
> + scopes += ("gcc_arm_linux",)
> + return scopes
> +
> + def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list:
> + ''' Filter other cases that this package should be built
> + based on changed files. This should cover things that can't
> + be detected as dependencies. '''
> + build_these_packages = []
> + possible_packages = potentialPackagesList.copy()
> + for f in changedFilesList:
> + # BaseTools files that might change the build
> + if "BaseTools" in f:
> + if os.path.splitext(f) not in [".txt", ".md"]:
> + build_these_packages = possible_packages
> + break
> +
> + # if the azure pipeline file changed
> + if ".azurepipelines" in f and "ArmVirt" in f:
> + build_these_packages = possible_packages
> + break
> +
> + # if the azure pipeline platform template file changed
> + if ".azurepipelines" in f and "platforms" in f and "templates" in f:
> + build_these_packages = possible_packages
> + break
> +
> + return build_these_packages
> +
> + def GetPlatformDscAndConfig(self) -> tuple:
> + ''' If a platform desires to provide its DSC then Policy 4 will evaluate if
> + any of the changes will be built in the dsc.
> +
> + The tuple should be (<workspace relative path to dsc file>, <input dictionary of dsc key value pairs>)
> + '''
> + return (os.path.join("ArmVirtPkg", "ArmVirtQemu.dsc"), {})
> +
> +
> + # ####################################################################################### #
> + # Actual Configuration for Platform Build #
> + # ####################################################################################### #
> +
> +
> +class PlatformBuilder(UefiBuilder, BuildSettingsManager):
> + def __init__(self):
> + UefiBuilder.__init__(self)
> +
> + def AddCommandLineOptions(self, parserObj):
> + ''' Add command line options to the argparser '''
> + parserObj.add_argument('-a', "--arch", dest="build_arch", type=str, default="AARCH64",
> + help="Optional - Architecture to build. Default = AARCH64")
> +
> + def RetrieveCommandLineOptions(self, args):
> + ''' Retrieve command line options from the argparser '''
> +
> + shell_environment.GetBuildVars().SetValue(
> + "TARGET_ARCH", args.build_arch.upper(), "From CmdLine")
> +
> + shell_environment.GetBuildVars().SetValue(
> + "ACTIVE_PLATFORM", "ArmVirtPkg/ArmVirtQemu.dsc", "From CmdLine")
> +
> + def GetWorkspaceRoot(self):
> + ''' get WorkspacePath '''
> + return CommonPlatform.WorkspaceRoot
> +
> + def GetPackagesPath(self):
> + ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath '''
> + return ()
> +
> + def GetActiveScopes(self):
> + ''' return tuple containing scopes that should be active for this process '''
> + scopes = CommonPlatform.Scopes
> + ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "")
> + Arch = shell_environment.GetBuildVars().GetValue("TARGET_ARCH", "")
> +
> + if GetHostInfo().os.upper() == "LINUX" and ActualToolChainTag.upper().startswith("GCC"):
> + if "AARCH64" == Arch:
> + scopes += ("gcc_aarch64_linux",)
> + elif "ARM" == Arch:
> + scopes += ("gcc_arm_linux",)
> + return scopes
> +
> + def GetName(self):
> + ''' Get the name of the repo, platform, or product being build '''
> + ''' Used for naming the log file, among others '''
> + # check the startup nsh flag and if set then rename the log file.
> + # this helps in CI so we don't overwrite the build log since running
> + # uses the stuart_build command.
> + if(shell_environment.GetBuildVars().GetValue("MAKE_STARTUP_NSH", "FALSE") == "TRUE"):
> + return "ArmVirtPkg_With_Run"
> + return "ArmVirtPkg"
> +
> + def GetLoggingLevel(self, loggerType):
> + ''' Get the logging level for a given type
> + base == lowest logging level supported
> + con == Screen logging
> + txt == plain text file logging
> + md == markdown file logging
> + '''
> + return logging.DEBUG
> +
> + def SetPlatformEnv(self):
> + logging.debug("PlatformBuilder SetPlatformEnv")
> + self.env.SetValue("PRODUCT_NAME", "ArmVirtQemu", "Platform Hardcoded")
> + self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
> + self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
> + return 0
> +
> + def PlatformPreBuild(self):
> + return 0
> +
> + def PlatformPostBuild(self):
> + return 0
> +
> + def FlashRomImage(self):
> + VirtualDrive = os.path.join(self.env.GetValue(
> + "BUILD_OUTPUT_BASE"), "VirtualDrive")
> + os.makedirs(VirtualDrive, exist_ok=True)
> + OutputPath_FV = os.path.join(
> + self.env.GetValue("BUILD_OUTPUT_BASE"), "FV")
> + Built_FV = os.path.join(OutputPath_FV, "QEMU_EFI.fd")
> +
> + # pad fd to 64mb
> + with open(Built_FV, "ab") as fvfile:
> + fvfile.seek(0, os.SEEK_END)
> + additional = b'\0' * ((64 * 1024 * 1024)-fvfile.tell())
> + fvfile.write(additional)
> +
> + # QEMU must be on that path
> +
> + # Unique Command and Args parameters per ARCH
> + if (self.env.GetValue("TARGET_ARCH").upper() == "AARCH64"):
> + cmd = "qemu-system-aarch64"
> + args = "-M virt"
> + args += " -cpu cortex-a57" # emulate cpu
> + elif(self.env.GetValue("TARGET_ARCH").upper() == "ARM"):
> + cmd = "qemu-system-arm"
> + args = "-M virt"
> + args += " -cpu cortex-a15" # emulate cpu
> + else:
> + raise NotImplementedError()
> +
> + # Common Args
> + args += " -pflash " + Built_FV # path to fw
> + args += " -m 1024" # 1gb memory
> + # turn off network
> + args += " -net none"
> + # Serial messages out
> + args += " -serial stdio"
> + # Mount disk with startup.nsh
> + args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk"
> +
> + # Conditional Args
> + if (self.env.GetValue("QEMU_HEADLESS").upper() == "TRUE"):
> + args += " -display none" # no graphics
> +
> + if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
> + f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
> + f.write("BOOT SUCCESS !!! \n")
> + # add commands here
> + f.write("reset -s\n")
> + f.close()
> +
> + ret = RunCmd(cmd, args)
> +
> + if ret == 0xc0000005:
> + # for some reason getting a c0000005 on successful return
> + return 0
> +
> + return ret
> diff --git a/ArmVirtPkg/README-pytools.md b/ArmVirtPkg/README-pytools.md
> new file mode 100644
> index 000000000000..ea70018e510a
> --- /dev/null
> +++ b/ArmVirtPkg/README-pytools.md
> @@ -0,0 +1,123 @@
> +# ArmVirtPkg
> +
> +This README-pytools.md summarizes the current state of Platform CI for ArmVirtPkg. It also describes how to _build_ ArmVirtPkg using the Pytools build system.
> +
> +## Platform CI Current Status
> +
> +<table>
> + <tr>
> + <th>Config</th>
> + <th colspan="3">Build & Run</th>
> + <th>Notes</th>
> + </tr>
> + <tr>
> + <th></th>
> + <th>DEBUG</th>
> + <th>RELEASE</th>
> + <th>NOOPT</th>
> + <th></th>
> + </tr>
> + <tr>
> + <th colspan="5" align="left">
> + Ubuntu 18.04 GCC5
> + </th>
> + </tr>
> + <tr>
> + <td>AARCH64</td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_DEBUG"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_RELEASE"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_NOOPT"/></a>
> + </td>
> + <td></td>
> + </tr>
> + <tr>
> + <td>ARM</td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_DEBUG"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_RELEASE"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_NOOPT"/></a>
> + </td>
> + <td></td>
> + </tr>
> +</table>
> +
> +### Config Details
> +
> +| Config | Architectures |Additional Flags |
> +| :---- | :----- | :---- |
> +| AARCH64 | AARCH64 | None |
> +| ARM | ARM | None |
> +
> +## Setup
> +
> +### The Usual EDK2 Build Setup
> +
> +- [Python 3.8.x - Download & Install](https://www.python.org/downloads/)
> +- [GIT - Download & Install](https://git-scm.com/download/)
> +- [GIT - Configure for EDK II](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#github-help)
> +- [QEMU - Download, Install, and add to your path](https://www.qemu.org/download/)
> +- [EDKII Source - Download/Checkout from Github](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#download)
> + - **NOTE:** Do _not_ follow the EDK II Compile Tools and Build instructions, see below...
> +
> +### Differences from EDK Classic Build Setup
> +
> +- Build BaseTools using "`C:\git\edk2>python BaseTools\Edk2ToolsBuild.py [-t <ToolChainTag>]`"
> + - This replaces "`edksetup Rebuild`" from the classic build system
> + - For Windows `<ToolChainTag>` examples, refer to [Windows ToolChain Matrix](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems-ToolChain-Matrix), defaults to `VS2017` if not specified
> +- **No Action:** Submodule initialization and manual installation/setup of NASM and iASL is **not** required, it is handled by the PyTools build system
> +
> +### Building with Pytools for ArmVirtPkg
> +
> +- Install Pytools
> + - `pip install --upgrade -r pip-requirements.txt`
> +- Initialize & Update Submodules
> + - `stuart_setup -c ArmVirt\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
> +- Initialize & Update Dependencies (e.g. iASL, NASM & GCC Arm/Aarch64 Compilers)
> + - `stuart_update -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
> +- Compile (AARCH64 supported / ARM support coming soon)
> + - `stuart_build -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
> +- Running Emulator
> + - You can add `--FlashRom` to the end of your build command and the emulator will run after the build is complete.
> + - or use the FlashOnly feature like `stuart_build -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH> --FlashOnly` to just run the emulator.
> +
> +### Notes
> +
> +1. Including the expected build architecture and toolchain to the _stuart_update_ command is critical. This is because there are extra scopes and tools that will be resolved during the update step that need to match your build step.
> +2. Configuring _ACTIVE_PLATFORM_ and _TARGET_ARCH_ in Conf/target.txt is _not_ required. This environment is set by PlatformBuild.py based upon the "`[-a <TARGET_ARCH>]`" parameter.
> +3. QEMU must be on your path. On Windows this is a manual process and not part of the QEMU installer.
> +
> +**NOTE:** Logging the execution output will be in the normal stuart log as well as to your console.
> +
> +## Custom Build Options
> +
> +**MAKE_STARTUP_NSH=TRUE** will output a _startup.nsh_ file to the location mapped as fs0. This is used in CI in combination with the --FlashOnly feature to run QEMU to the UEFI shell and then execute the contents of startup.nsh.
> +
> +**QEMU_HEADLESS=TRUE** Since CI servers run headless QEMU must be told to run with no display otherwise an error occurs. Locally you don't need to set this.
> +
> +## Passing Build Defines
> +
> +To pass build defines through _stuart_build_, prepend `BLD_*_`to the define name and pass it on the command-line. _stuart_build_ currently requires values to be assigned, so add an`=1` suffix for bare defines.
> +For example, to enable the TPM2 support, instead of the traditional "-D TPM2_ENABLE=TRUE", the stuart_build command-line would be:
> +
> +`stuart_build -c ArmVirtPkg/PlatformBuild.py BLD_*_TPM2_ENABLE=TRUE`
> +
> +## References
> +
> +- [Installing Pytools](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md#installing)
> +- For each workspace, consider creating & using a [Python Virtual Environment](https://docs.python.org/3/library/venv.html). For example <https://microsoft.github.io/mu/CodeDevelopment/prerequisites/#workspace-virtual-environment-setup-process>
> +- [stuart_build command-line parser](https://github.com/tianocore/edk2-pytool-extensions/blob/56f6a7aee09995c2f22da4765e8b0a29c1cbf5de/edk2toolext/edk2_invocable.py#L109)
> diff --git a/ArmVirtPkg/iasl_ext_dep.yaml b/ArmVirtPkg/iasl_ext_dep.yaml
> new file mode 100644
> index 000000000000..8869ed3ecef1
> --- /dev/null
> +++ b/ArmVirtPkg/iasl_ext_dep.yaml
> @@ -0,0 +1,21 @@
> +## @file
> +# Download iasl executable tool from a nuget.org package
> +# - package contains different binaries based on host
> +# Add the folder with the tool to the path
> +#
> +# This is only downloaded for scope armvirt thus
> +# should have no impact on the asl compiler used by any
> +# other platform build
> +#
> +# Copyright (c) Microsoft Corporation.
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +##
> +{
> + "id": "iasl-armvirt-1",
> + "scope": "armvirt",
> + "type": "nuget",
> + "name": "iasl",
> + "source": "https://api.nuget.org/v3/index.json",
> + "version": "20190215.0.0",
> + "flags": ["set_path", "host_specific"],
> +}
>
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-08 18:13 ` [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI Michael Kubacki
2020-04-09 6:47 ` Laszlo Ersek
@ 2020-04-09 8:05 ` Ard Biesheuvel
2020-04-09 19:57 ` [edk2-devel] " Sean
2020-04-09 9:17 ` Laszlo Ersek
2 siblings, 1 reply; 29+ messages in thread
From: Ard Biesheuvel @ 2020-04-09 8:05 UTC (permalink / raw)
To: michael.kubacki, devel; +Cc: Laszlo Ersek, Leif Lindholm
On 4/8/20 8:13 PM, michael.kubacki@outlook.com wrote:
> From: Sean Brogan <sean.brogan@microsoft.com>
>
> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
>
> Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
> * Ubuntu GCC5
> Add PyTool based build of ArmVirtPkg
> Add extdep for managing the iasl dependency
> Add ArmVirtPkg.ci.yaml for Core CI
> Add README-pytools for details and instructions
>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Cc: Leif Lindholm <leif@nuviainc.com>
> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
> ---
> ArmVirtPkg/.azurepipelines/Ubuntu-GCC5.yml | 89 +++++++
> ArmVirtPkg/ArmVirtPkg.ci.yaml | 103 ++++++++
> ArmVirtPkg/PlatformBuild.py | 263 ++++++++++++++++++++
> ArmVirtPkg/README-pytools.md | 123 +++++++++
> ArmVirtPkg/iasl_ext_dep.yaml | 21 ++
> 5 files changed, 599 insertions(+)
>
Hello Michael,
I agree with Laszlo's point here: it would be nice if we can keep these
files organized a bit better, although I suppose there is already
precent for keep <package>.ci.yaml files at the package's root level, so
that one should stay where it is.
More comments below.
...
> diff --git a/ArmVirtPkg/README-pytools.md b/ArmVirtPkg/README-pytools.md
> new file mode 100644
> index 000000000000..ea70018e510a
> --- /dev/null
> +++ b/ArmVirtPkg/README-pytools.md
> @@ -0,0 +1,123 @@
> +# ArmVirtPkg
> +
> +This README-pytools.md summarizes the current state of Platform CI for ArmVirtPkg. It also describes how to _build_ ArmVirtPkg using the Pytools build system.
> +
> +## Platform CI Current Status
> +
> +<table>
> + <tr>
> + <th>Config</th>
> + <th colspan="3">Build & Run</th>
> + <th>Notes</th>
> + </tr>
> + <tr>
> + <th></th>
> + <th>DEBUG</th>
> + <th>RELEASE</th>
> + <th>NOOPT</th>
> + <th></th>
> + </tr>
> + <tr>
> + <th colspan="5" align="left">
> + Ubuntu 18.04 GCC5
> + </th>
> + </tr>
> + <tr>
> + <td>AARCH64</td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_DEBUG"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_RELEASE"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_AARCH64_NOOPT"/></a>
> + </td>
> + <td></td>
> + </tr>
> + <tr>
> + <td>ARM</td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_DEBUG"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_RELEASE"/></a>
> + </td>
> + <td>
> + <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=41&branchName=master">
> + <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/ArmVirtPkg/ArmVirtQemu%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20QEMU_ARM_NOOPT"/></a>
> + </td>
> + <td></td>
> + </tr>
> +</table>
> +
> +### Config Details
> +
> +| Config | Architectures |Additional Flags |
> +| :---- | :----- | :---- |
> +| AARCH64 | AARCH64 | None |
> +| ARM | ARM | None |
> +
> +## Setup
> +
> +### The Usual EDK2 Build Setup
> +
> +- [Python 3.8.x - Download & Install](https://www.python.org/downloads/)
> +- [GIT - Download & Install](https://git-scm.com/download/)
> +- [GIT - Configure for EDK II](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#github-help)
> +- [QEMU - Download, Install, and add to your path](https://www.qemu.org/download/)
> +- [EDKII Source - Download/Checkout from Github](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#download)
> + - **NOTE:** Do _not_ follow the EDK II Compile Tools and Build instructions, see below...
> +
> +### Differences from EDK Classic Build Setup
> +
> +- Build BaseTools using "`C:\git\edk2>python BaseTools\Edk2ToolsBuild.py [-t <ToolChainTag>]`"
> + - This replaces "`edksetup Rebuild`" from the classic build system
> + - For Windows `<ToolChainTag>` examples, refer to [Windows ToolChain Matrix](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems-ToolChain-Matrix), defaults to `VS2017` if not specified
> +- **No Action:** Submodule initialization and manual installation/setup of NASM and iASL is **not** required, it is handled by the PyTools build system
> +
> +### Building with Pytools for ArmVirtPkg
> +
> +- Install Pytools
> + - `pip install --upgrade -r pip-requirements.txt`
> +- Initialize & Update Submodules
> + - `stuart_setup -c ArmVirt\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
Typo here ^^^
Also, could we standardize on forward slashes everywhere (as in the
examples below)?
> +- Initialize & Update Dependencies (e.g. iASL, NASM & GCC Arm/Aarch64 Compilers)
> + - `stuart_update -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
> +- Compile (AARCH64 supported / ARM support coming soon)
> + - `stuart_build -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH>`
> +- Running Emulator
> + - You can add `--FlashRom` to the end of your build command and the emulator will run after the build is complete.
> + - or use the FlashOnly feature like `stuart_build -c ArmVirtPkg\PlatformBuild.py TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG> -a <TARGET_ARCH> --FlashOnly` to just run the emulator.
> +
> +### Notes
> +
> +1. Including the expected build architecture and toolchain to the _stuart_update_ command is critical. This is because there are extra scopes and tools that will be resolved during the update step that need to match your build step.
> +2. Configuring _ACTIVE_PLATFORM_ and _TARGET_ARCH_ in Conf/target.txt is _not_ required. This environment is set by PlatformBuild.py based upon the "`[-a <TARGET_ARCH>]`" parameter.
> +3. QEMU must be on your path. On Windows this is a manual process and not part of the QEMU installer.
> +
> +**NOTE:** Logging the execution output will be in the normal stuart log as well as to your console.
> +
> +## Custom Build Options
> +
> +**MAKE_STARTUP_NSH=TRUE** will output a _startup.nsh_ file to the location mapped as fs0. This is used in CI in combination with the --FlashOnly feature to run QEMU to the UEFI shell and then execute the contents of startup.nsh.
> +
> +**QEMU_HEADLESS=TRUE** Since CI servers run headless QEMU must be told to run with no display otherwise an error occurs. Locally you don't need to set this.
> +
> +## Passing Build Defines
> +
> +To pass build defines through _stuart_build_, prepend `BLD_*_`to the define name and pass it on the command-line. _stuart_build_ currently requires values to be assigned, so add an`=1` suffix for bare defines.
> +For example, to enable the TPM2 support, instead of the traditional "-D TPM2_ENABLE=TRUE", the stuart_build command-line would be:
> +
> +`stuart_build -c ArmVirtPkg/PlatformBuild.py BLD_*_TPM2_ENABLE=TRUE`
> +
> +## References
> +
> +- [Installing Pytools](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md#installing)
> +- For each workspace, consider creating & using a [Python Virtual Environment](https://docs.python.org/3/library/venv.html). For example <https://microsoft.github.io/mu/CodeDevelopment/prerequisites/#workspace-virtual-environment-setup-process>
> +- [stuart_build command-line parser](https://github.com/tianocore/edk2-pytool-extensions/blob/56f6a7aee09995c2f22da4765e8b0a29c1cbf5de/edk2toolext/edk2_invocable.py#L109)
> diff --git a/ArmVirtPkg/iasl_ext_dep.yaml b/ArmVirtPkg/iasl_ext_dep.yaml
> new file mode 100644
> index 000000000000..8869ed3ecef1
> --- /dev/null
> +++ b/ArmVirtPkg/iasl_ext_dep.yaml
> @@ -0,0 +1,21 @@
> +## @file
> +# Download iasl executable tool from a nuget.org package
> +# - package contains different binaries based on host
> +# Add the folder with the tool to the path
> +#
> +# This is only downloaded for scope armvirt thus
> +# should have no impact on the asl compiler used by any
> +# other platform build
> +#
> +# Copyright (c) Microsoft Corporation.
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +##
> +{
> + "id": "iasl-armvirt-1",
> + "scope": "armvirt",
> + "type": "nuget",
> + "name": "iasl",
> + "source": "https://api.nuget.org/v3/index.json",
> + "version": "20190215.0.0",
> + "flags": ["set_path", "host_specific"],
> +}
>
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 8:05 ` Ard Biesheuvel
@ 2020-04-09 19:57 ` Sean
2020-04-15 6:55 ` Sean
2020-04-15 17:18 ` Laszlo Ersek
0 siblings, 2 replies; 29+ messages in thread
From: Sean @ 2020-04-09 19:57 UTC (permalink / raw)
To: Ard Biesheuvel, devel
[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]
Ard/Laszlo,
On the topic of Package files.
1. The *.ci.yaml is designed to be at the root of the package and this follows all other Core ci usage. The Core CI will only look for this file at the root as of now.
2. The PlatformBuild.py file is exactly like a build.bat or build.sh file found in many platform packages. This file is relevant for local dev as it will build the package and should support the feature set of the platform. I would hope that the package maintainers would continue to update this file to match the features of their package otherwise CI will become less and less relevant overtime. Thus I think it belongs in the same place as build.sh. The only other pattern we could follow is a ".pytools" folder like for Core CI. But again unless you strongly disagree I would prefer the first.
3. The readme file...You tell me where? Or it could be combined with the existing readme.md file?
4. The iasl ext_dep file. I could see this moving deeper into the package. Where would you like it? This file is an external dependency to control the version of iasl used on this platform. It should be kept current with what the platform expects for iasl usage.
Anyway, I need more actionable feedback if you don't agree with the above.
On the forward slashes and typo. I agree and will get them in the next version. Thanks for the review.
Thanks
Sean
[-- Attachment #2: Type: text/html, Size: 1568 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 19:57 ` [edk2-devel] " Sean
@ 2020-04-15 6:55 ` Sean
2020-04-15 16:57 ` Laszlo Ersek
2020-04-15 17:18 ` Laszlo Ersek
1 sibling, 1 reply; 29+ messages in thread
From: Sean @ 2020-04-15 6:55 UTC (permalink / raw)
To: Sean, devel
[-- Attachment #1: Type: text/plain, Size: 297 bytes --]
Any thoughts? I would like to get this in before any more failures get checked in causing more dependencies.
I have a branch here: https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v7
but am waiting for more feedback to make V2 patchset.
Thanks
Sean
[-- Attachment #2: Type: text/html, Size: 451 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-15 6:55 ` Sean
@ 2020-04-15 16:57 ` Laszlo Ersek
2020-04-15 19:31 ` Sean
0 siblings, 1 reply; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-15 16:57 UTC (permalink / raw)
To: devel, sean.brogan; +Cc: Ard Biesheuvel
Hi Sean,
On 04/15/20 08:55, Sean via groups.io wrote:
> Any thoughts?
I have not ignored your email. Instead, I have not seen it.
The reason is that you messaged the list only -- you didn't put me in
the To: or Cc: headers, despite addressing me by name in the message body.
While I do follow the entire list, my personal inbox enjoys a much
higher priority than my edk2-devel list folder -- the latter is (almost)
just one of my many list folders. Therefore, if a query is sent only to
the list, and not copied to me, I could come to it much later, or (in
the worst case) I might even totally miss it. (E.g. if I were
overloaded, and the message were deeply nested in an existent thread
and/or the subject line didn't obviously call for my attention.)
General mailing list etiquette (not specific to edk2-devel) is that,
when asking a public question to a specific subscriber, then both the
list and that subscriber be copied on the message. This way all other
interested subscribers (and the list archive) can follow the discussion,
*and* the specific subscriber will not only find the query in their list
folder, but also in their inbox.
What's even worse is that, your email was sent in response to Ard's (the
one that said "I agree with Laszlo's point here"), and I *was* CC'd on
Ard's message. Which means that your MUA explicitly *dropped* me from
your message. I'm not sure what MUA does that -- I can even imagine it
was the groups.io WebUI! --, but whatever MUA it was, it was very wrong.
Dropping CC's *breaks* mailing list discussions.
Anyway, I'll now read your previous email and attempt an answer.
(To be clear, I've now seen your *current* message too only because I'm
at the moment a bit less tired, and happened to catch it in my list folder.)
Thanks
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-15 16:57 ` Laszlo Ersek
@ 2020-04-15 19:31 ` Sean
2020-04-16 15:12 ` Laszlo Ersek
0 siblings, 1 reply; 29+ messages in thread
From: Sean @ 2020-04-15 19:31 UTC (permalink / raw)
To: Laszlo Ersek, devel
[-- Attachment #1: Type: text/plain, Size: 965 bytes --]
On Wed, Apr 15, 2020 at 09:58 AM, Laszlo Ersek wrote:
>
> I have not ignored your email. Instead, I have not seen it.
>
> The reason is that you messaged the list only -- you didn't put me in
> the To: or Cc: headers, despite addressing me by name in the message body.
I use the groups.io web client for all mailing list activities. Did i tell you how much I love the mailing list. :) My corporate mail server changes the subject line which also destroys threading so I figured this was better. Maybe not as i have no control over the CC/To lines of the email that is sent. Do Bret's responses which add "[External]" (example here: https://edk2.groups.io/g/devel/message/57377) ( https://edk2.groups.io/g/devel/message/57377 ) to every reply show up better (in groups.io web ui this is harder to follow since it starts a new thread)?
Thanks for continuing to check the full list for messages as I appreciate your feedback.
Thanks
Sean
[-- Attachment #2: Type: text/html, Size: 1082 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-15 19:31 ` Sean
@ 2020-04-16 15:12 ` Laszlo Ersek
0 siblings, 0 replies; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-16 15:12 UTC (permalink / raw)
To: Sean, devel; +Cc: Ard Biesheuvel, Bret Barkelew
Hi Sean,
On 04/15/20 21:31, sean.brogan via [] wrote:
> My corporate mail server changes the subject line which also destroys
> threading so I figured this was better. Maybe not as i have no
> control over the CC/To lines of the email that is sent. Do Bret's
> responses which add "[External]" (example here:
> https://edk2.groups.io/g/devel/message/57377) (
> https://edk2.groups.io/g/devel/message/57377 ) to every reply show up
> better (in groups.io web ui this is harder to follow since it starts
> a new thread)?
Thank you very much for providing Bret's email as an example!
Bret's answers in that thread, namely:
https://edk2.groups.io/g/devel/message/57377
https://edk2.groups.io/g/devel/message/57381
are well-threaded. Both in my personal list folder, as presented by my
MUA (Thunderbird), and also in e.g. the following (secondary) mailing
list archive:
http://mid.mail-archive.com/CY4PR21MB074349715A82905EB420D7DBEFDA0@CY4PR21MB0743.namprd21.prod.outlook.com
http://mid.mail-archive.com/CY4PR21MB07431322730AF342BF898A74EFDA0@CY4PR21MB0743.namprd21.prod.outlook.com
If you open each one of those "mail-archive.com" links in separate
browser windows or tabs, and scroll to the bottom of both pages, you'll
see that Bret's answer's are correctly threaded against the rest of the
messages in that discussion.
The subject line is not directly relevant for threading. Some MUAs use
the subject as a fallback option for reconstructing threads, but the
primary linkage for threading is the In-Reply-To header of the response
email -- which is supposed to reference the Message-Id header of the
original email.
So, for example, Rebecca's thread starter carried the header
Message-ID: <62933ee8-7fb9-7cd3-08e5-6d6e7dac31e9@bsdio.com>
and Bret's reply correctly carried the header
In-Reply-To: <62933ee8-7fb9-7cd3-08e5-6d6e7dac31e9@bsdio.com>
It doesn't matter (for threading) that Bret's infrastructure changed the
subject. Sometimes people intentionally change the subject line, while
staying in the same thread -- that signals that they go off on a
tangent, but still want to stay on topic a little (or else they want to
preserve the linkage to the original discussion). For example, the
original subject could be
foo bar
Re: foo bar
Re: foo bar
and then someone could go off on a tangent (in the same thread!) like
this:
quux [was: foo bar]
Re: quux [was: foo bar]
So while the "[External]" stuff is not particularly useful or welcome,
regarding the list traffic, it is not a deal breaker. The threading does
remain correct. If the groups.io WebUI loses track of threading just
because of subject line changes, then that's a groups.io WebUI bug.
Bret's MUA also seems to correctly preserve CC's -- Rebecca's thread
starter CC'd Bob and Liming, and Bret's responses carry those CC's too.
If you could adopt Bret's way of interacting with the list, I think that
would be great!
Thanks!
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 19:57 ` [edk2-devel] " Sean
2020-04-15 6:55 ` Sean
@ 2020-04-15 17:18 ` Laszlo Ersek
2020-04-15 17:26 ` Ard Biesheuvel
2020-04-15 20:38 ` Sean
1 sibling, 2 replies; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-15 17:18 UTC (permalink / raw)
To: devel, sean.brogan, Ard Biesheuvel
On 04/09/20 21:57, Sean via groups.io wrote:
> Ard/Laszlo,
>
> On the topic of Package files.
> 1. The *.ci.yaml is designed to be at the root of the package and this follows all other Core ci usage. The Core CI will only look for this file at the root as of now.
OK. This makes sense to me. Even if we wanted to put the files away
under a subdirectory, we'd have to inform the Core CI system somehow --
and that would likely take the form of having to name the subdirectory
in a particular way. Which is not different from having a fixed name
regular file in the package root dir.
Thankfully, this file does have ".ci." in the name.
> 2. The PlatformBuild.py file is exactly like a build.bat or build.sh file found in many platform packages. This file is relevant for local dev as it will build the package and should support the feature set of the platform. I would hope that the package maintainers would continue to update this file to match the features of their package otherwise CI will become less and less relevant overtime. Thus I think it belongs in the same place as build.sh.
Thank you, this explanation is very helpful!
ArmVirtPkg does not have a "build.sh" of the kind that you mention --
and I strongly prefer it that way.
OvmfPkg happens to have a "build.sh" -- and if it were up to me, I would
remove it. I have stated this before, perhaps multiple times. I strongly
prefer people invoking "build" manually, or writing their *own* scripts
for wrapping the "build" utility. I never ever use "OvmfPkg/build.sh"
myself -- it doesn't do what I want (and what I want, in this aspect,
should not be in the upstream edk2 tree).
But, again, OVMF's "build.sh" was added in commit 66325870afaf
("OvmfPkg/build.sh: Add features and replace build32/64.sh",
2011-01-09), and by the time I "arrived" and had grown a distaste for
it, other users had become dependent on it. So I can not propose its
removal. I can certainly argue against spreading the practice to ArmVirtPkg.
I *do* agree that "PlatformBuild.py" is necessary for CI, and keeping it
up-to-date is also necessary for good coverage by CI. I'm just
requesting to move it out of the package root dir.
> The only other pattern we could follow is a ".pytools" folder like for Core CI. But again unless you strongly disagree I would prefer the first.
The project root already has a .pytool subdir, so (I think?) there would
be a precedent.
But, the leading dot (for "hiding" the subdirectory) is totally not my
point. I don't wish to hide the CI metafiles to *that* extent. I'd just
like to clearly associate them with the CI purpose -- it's fine if they
are visible when the directory is listed with default switches, but the
names should be specific.
> 3. The readme file...You tell me where? Or it could be combined with the existing readme.md file?
> 4. The iasl ext_dep file. I could see this moving deeper into the package. Where would you like it? This file is an external dependency to control the version of iasl used on this platform. It should be kept current with what the platform expects for iasl usage.
>
> Anyway, I need more actionable feedback if you don't agree with the above.
How do you feel about the following directory structure:
ArmVirtPkg/ArmVirtPkg.ci.yaml
ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
ArmVirtPkg/PlatformCI/PlatformBuild.py
ArmVirtPkg/PlatformCI/README-pytools.md
ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
(Note that I also removed the ".azurepipelines/" prefix from
"Ubuntu-GCC5.yml", in addition to pushing it under "PlatformCI/". If
".azurepipelines/" is required, then I'm fine with it, as long as we can
also keep it under "ArmVirtPkg/PlatformCI/".)
The result would be that someone running "ls" or "dir" in ArmVirtPkg
would see two CI-related entries: the "ArmVirtPkg.ci.yaml" control file,
and the "PlatformCI" directory. Both entries clearly state "CI" in the
name, and separate CI objects from the rest of the package.
Thanks!
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-15 17:18 ` Laszlo Ersek
@ 2020-04-15 17:26 ` Ard Biesheuvel
2020-04-15 20:38 ` Sean
1 sibling, 0 replies; 29+ messages in thread
From: Ard Biesheuvel @ 2020-04-15 17:26 UTC (permalink / raw)
To: Laszlo Ersek, devel, sean.brogan
On 4/15/20 7:18 PM, Laszlo Ersek wrote:
> On 04/09/20 21:57, Sean via groups.io wrote:
>> Ard/Laszlo,
>>
>> On the topic of Package files.
>> 1. The *.ci.yaml is designed to be at the root of the package and this follows all other Core ci usage. The Core CI will only look for this file at the root as of now.
>
> OK. This makes sense to me. Even if we wanted to put the files away
> under a subdirectory, we'd have to inform the Core CI system somehow --
> and that would likely take the form of having to name the subdirectory
> in a particular way. Which is not different from having a fixed name
> regular file in the package root dir.
>
> Thankfully, this file does have ".ci." in the name.
>
>> 2. The PlatformBuild.py file is exactly like a build.bat or build.sh file found in many platform packages. This file is relevant for local dev as it will build the package and should support the feature set of the platform. I would hope that the package maintainers would continue to update this file to match the features of their package otherwise CI will become less and less relevant overtime. Thus I think it belongs in the same place as build.sh.
>
> Thank you, this explanation is very helpful!
>
> ArmVirtPkg does not have a "build.sh" of the kind that you mention --
> and I strongly prefer it that way.
>
Agreed. I never added one because I never felt the need for it. 'build'
has a standard interface across platforms and works in a uniform way.
build.sh are bespoke helpers, and so I don't think they should have been
added in the first place.
> OvmfPkg happens to have a "build.sh" -- and if it were up to me, I would
> remove it. I have stated this before, perhaps multiple times. I strongly
> prefer people invoking "build" manually, or writing their *own* scripts
> for wrapping the "build" utility. I never ever use "OvmfPkg/build.sh"
> myself -- it doesn't do what I want (and what I want, in this aspect,
> should not be in the upstream edk2 tree).
>
> But, again, OVMF's "build.sh" was added in commit 66325870afaf
> ("OvmfPkg/build.sh: Add features and replace build32/64.sh",
> 2011-01-09), and by the time I "arrived" and had grown a distaste for
> it, other users had become dependent on it. So I can not propose its
> removal. I can certainly argue against spreading the practice to ArmVirtPkg.
>
> I *do* agree that "PlatformBuild.py" is necessary for CI, and keeping it
> up-to-date is also necessary for good coverage by CI. I'm just
> requesting to move it out of the package root dir.
>
Agreed. And if it has a uniform interface across packages, even better.
>> The only other pattern we could follow is a ".pytools" folder like for Core CI. But again unless you strongly disagree I would prefer the first.
>
> The project root already has a .pytool subdir, so (I think?) there would
> be a precedent.
>
> But, the leading dot (for "hiding" the subdirectory) is totally not my
> point. I don't wish to hide the CI metafiles to *that* extent. I'd just
> like to clearly associate them with the CI purpose -- it's fine if they
> are visible when the directory is listed with default switches, but the
> names should be specific.
>
>> 3. The readme file...You tell me where? Or it could be combined with the existing readme.md file?
>> 4. The iasl ext_dep file. I could see this moving deeper into the package. Where would you like it? This file is an external dependency to control the version of iasl used on this platform. It should be kept current with what the platform expects for iasl usage.
>>
>> Anyway, I need more actionable feedback if you don't agree with the above.
>
> How do you feel about the following directory structure:
>
> ArmVirtPkg/ArmVirtPkg.ci.yaml
> ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
> ArmVirtPkg/PlatformCI/PlatformBuild.py
> ArmVirtPkg/PlatformCI/README-pytools.md
> ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
>
This looks good to me, and could be used for other packages as well, afaict.
--
Ard.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-15 17:18 ` Laszlo Ersek
2020-04-15 17:26 ` Ard Biesheuvel
@ 2020-04-15 20:38 ` Sean
2020-04-16 14:51 ` Laszlo Ersek
1 sibling, 1 reply; 29+ messages in thread
From: Sean @ 2020-04-15 20:38 UTC (permalink / raw)
To: Laszlo Ersek, devel
[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]
On Wed, Apr 15, 2020 at 10:18 AM, Laszlo Ersek wrote:
>
> ArmVirtPkg/ArmVirtPkg.ci.yaml
> ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
> ArmVirtPkg/PlatformCI/PlatformBuild.py
> ArmVirtPkg/PlatformCI/README-pytools.md
> ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
I am ok with the above except one thought on the readme. One nice thing about the markdown readme files are the badge shows up in github when you view the package. This is a quick and easy way to see the current status.
* Ovmf has a pretty stale readme and does not take advantage of markdown. We could convert it to MD, clean up, and then merge in the content from the pytools.md. I would need help or a package maintainer to do the cleanup of the readme to make sure it contained the content you desired.
* ArmVirtPkg doesn't have a readme and this is definitely a barrier to entry for the package. I would suggest creating one and then merging in the content from the pytools.md.
* EmulatorPkg has one. I would just suggest a merge but i am yet to get any feedback from those maintainers.
If that isn't desirable i would at least suggest we change the title to just ReadMe.md so that GitHub shows it by default when the PaltformCI folder is viewed form the web or in editor like vscode.
Thanks
Sean
[-- Attachment #2: Type: text/html, Size: 1460 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [edk2-devel] [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-15 20:38 ` Sean
@ 2020-04-16 14:51 ` Laszlo Ersek
0 siblings, 0 replies; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-16 14:51 UTC (permalink / raw)
To: Sean; +Cc: devel, Ard Biesheuvel
On 04/15/20 22:38, sean.brogan via [] wrote:
> On Wed, Apr 15, 2020 at 10:18 AM, Laszlo Ersek wrote:
>
>>
>> ArmVirtPkg/ArmVirtPkg.ci.yaml
>> ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
>> ArmVirtPkg/PlatformCI/PlatformBuild.py
>> ArmVirtPkg/PlatformCI/README-pytools.md
>> ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
>
> I am ok with the above except one thought on the readme. One nice
> thing about the markdown readme files are the badge shows up in github
> when you view the package. This is a quick and easy way to see the
> current status.
I agree this is very useful.
What I dislike is that, when I open "Readme.md" (e.g. in the project
root) in a normal terminal, I'm greeted by a HTML tag soup under the
heading "# Build Status".
Markdown is supposed to be readable as plain text. Embedding the
page-ful of build status HTML in "Readme.md" defeats that purpose.
github should either consume a different file (too) for displaying
status badges, or else the "Readme.md" file should reference the HTML
snippet in question with some kind of link or directive, rather than
directly containing it.
Perhaps github already offers this feature -- that would be awesome. I
would be happy with the following variant, for example:
ArmVirtPkg/ArmVirtPkg.ci.yaml
ArmVirtPkg/PlatformCI/PlatformBuild.py
ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
ArmVirtPkg/README-ci.md
ArmVirtPkg/README.md
"README.md" would contain the package description that read nice in a
terminal too. Then, "README.md" would either (somehow?) include
"README-ci.md" by reference, or else github would render both
"README-ci.md" and "README.md".
>
> * Ovmf has a pretty stale readme
Hm, I'd say "somewhat" stale. We do keep it up-to-date with "very
important" stuff.
My excuse for not polishing it more -- which I honestly do believe is a
*valid* excuse -- is that users have shown repeatedly that they don't
read the README at all. I've explained basic stuff like "how to capture
OVMF's debug log" umpteen times on the list, despite it being spelled
out in the README. The fact is that effort put towards careful
documentation is almost entirely lost effort -- this was also clearly
proved by the (non-)reaction that I got to my OVMF white paper that I
wrote a few years back (~60 A4 pages, if I recall correctly).
Documentation is just not *worth* polishing, considering the user base
as a whole. I for one go to the available documentation *before*
starting to use new software, or when questions pop up, but it seems
like I belong to a vanishingly small camp with that. People just flock
to social media (or, in the least wrong case: they come to this mailing
list), and ask questions they could already find the answers to in
existent documentation.
This is a bitter realization for me, especially having written
relatively substantial articles for the edk2 wiki:
https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers
https://github.com/tianocore/tianocore.github.io/wiki/Testing-SMM-with-QEMU,-KVM-and-libvirt
but it is what I now believe.
> and does not take advantage of markdown.
Correct. That's not intentional; I think the README just predates the
usefulness of markdown (i.e. it predates moving the project to
github.com). IIRC.
> We could convert it to MD, clean up, and then merge in the content
> from the pytools.md. I would need help or a package maintainer to do
> the cleanup of the readme to make sure it contained the content you
> desired.
So my problem with this is two-fold. First, regarding just the markdown
conversion, I agree it would be nice, but I don't wish to sink any work
into it (see my opinion above, about polishing documentation). If
someone wanted to spend time on just a structural conversion to markdown
(not modifying content), I'd be OK to review that.
Second, merging (i.e., flattening) the tag soup from "README-pytools.md"
into the main package "Readme.md" is something that I'm opposed to, as
it interferes with consuming the readme from a plain terminal or text
editor.
> * ArmVirtPkg doesn't have a readme and this is definitely a barrier to
> entry for the package. I would suggest creating one and then merging
> in the content from the pytools.md.
Creating a readme in MD format: would be nice if someone contributed
that ("patches welcome" :) ).
*Merging* the HTML tag soup: please let's not do that. (I'm totally fine
if it is introduced in a separate, appropriately named or located file.)
> * EmulatorPkg has one. I would just suggest a merge but i am yet to
> get any feedback from those maintainers.
>
> If that isn't desirable i would at least suggest we change the title
> to just ReadMe.md so that GitHub shows it by default when the
> PaltformCI folder is viewed form the web or in editor like vscode.
This sounds 100% viable and great to me. I didn't expect this could
work! (I'm generally unaware of the readme filename patterns, and
locations, that github.com recognizes; sorry about that.) Having an
"unadorned" ReadMe.md file under PlatformCI is just perfect.
So if I understand correctly, we could choose:
ArmVirtPkg/ArmVirtPkg.ci.yaml
ArmVirtPkg/PlatformCI/PlatformBuild.py
ArmVirtPkg/PlatformCI/ReadMe.md
ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
Do I understand right?
Because, I'd find this great!
Thank you!
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-08 18:13 ` [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI Michael Kubacki
2020-04-09 6:47 ` Laszlo Ersek
2020-04-09 8:05 ` Ard Biesheuvel
@ 2020-04-09 9:17 ` Laszlo Ersek
2020-04-09 9:23 ` Leif Lindholm
2 siblings, 1 reply; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-09 9:17 UTC (permalink / raw)
To: michael.kubacki, devel; +Cc: Ard Biesheuvel, Leif Lindholm
Hi Michael,
On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
> From: Sean Brogan <sean.brogan@microsoft.com>
>
> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
>
> Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
> * Ubuntu GCC5
> Add PyTool based build of ArmVirtPkg
> Add extdep for managing the iasl dependency
> Add ArmVirtPkg.ci.yaml for Core CI
> Add README-pytools for details and instructions
>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Cc: Leif Lindholm <leif@nuviainc.com>
> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
The commit message is missing your Signed-off-by. You can & should keep
Sean's of course, but since you are submitting it, I think yours (too)
is required, per "Developer Certificate of Origin" in "Readme.md".
Thanks
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 9:17 ` Laszlo Ersek
@ 2020-04-09 9:23 ` Leif Lindholm
2020-04-09 13:18 ` Laszlo Ersek
0 siblings, 1 reply; 29+ messages in thread
From: Leif Lindholm @ 2020-04-09 9:23 UTC (permalink / raw)
To: Laszlo Ersek; +Cc: michael.kubacki, devel, Ard Biesheuvel
On Thu, Apr 09, 2020 at 11:17:31 +0200, Laszlo Ersek wrote:
> Hi Michael,
>
> On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
> > From: Sean Brogan <sean.brogan@microsoft.com>
> >
> > REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
> >
> > Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
> > * Ubuntu GCC5
> > Add PyTool based build of ArmVirtPkg
> > Add extdep for managing the iasl dependency
> > Add ArmVirtPkg.ci.yaml for Core CI
> > Add README-pytools for details and instructions
> >
> > Cc: Laszlo Ersek <lersek@redhat.com>
> > Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> > Cc: Leif Lindholm <leif@nuviainc.com>
> > Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
>
> The commit message is missing your Signed-off-by. You can & should keep
> Sean's of course, but since you are submitting it, I think yours (too)
> is required, per "Developer Certificate of Origin" in "Readme.md".
Michael cannot attest for Sean's legal opinions about this
contribution, only Sean can do that. The attribution of authorship is
already covered by the From: tag.
/
Leif
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 9:23 ` Leif Lindholm
@ 2020-04-09 13:18 ` Laszlo Ersek
2020-04-09 14:38 ` Leif Lindholm
2020-04-09 17:09 ` Michael Kubacki
0 siblings, 2 replies; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-09 13:18 UTC (permalink / raw)
To: Leif Lindholm; +Cc: michael.kubacki, devel, Ard Biesheuvel
On 04/09/20 11:23, Leif Lindholm wrote:
> On Thu, Apr 09, 2020 at 11:17:31 +0200, Laszlo Ersek wrote:
>> Hi Michael,
>>
>> On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
>>> From: Sean Brogan <sean.brogan@microsoft.com>
>>>
>>> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
>>>
>>> Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
>>> * Ubuntu GCC5
>>> Add PyTool based build of ArmVirtPkg
>>> Add extdep for managing the iasl dependency
>>> Add ArmVirtPkg.ci.yaml for Core CI
>>> Add README-pytools for details and instructions
>>>
>>> Cc: Laszlo Ersek <lersek@redhat.com>
>>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
>>> Cc: Leif Lindholm <leif@nuviainc.com>
>>> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
>>
>> The commit message is missing your Signed-off-by. You can & should
>> keep Sean's of course, but since you are submitting it, I think yours
>> (too) is required, per "Developer Certificate of Origin" in
>> "Readme.md".
>
> Michael cannot attest for Sean's legal opinions about this
> contribution, only Sean can do that.
Not so sure about that (see below), but here I'm not questioning that.
> The attribution of authorship is already covered by the From: tag.
Agreed.
My point is that Michael is technically partaking in the contribution of
this patch, even if he's only forwarding (upstreaming) Sean's patch
verbatim. Therefore the language in "Readme.md" applies to him as a
contributor:
> # Code Contributions
> To make a contribution to a TianoCore project, follow these steps.
> [...]
> 2. Your commit message must include your `Signed-off-by` signature
> [...]
>
> # Developer Certificate of Origin
>
> Your change description should use the standard format for a
> commit message, and must include your `Signed-off-by` signature.
>
> [...]
>
> By making a contribution to this project, I certify that:
>
> (a) The contribution was created in whole or in part by me and I
> have the right to submit it under the open source license
> indicated in the file; or
>
> (b) The contribution is based upon previous work that, to the best
> of my knowledge, is covered under an appropriate open source
> license and I have the right under that license to submit that
> work with modifications, whether created in whole or in part
> by me, under the same open source license (unless I am
> permitted to submit under a different license), as indicated
> in the file; or
>
> (c) The contribution was provided directly to me by some other
> person who certified (a), (b) or (c) and I have not modified
> it.
>
> [...]
So, two points:
- under case (c), I do think Michael would attest for Sean's legal
opinions,
- even if it's not case (c) that applies, a Signed-off-by is needed from
Michael, for one of the other cases.
The DCO that's included in "Readme.md" spells out the following goal (or
"spirit"):
> In order to keep track of who did what, all patches contributed must
> include a statement that to the best of the contributor's knowledge
> they have the right to contribute it under the specified license.
If we don't include an S-o-b from Michael on the patch, his
participation in the upstreaming of this code change will disappear
entirely. The Committer field will refer to the mergify bot, the Author
field will refer (correctly) to Sean, and the [f]act of Michael posting
the patch to the list will be lost from the git history.
The one S-o-b that we could technically do without, IMO, is Sean's.
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 13:18 ` Laszlo Ersek
@ 2020-04-09 14:38 ` Leif Lindholm
2020-04-09 17:09 ` Michael Kubacki
1 sibling, 0 replies; 29+ messages in thread
From: Leif Lindholm @ 2020-04-09 14:38 UTC (permalink / raw)
To: Laszlo Ersek; +Cc: michael.kubacki, devel, Ard Biesheuvel
On Thu, Apr 09, 2020 at 15:18:42 +0200, Laszlo Ersek wrote:
> On 04/09/20 11:23, Leif Lindholm wrote:
> > On Thu, Apr 09, 2020 at 11:17:31 +0200, Laszlo Ersek wrote:
> >> Hi Michael,
> >>
> >> On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
> >>> From: Sean Brogan <sean.brogan@microsoft.com>
> >>>
> >>> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
> >>>
> >>> Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
> >>> * Ubuntu GCC5
> >>> Add PyTool based build of ArmVirtPkg
> >>> Add extdep for managing the iasl dependency
> >>> Add ArmVirtPkg.ci.yaml for Core CI
> >>> Add README-pytools for details and instructions
> >>>
> >>> Cc: Laszlo Ersek <lersek@redhat.com>
> >>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> >>> Cc: Leif Lindholm <leif@nuviainc.com>
> >>> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
> >>
> >> The commit message is missing your Signed-off-by. You can & should
> >> keep Sean's of course, but since you are submitting it, I think yours
> >> (too) is required, per "Developer Certificate of Origin" in
> >> "Readme.md".
> >
> > Michael cannot attest for Sean's legal opinions about this
> > contribution, only Sean can do that.
>
> Not so sure about that (see below), but here I'm not questioning that.
>
> > The attribution of authorship is already covered by the From: tag.
>
> Agreed.
>
> My point is that Michael is technically partaking in the contribution of
> this patch, even if he's only forwarding (upstreaming) Sean's patch
> verbatim. Therefore the language in "Readme.md" applies to him as a
> contributor:
Yes, Michael's SoB is absolutely required.
> > # Code Contributions
> > To make a contribution to a TianoCore project, follow these steps.
> > [...]
> > 2. Your commit message must include your `Signed-off-by` signature
> > [...]
> >
> > # Developer Certificate of Origin
> >
> > Your change description should use the standard format for a
> > commit message, and must include your `Signed-off-by` signature.
> >
> > [...]
> >
> > By making a contribution to this project, I certify that:
> >
> > (a) The contribution was created in whole or in part by me and I
> > have the right to submit it under the open source license
> > indicated in the file; or
> >
> > (b) The contribution is based upon previous work that, to the best
> > of my knowledge, is covered under an appropriate open source
> > license and I have the right under that license to submit that
> > work with modifications, whether created in whole or in part
> > by me, under the same open source license (unless I am
> > permitted to submit under a different license), as indicated
> > in the file; or
> >
> > (c) The contribution was provided directly to me by some other
> > person who certified (a), (b) or (c) and I have not modified
> > it.
> >
> > [...]
>
> So, two points:
> - under case (c), I do think Michael would attest for Sean's legal
> opinions,
> - even if it's not case (c) that applies, a Signed-off-by is needed from
> Michael, for one of the other cases.
>
> The DCO that's included in "Readme.md" spells out the following goal (or
> "spirit"):
>
> > In order to keep track of who did what, all patches contributed must
> > include a statement that to the best of the contributor's knowledge
> > they have the right to contribute it under the specified license.
Yes.
> If we don't include an S-o-b from Michael on the patch, his
> participation in the upstreaming of this code change will disappear
> entirely. The Committer field will refer to the mergify bot, the Author
> field will refer (correctly) to Sean, and the [f]act of Michael posting
> the patch to the list will be lost from the git history.
>
> The one S-o-b that we could technically do without, IMO, is Sean's.
Yes, we're agreed on that bit. My statement is that unless Sean has
himself given his SoB *in this context*, leaving that in is at best
misleading and at worst directly false.
Regards,
Leif
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI
2020-04-09 13:18 ` Laszlo Ersek
2020-04-09 14:38 ` Leif Lindholm
@ 2020-04-09 17:09 ` Michael Kubacki
1 sibling, 0 replies; 29+ messages in thread
From: Michael Kubacki @ 2020-04-09 17:09 UTC (permalink / raw)
To: Laszlo Ersek, Leif Lindholm; +Cc: devel, Ard Biesheuvel
Thanks for looking into this. I agree per case (c) that I should add my
S-o-b. I will plan to add both mine and Sean's S-o-b in v2 of the series.
Thanks,
Michael
On 4/9/2020 6:18 AM, Laszlo Ersek wrote:
> On 04/09/20 11:23, Leif Lindholm wrote:
>> On Thu, Apr 09, 2020 at 11:17:31 +0200, Laszlo Ersek wrote:
>>> Hi Michael,
>>>
>>> On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
>>>> From: Sean Brogan <sean.brogan@microsoft.com>
>>>>
>>>> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
>>>>
>>>> Add new Azure Pipeline definitions to build and run ArmVirtPkg with:
>>>> * Ubuntu GCC5
>>>> Add PyTool based build of ArmVirtPkg
>>>> Add extdep for managing the iasl dependency
>>>> Add ArmVirtPkg.ci.yaml for Core CI
>>>> Add README-pytools for details and instructions
>>>>
>>>> Cc: Laszlo Ersek <lersek@redhat.com>
>>>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
>>>> Cc: Leif Lindholm <leif@nuviainc.com>
>>>> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
>>>
>>> The commit message is missing your Signed-off-by. You can & should
>>> keep Sean's of course, but since you are submitting it, I think yours
>>> (too) is required, per "Developer Certificate of Origin" in
>>> "Readme.md".
>>
>> Michael cannot attest for Sean's legal opinions about this
>> contribution, only Sean can do that.
>
> Not so sure about that (see below), but here I'm not questioning that.
>
>> The attribution of authorship is already covered by the From: tag.
>
> Agreed.
>
> My point is that Michael is technically partaking in the contribution of
> this patch, even if he's only forwarding (upstreaming) Sean's patch
> verbatim. Therefore the language in "Readme.md" applies to him as a
> contributor:
>
>> # Code Contributions
>> To make a contribution to a TianoCore project, follow these steps.
>> [...]
>> 2. Your commit message must include your `Signed-off-by` signature
>> [...]
>>
>> # Developer Certificate of Origin
>>
>> Your change description should use the standard format for a
>> commit message, and must include your `Signed-off-by` signature.
>>
>> [...]
>>
>> By making a contribution to this project, I certify that:
>>
>> (a) The contribution was created in whole or in part by me and I
>> have the right to submit it under the open source license
>> indicated in the file; or
>>
>> (b) The contribution is based upon previous work that, to the best
>> of my knowledge, is covered under an appropriate open source
>> license and I have the right under that license to submit that
>> work with modifications, whether created in whole or in part
>> by me, under the same open source license (unless I am
>> permitted to submit under a different license), as indicated
>> in the file; or
>>
>> (c) The contribution was provided directly to me by some other
>> person who certified (a), (b) or (c) and I have not modified
>> it.
>>
>> [...]
>
> So, two points:
> - under case (c), I do think Michael would attest for Sean's legal
> opinions,
> - even if it's not case (c) that applies, a Signed-off-by is needed from
> Michael, for one of the other cases.
>
> The DCO that's included in "Readme.md" spells out the following goal (or
> "spirit"):
>
>> In order to keep track of who did what, all patches contributed must
>> include a statement that to the best of the contributor's knowledge
>> they have the right to contribute it under the specified license.
>
> If we don't include an S-o-b from Michael on the patch, his
> participation in the upstreaming of this code change will disappear
> entirely. The Committer field will refer to the mergify bot, the Author
> field will refer (correctly) to Sean, and the [f]act of Michael posting
> the patch to the list will be lost from the git history.
>
> The one S-o-b that we could technically do without, IMO, is Sean's.
>
> Laszlo
>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v1 3/6] EmulatorPkg: Add Platform CI and configuration for Core CI
[not found] <20200408181327.4324-1-michael.kubacki@outlook.com>
2020-04-08 18:13 ` [PATCH v1 1/6] .azurepipelines: Add Platform CI template Michael Kubacki
2020-04-08 18:13 ` [PATCH v1 2/6] ArmVirtPkg: Add Platform CI and configuration for Core CI Michael Kubacki
@ 2020-04-08 18:13 ` Michael Kubacki
2020-04-15 19:36 ` [edk2-devel] " Sean
2020-04-08 18:13 ` [PATCH v1 4/6] OvmfPkg: " Michael Kubacki
` (2 subsequent siblings)
5 siblings, 1 reply; 29+ messages in thread
From: Michael Kubacki @ 2020-04-08 18:13 UTC (permalink / raw)
To: devel; +Cc: Jordan Justen, Andrew Fish, Ray Ni
From: Sean Brogan <sean.brogan@microsoft.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
Add new Azure Pipeline definitions to build and run EmulatorPkg with:
* Ubuntu GCC5
* Windows VS2019
Add PyTool based build of EmulatorPkg
Add EmulatorPkg.ci.yaml for Core CI
Add README-pytools for details and instructions
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
---
EmulatorPkg/.azurepipelines/Ubuntu-GCC5.yml | 95 ++++++++
EmulatorPkg/.azurepipelines/Windows-VS2019.yml | 85 +++++++
EmulatorPkg/EmulatorPkg.ci.yaml | 85 +++++++
EmulatorPkg/PlatformBuild.py | 252 ++++++++++++++++++++
EmulatorPkg/README-pytools.md | 173 ++++++++++++++
5 files changed, 690 insertions(+)
diff --git a/EmulatorPkg/.azurepipelines/Ubuntu-GCC5.yml b/EmulatorPkg/.azurepipelines/Ubuntu-GCC5.yml
new file mode 100644
index 000000000000..1aa5904bb379
--- /dev/null
+++ b/EmulatorPkg/.azurepipelines/Ubuntu-GCC5.yml
@@ -0,0 +1,95 @@
+## @file
+# Azure Pipeline build file for building a platform.
+#
+# Platform: EmulatorPkg
+# OS: Ubuntu
+# Toolchain: GCC5
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+ - master
+pr:
+ - master
+jobs:
+ - job: Platform_CI
+ variables:
+ package: 'EmulatorPkg'
+ vm_image: 'ubuntu-latest'
+ should_run: false
+ run_flags: "MAKE_STARTUP_NSH=TRUE"
+
+ #Use matrix to speed up the build process
+ strategy:
+ matrix:
+ EmulatorPkg_X64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_X64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_X64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_IA32_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_IA32_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_IA32_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ workspace:
+ clean: all
+
+ pool:
+ vmImage: $(vm_image)
+
+ steps:
+ - template: ../../.azurepipelines/templates/platform-build-run-steps.yml
+ parameters:
+ tool_chain_tag: GCC5
+ build_pkg: $(package)
+ build_target: $(Build.Target)
+ build_arch: $(Build.Arch)
+ build_file: $(Build.File)
+ build_flags: $(Build.Flags)
+ run_flags: $(Run.Flags)
+ # Add steps to install some IA32 only dependencies
+ extra_install_step:
+ - bash: sudo dpkg --add-architecture i386
+ displayName: Add i386 to dpkg
+ condition: and(gt(variables.pkg_count, 0), eq(variables['Build.Arch'], 'IA32'), succeeded())
+ - bash: sudo apt-get update
+ displayName: do apt-get update
+ condition: and(gt(variables.pkg_count, 0), eq(variables['Build.Arch'], 'IA32'), succeeded())
+ - bash: sudo apt-get install libc6-dev:i386 libx11-dev:i386 libxext-dev:i386 lib32gcc-7-dev
+ displayName: Add additional i386 packages
+ condition: and(gt(variables.pkg_count, 0), eq(variables['Build.Arch'], 'IA32'), succeeded())
diff --git a/EmulatorPkg/.azurepipelines/Windows-VS2019.yml b/EmulatorPkg/.azurepipelines/Windows-VS2019.yml
new file mode 100644
index 000000000000..e73eb910e04b
--- /dev/null
+++ b/EmulatorPkg/.azurepipelines/Windows-VS2019.yml
@@ -0,0 +1,85 @@
+## @file
+# Azure Pipeline build file for building a platform.
+#
+# Platform: EMULATORPKG
+# OS: Windows
+# Toolchain: VS2019
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+ - master
+pr:
+ - master
+
+jobs:
+ - job: Platform_CI
+ variables:
+ package: 'EmulatorPkg'
+ vm_image: 'windows-latest'
+ should_run: true
+ run_flags: "MAKE_STARTUP_NSH=TRUE"
+
+ #Use matrix to speed up the build process
+ strategy:
+ matrix:
+ EmulatorPkg_X64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_X64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_X64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_IA32_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32 "
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_IA32_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32 "
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ EmulatorPkg_IA32_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32 "
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ workspace:
+ clean: all
+
+ pool:
+ vmImage: $(vm_image)
+
+ steps:
+ - template: ../../.azurepipelines/templates/platform-build-run-steps.yml
+ parameters:
+ tool_chain_tag: VS2019
+ build_pkg: $(package)
+ build_target: $(Build.Target)
+ build_arch: $(Build.Arch)
+ build_file: $(Build.File)
+ build_flags: $(Build.Flags)
+ run_flags: $(Run.Flags)
diff --git a/EmulatorPkg/EmulatorPkg.ci.yaml b/EmulatorPkg/EmulatorPkg.ci.yaml
new file mode 100644
index 000000000000..81f81780ec76
--- /dev/null
+++ b/EmulatorPkg/EmulatorPkg.ci.yaml
@@ -0,0 +1,85 @@
+## @file
+# Core CI configuration for EmulatorPkg
+#
+# EmulatorPkg is part of Platform Ci for builds so this is only
+# used for code analysis.
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ ## options defined .pytool/Plugin/CompilerPlugin
+ "CompilerPlugin": {
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin
+ "HostUnitTestCompilerPlugin": {
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/CharEncodingCheck
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+
+ ## options defined .pytool/Plugin/DependencyCheck
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ # For this platform all packages are allowed???
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "EmulatorPkg/EmulatorPkg.dec",
+ "NetworkPkg/NetworkPkg.dec",
+ "EmbeddedPkg/EmbeddedPkg.dec", ## is this one OK??
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[
+ "UnitTestFrameworkPkg/UnitTestFrameworkPkg.dec"
+ ],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+
+ ## options defined .pytool/Plugin/DscCompleteCheck
+ "DscCompleteCheck": {
+ "IgnoreInf": [""],
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/HostUnitTestDscCompleteCheck
+ "HostUnitTestDscCompleteCheck": {
+ "IgnoreInf": [""],
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/GuidCheck
+ "GuidCheck": {
+ "IgnoreGuidName": [],
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [],
+ },
+
+ ## options defined .pytool/Plugin/LibraryClassCheck
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined .pytool/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails right now with over 270 errors
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [
+ "setjump",
+ "plong",
+ "lparam",
+ "lpdword",
+ "lpthread",
+ "lresult",
+ ], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
diff --git a/EmulatorPkg/PlatformBuild.py b/EmulatorPkg/PlatformBuild.py
new file mode 100644
index 000000000000..4ed87f58e14d
--- /dev/null
+++ b/EmulatorPkg/PlatformBuild.py
@@ -0,0 +1,252 @@
+# @file
+# Script to Build EmulatorPkg UEFI firmware
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+
+from edk2toolext.environment import shell_environment
+from edk2toolext.environment.uefi_build import UefiBuilder
+from edk2toolext.invocables.edk2_platform_build import BuildSettingsManager
+from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule
+from edk2toolext.invocables.edk2_update import UpdateSettingsManager
+from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager
+from edk2toollib.utility_functions import RunCmd
+from edk2toollib.utility_functions import GetHostInfo
+
+# ####################################################################################### #
+# Common Configuration #
+# ####################################################################################### #
+
+
+class CommonPlatform():
+ ''' Common settings for this platform. Define static data here and use
+ for the different parts of stuart
+ '''
+ PackagesSupported = ("EmulatorPkg",)
+ ArchSupported = ("X64", "IA32")
+ TargetsSupported = ("DEBUG", "RELEASE", "NOOPT")
+ Scopes = ('emulatorpkg', 'edk2-build')
+ WorkspaceRoot = os.path.realpath(os.path.join(
+ os.path.dirname(os.path.abspath(__file__)), ".."))
+
+ # ####################################################################################### #
+ # Configuration for Update & Setup #
+ # ####################################################################################### #
+
+
+class SettingsManager(UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager):
+
+ def GetPackagesSupported(self):
+ ''' return iterable of edk2 packages supported by this build.
+ These should be edk2 workspace relative paths '''
+ return CommonPlatform.PackagesSupported
+
+ def GetArchitecturesSupported(self):
+ ''' return iterable of edk2 architectures supported by this build '''
+ return CommonPlatform.ArchSupported
+
+ def GetTargetsSupported(self):
+ ''' return iterable of edk2 target tags supported by this build '''
+ return CommonPlatform.TargetsSupported
+
+ def GetRequiredSubmodules(self):
+ ''' return iterable containing RequiredSubmodule objects.
+ If no RequiredSubmodules return an empty iterable
+ '''
+ rs = []
+ rs.append(RequiredSubmodule(
+ "ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3", False))
+ rs.append(RequiredSubmodule(
+ "CryptoPkg/Library/OpensslLib/openssl", False))
+ return rs
+
+ def SetArchitectures(self, list_of_requested_architectures):
+ ''' Confirm the requests architecture list is valid and configure SettingsManager
+ to run only the requested architectures.
+
+ Raise Exception if a list_of_requested_architectures is not supported
+ '''
+ unsupported = set(list_of_requested_architectures) - \
+ set(self.GetArchitecturesSupported())
+ if(len(unsupported) > 0):
+ errorString = (
+ "Unsupported Architecture Requested: " + " ".join(unsupported))
+ logging.critical(errorString)
+ raise Exception(errorString)
+ self.ActualArchitectures = list_of_requested_architectures
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return CommonPlatform.WorkspaceRoot
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ return CommonPlatform.Scopes
+
+ def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list:
+ ''' Filter other cases that this package should be built
+ based on changed files. This should cover things that can't
+ be detected as dependencies. '''
+ build_these_packages = []
+ possible_packages = potentialPackagesList.copy()
+ for f in changedFilesList:
+ # BaseTools files that might change the build
+ if "BaseTools" in f:
+ if os.path.splitext(f) not in [".txt", ".md"]:
+ build_these_packages = possible_packages
+ break
+ return build_these_packages
+
+ def GetPlatformDscAndConfig(self) -> tuple:
+ ''' If a platform desires to provide its DSC then Policy 4 will evaluate if
+ any of the changes will be built in the dsc.
+
+ The tuple should be (<workspace relative path to dsc file>, <input dictionary of dsc key value pairs>)
+ '''
+ return (os.path.join("EmulatorPkg", "EmulatorPkg.dsc"), {})
+
+ # ####################################################################################### #
+ # Actual Configuration for Platform Build #
+ # ####################################################################################### #
+
+
+class PlatformBuilder(UefiBuilder, BuildSettingsManager):
+ def __init__(self):
+ UefiBuilder.__init__(self)
+
+ def AddCommandLineOptions(self, parserObj):
+ ''' Add command line options to the argparser '''
+ parserObj.add_argument('-a', "--arch", dest="build_arch", type=str, default="X64",
+ help="Optional - architecture to build. IA32 will use IA32 for Pei & Dxe. "
+ "X64 will use X64 for both PEI and DXE.")
+
+ def RetrieveCommandLineOptions(self, args):
+ ''' Retrieve command line options from the argparser '''
+
+ shell_environment.GetBuildVars().SetValue(
+ "TARGET_ARCH", args.build_arch.upper(), "From CmdLine")
+ shell_environment.GetBuildVars().SetValue(
+ "ACTIVE_PLATFORM", "EmulatorPkg/EmulatorPkg.dsc", "From CmdLine")
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return CommonPlatform.WorkspaceRoot
+
+ def GetPackagesPath(self):
+ ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath '''
+ return ()
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ return CommonPlatform.Scopes
+
+ def GetName(self):
+ ''' Get the name of the repo, platform, or product being build '''
+ ''' Used for naming the log file, among others '''
+
+ # check the startup nsh flag and if set then rename the log file.
+ # this helps in CI so we don't overwrite the build log since running
+ # uses the stuart_build command.
+ if(shell_environment.GetBuildVars().GetValue("MAKE_STARTUP_NSH", "FALSE") == "TRUE"):
+ return "EmulatorPkg_With_Run"
+ return "EmulatorPkg"
+
+ def GetLoggingLevel(self, loggerType):
+ ''' Get the logging level for a given type
+ base == lowest logging level supported
+ con == Screen logging
+ txt == plain text file logging
+ md == markdown file logging
+ '''
+ return logging.DEBUG
+
+ def SetPlatformEnv(self):
+ logging.debug("PlatformBuilder SetPlatformEnv")
+ self.env.SetValue("PRODUCT_NAME", "EmulatorPkg", "Platform Hardcoded")
+ self.env.SetValue("TOOL_CHAIN_TAG", "VS2019", "Default Toolchain")
+
+ # Add support for using the correct Platform Headers, tools, and Libs based on emulator architecture
+ # requested to be built when building VS2019 or VS2017
+ if self.env.GetValue("TOOL_CHAIN_TAG") == "VS2019" or self.env.GetValue("TOOL_CHAIN_TAG") == "VS2017":
+ key = self.env.GetValue("TOOL_CHAIN_TAG") + "_HOST"
+ if self.env.GetValue("TARGET_ARCH") == "IA32":
+ shell_environment.ShellEnvironment().set_shell_var(key, "x86")
+ elif self.env.GetValue("TARGET_ARCH") == "X64":
+ shell_environment.ShellEnvironment().set_shell_var(key, "x64")
+
+ # Add support for using the correct Platform Headers, tools, and Libs based on emulator architecture
+ # requested to be built when building on linux.
+ if GetHostInfo().os.upper() == "LINUX":
+ self.ConfigureLinuxDLinkPath()
+
+ if GetHostInfo().os.upper() == "WINDOWS":
+ self.env.SetValue("BLD_*_WIN_HOST_BUILD", "TRUE",
+ "Trigger Windows host build")
+
+ self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
+
+ # I don't see what this does but it is in build.sh
+ key = "BLD_*_BUILD_" + self.env.GetValue("TARGET_ARCH")
+ self.env.SetValue(key, "TRUE", "match script in build.sh")
+ return 0
+
+ def PlatformPreBuild(self):
+ return 0
+
+ def PlatformPostBuild(self):
+ return 0
+
+ def FlashRomImage(self):
+ ''' Use the FlashRom Function to run the emulator. This gives an easy stuart command line to
+ activate the emulator. '''
+
+ OutputPath = os.path.join(self.env.GetValue(
+ "BUILD_OUTPUT_BASE"), self.env.GetValue("TARGET_ARCH"))
+
+ if (self.env.GetValue("MAKE_STARTUP_NSH") == "TRUE"):
+ f = open(os.path.join(OutputPath, "startup.nsh"), "w")
+ f.write("BOOT SUCCESS !!! \n")
+ # add commands here
+ f.write("reset\n")
+ f.close()
+
+ if GetHostInfo().os.upper() == "WINDOWS":
+ cmd = "WinHost.exe"
+ elif GetHostInfo().os.upper() == "LINUX":
+ cmd = "./Host"
+ else:
+ logging.critical("Unsupported Host")
+ return -1
+ return RunCmd(cmd, "", workingdir=OutputPath)
+
+ def ConfigureLinuxDLinkPath(self):
+ '''
+ logic copied from build.sh to setup the correct libraries
+ '''
+ if self.env.GetValue("TARGET_ARCH") == "IA32":
+ LIB_NAMES = ["ld-linux.so.2", "libdl.so.2 crt1.o", "crti.o crtn.o"]
+ LIB_SEARCH_PATHS = ["/usr/lib/i386-linux-gnu",
+ "/usr/lib32", "/lib32", "/usr/lib", "/lib"]
+ elif self.env.GetValue("TARGET_ARCH") == "X64":
+ LIB_NAMES = ["ld-linux-x86-64.so.2",
+ "libdl.so.2", "crt1.o", "crti.o", "crtn.o"]
+ LIB_SEARCH_PATHS = ["/usr/lib/x86_64-linux-gnu",
+ "/usr/lib64", "/lib64", "/usr/lib", "/lib"]
+
+ HOST_DLINK_PATHS = ""
+ for lname in LIB_NAMES:
+ logging.debug(f"Looking for {lname}")
+ for dname in LIB_SEARCH_PATHS:
+ logging.debug(f"In {dname}")
+ if os.path.isfile(os.path.join(dname, lname)):
+ logging.debug(f"Found {lname} in {dname}")
+ HOST_DLINK_PATHS += os.path.join(
+ os.path.join(dname, lname)) + os.pathsep
+ break
+ HOST_DLINK_PATHS = HOST_DLINK_PATHS.rstrip(os.pathsep)
+ logging.critical(f"Setting HOST_DLINK_PATHS to {HOST_DLINK_PATHS}")
+ shell_environment.ShellEnvironment().set_shell_var(
+ "HOST_DLINK_PATHS", HOST_DLINK_PATHS)
diff --git a/EmulatorPkg/README-pytools.md b/EmulatorPkg/README-pytools.md
new file mode 100644
index 000000000000..a99b050bfa65
--- /dev/null
+++ b/EmulatorPkg/README-pytools.md
@@ -0,0 +1,173 @@
+# EmulatorPkg
+
+This README-pytools.md summarizes the current state of Platform CI for EmulatorPkg.
+It also describes how to _build_ EmulatorPkg using the Pytools build system.
+For general documentation on EmulatorPkg, refer to the [local README](./README).
+
+## Platform CI Current Status
+
+<table>
+ <tr>
+ <th>Config</th>
+ <th colspan="3">Build & Run</th>
+ <th>Notes</th>
+ </tr>
+ <tr>
+ <th></th>
+ <th>DEBUG</th>
+ <th>RELEASE</th>
+ <th>NOOPT</th>
+ <th></th>
+ </tr>
+ <tr>
+ <th colspan="5" align="left">
+ Windows VS2019
+ </th>
+ </tr>
+ <tr>
+ <td>IA32</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=40&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_IA32_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=40&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_IA32_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=40&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_IA32_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>X64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_X64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_X64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_X64_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <th colspan="5" align="left">
+ Ubuntu 18.04 GCC5
+ </th>
+ </tr>
+ <tr>
+ <td>IA32</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_IA32_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_IA32_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_IA32_NOOPT"/></a>
+ </td>
+ <td>Run to shell disabled due to SegFault</td>
+ </tr>
+ <tr>
+ <td>X64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_X64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_X64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=39&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/EmulatorPkg/EmulatorPkg%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20EmulatorPkg_X64_NOOPT"/></a>
+ </td>
+ <td>Run to shell disabled due to SegFault</td>
+ </tr>
+</table>
+
+### Config Details
+
+| Config | Architectures |Additional Flags |
+| :---- | :----- | :---- |
+| IA32 | IA32 | None |
+| X64 | X64 | None |
+
+## Setup
+
+### The Usual EDK2 Build Setup
+
+- [Download & Install Python 3.x](https://www.python.org/downloads/)
+- [Download & Install git](https://git-scm.com/download/)
+- [Configure Git for EDK II](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#github-help)
+- [Download/Checkout the EDK II source tree from Github](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#download)
+ - **NOTE:** Do _not_ follow the EDK II Compile Tools and Build instructions, see below...
+
+### Install the necessary development packages for your distribution
+
+This varies by distribution, toolchain, and your configuration but here are a few hints.
+
+- For building ARCH IA32 on X64 Ubuntu 18.04 LTS these steps where needed.
+
+ ``` bash
+ sudo dpkg --add-architecture i386
+ sudo apt-get update
+ sudo apt-get install libc6-dev:i386 libx11-dev:i386 libxext-dev:i386 lib32gcc-7-dev
+ ```
+
+- For building Basetools and other host applications
+
+ ``` bash
+ sudo apt-get update
+ sudo apt-get install gcc g++ make uuid-dev
+
+ ```
+
+### Differences from EDK Classic Build Setup
+
+- Build BaseTools using "`C:\git\edk2>python BaseTools\Edk2ToolsBuild.py [-t <ToolChainTag>]`"
+ - This replaces "`edksetup Rebuild`" from the classic build system
+ - For Windows `<ToolChainTag>` examples, refer to [Windows ToolChain Matrix](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems-ToolChain-Matrix), defaults to `VS2017` if not specified
+- **No Action:** Submodule initialization and manual installation/setup of NASM and iASL is **not** required, it is handled by the Pytools build system
+
+### Install & Configure Pytools for EmulatorPkg
+
+- Install Pytools
+ - `pip install --upgrade -r pip-requirements.txt`
+- Initialize & Update Submodules
+ - `stuart_setup -c EmulatorPkg\PlatformBuild.py`
+- Initialize & Update Dependencies (e.g. iASL & NASM)
+ - `stuart_update -c EmulatorPkg\PlatformBuild.py`
+- Compile (IA32 or X64 supported)
+ - `stuart_build -c EmulatorPkg\PlatformBuild.py [TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG>] -a <TARGET_ARCH>`
+- Running Emulator
+ - You can add `--FlashRom` to the end of your build command and the emulator will run after the build is complete.
+ - or use the FlashOnly feature like `stuart_build -c EmulatorPkg\PlatformBuild.py [TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG>] -a <TARGET_ARCH> --FlashOnly` to just run the emulator.
+
+**NOTE:** configuring ACTIVE_PLATFORM and TARGET_ARCH in Conf/target.txt is __not__ required. This environment is set by PlatformBuild.py based upon the `[-a <TARGET_ARCH>]` parameter.
+
+## Custom Build Options
+
+**MAKE_STARTUP_NSH=TRUE** will output a _startup.nsh_ file to the location mapped as fs0. This is used in CI in combination with the --FlashOnly feature to run the emulator to the UEFI shell and then execute the contents of startup.nsh.
+
+## Passing Build Defines
+
+To pass build defines through stuart*build to the edk2 build, prepend `BLD_*_` to the define name and pass it on the command-line. stuart_build currently requires values to be assigned, so add an `=1` suffix for bare defines.
+For example, to enable the IP6 Network Stack, the stuart_build command-line would be:
+
+`stuart_build -c EmulatorPkg/PlatformBuild.py BLD_*_NETWORK_IP6_ENABLE=1`
+
+## References
+
+- [Installing Pytools](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md#installing)
+- For each workspace, consider creating & using a [Python Virtual Environment](https://docs.python.org/3/library/venv.html). For example <https://microsoft.github.io/mu/CodeDevelopment/prerequisites/#workspace-virtual-environment-setup-process>
+- [stuart_build command-line parser](https://github.com/tianocore/edk2-pytool-extensions/blob/56f6a7aee09995c2f22da4765e8b0a29c1cbf5de/edk2toolext/edk2_invocable.py#L109)
--
2.16.3.windows.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v1 4/6] OvmfPkg: Add Platform CI and configuration for Core CI
[not found] <20200408181327.4324-1-michael.kubacki@outlook.com>
` (2 preceding siblings ...)
2020-04-08 18:13 ` [PATCH v1 3/6] EmulatorPkg: " Michael Kubacki
@ 2020-04-08 18:13 ` Michael Kubacki
2020-04-09 9:17 ` Laszlo Ersek
2020-04-08 18:13 ` [PATCH v1 5/6] .pytool: Update CI Settings to support Emulator, ArmVirt, and Ovmf packages Michael Kubacki
2020-04-08 18:13 ` [PATCH v1 6/6] .azurepipelines: Update Core CI build matrix to include platforms Michael Kubacki
5 siblings, 1 reply; 29+ messages in thread
From: Michael Kubacki @ 2020-04-08 18:13 UTC (permalink / raw)
To: devel; +Cc: Jordan Justen, Laszlo Ersek, Ard Biesheuvel
From: Sean Brogan <sean.brogan@microsoft.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
Add new Azure Pipeline definitions to build and run OvmfPkg with:
* Ubuntu GCC5
* Windows VS2019
Add PyTool based build of OvmfPkg
Add extdep for managing the iasl dependency
Add OvmfPkg.ci.yaml for Core CI
Add README-pytools for details and instructions
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
---
OvmfPkg/.azurepipelines/Ubuntu-GCC5.yml | 133 +++++++++++
OvmfPkg/.azurepipelines/Windows-VS2019.yml | 138 +++++++++++
OvmfPkg/OvmfPkg.ci.yaml | 83 +++++++
OvmfPkg/PlatformBuild.py | 242 ++++++++++++++++++++
OvmfPkg/README-pytools.md | 238 +++++++++++++++++++
OvmfPkg/iasl_ext_dep.yaml | 21 ++
6 files changed, 855 insertions(+)
diff --git a/OvmfPkg/.azurepipelines/Ubuntu-GCC5.yml b/OvmfPkg/.azurepipelines/Ubuntu-GCC5.yml
new file mode 100644
index 000000000000..c2da388aba78
--- /dev/null
+++ b/OvmfPkg/.azurepipelines/Ubuntu-GCC5.yml
@@ -0,0 +1,133 @@
+## @file
+# Azure Pipeline build file for building a platform.
+#
+# Platform: OVMF
+# OS: Ubuntu
+# Toolchain: GCC5
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+ - master
+pr:
+ - master
+
+jobs:
+ - job: Platform_CI
+ variables:
+ package: 'OvmfPkg'
+ vm_image: 'ubuntu-latest'
+ should_run: true
+ run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
+
+ #Use matrix to speed up the build process
+ strategy:
+ matrix:
+ OVMF_IA32_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ OVMF_X64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_X64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_X64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ OVMF_IA32X64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ OVMF_IA32X64_FULL_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: "BLD_*_SECURE_BOOT_ENABLE=1 BLD_*_SMM_REQUIRE=1 BLD_*_TPM_ENABLE=1 BLD_*_TPM_CONFIG_ENABLE=1 BLD_*_NETWORK_TLS_ENABLE=1 BLD_*_NETWORK_IP6_ENABLE=1 BLD_*_NETWORK_HTTP_BOOT_ENABLE=1"
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_FULL_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: "BLD_*_SECURE_BOOT_ENABLE=1 BLD_*_SMM_REQUIRE=1 BLD_*_TPM_ENABLE=1 BLD_*_TPM_CONFIG_ENABLE=1 BLD_*_NETWORK_TLS_ENABLE=1 BLD_*_NETWORK_IP6_ENABLE=1 BLD_*_NETWORK_HTTP_BOOT_ENABLE=1"
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_FULL_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: "BLD_*_SECURE_BOOT_ENABLE=1 BLD_*_SMM_REQUIRE=1 BLD_*_TPM_ENABLE=1 BLD_*_TPM_CONFIG_ENABLE=1 BLD_*_NETWORK_TLS_ENABLE=1 BLD_*_NETWORK_IP6_ENABLE=1 BLD_*_NETWORK_HTTP_BOOT_ENABLE=1"
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ workspace:
+ clean: all
+
+ pool:
+ vmImage: $(vm_image)
+
+ steps:
+ - template: ../../.azurepipelines/templates/platform-build-run-steps.yml
+ parameters:
+ tool_chain_tag: GCC5
+ build_pkg: $(package)
+ build_target: $(Build.Target)
+ build_arch: $(Build.Arch)
+ build_file: $(Build.File)
+ build_flags: $(Build.Flags)
+ run_flags: $(Run.Flags)
+ extra_install_step:
+ - bash: sudo apt-get install qemu
+ displayName: Install qemu
+ condition: and(gt(variables.pkg_count, 0), succeeded())
diff --git a/OvmfPkg/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/.azurepipelines/Windows-VS2019.yml
new file mode 100644
index 000000000000..0de24017a2a6
--- /dev/null
+++ b/OvmfPkg/.azurepipelines/Windows-VS2019.yml
@@ -0,0 +1,138 @@
+## @file
+# Azure Pipeline build file for building a platform.
+#
+# Platform: OVMF
+# OS: Windows
+# Toolchain: VS2019
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+trigger:
+ - master
+pr:
+ - master
+jobs:
+ - job: Platform_CI
+ variables:
+ package: 'OvmfPkg'
+ vm_image: 'windows-latest'
+ should_run: true
+ run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
+
+ #Use matrix to speed up the build process
+ strategy:
+ matrix:
+ OVMF_IA32_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ OVMF_X64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_X64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_X64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "X64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ OVMF_IA32X64_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: ""
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: ""
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_NOOPT:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: ""
+ Build.Target: "NOOPT"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ OVMF_IA32X64_FULL_DEBUG:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: "BLD_*_SECURE_BOOT_ENABLE=1 BLD_*_SMM_REQUIRE=1 BLD_*_TPM_ENABLE=1 BLD_*_TPM_CONFIG_ENABLE=1 BLD_*_NETWORK_TLS_ENABLE=1 BLD_*_NETWORK_IP6_ENABLE=1 BLD_*_NETWORK_HTTP_BOOT_ENABLE=1"
+ Build.Target: "DEBUG"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+ OVMF_IA32X64_FULL_RELEASE:
+ Build.File: "$(package)/PlatformBuild.py"
+ Build.Arch: "IA32,X64"
+ Build.Flags: "BLD_*_SECURE_BOOT_ENABLE=1 BLD_*_SMM_REQUIRE=1 BLD_*_TPM_ENABLE=1 BLD_*_TPM_CONFIG_ENABLE=1 BLD_*_NETWORK_TLS_ENABLE=1 BLD_*_NETWORK_IP6_ENABLE=1 BLD_*_NETWORK_HTTP_BOOT_ENABLE=1"
+ Build.Target: "RELEASE"
+ Run.Flags: $(run_flags)
+ Run: $(should_run)
+
+ # This currently creates a very large image which is too big for the FDF declared range
+ # Ovmf maintainers suggest to skip this build for now.
+ #
+ # OVMF_IA32X64_FULL_NOOPT:
+ # Build.File: "$(package)/PlatformBuild.py"
+ # Build.Arch: "IA32,X64"
+ # Build.Flags: "BLD_*_SECURE_BOOT_ENABLE=1 BLD_*_SMM_REQUIRE=1 BLD_*_TPM_ENABLE=1 BLD_*_TPM_CONFIG_ENABLE=1 BLD_*_NETWORK_TLS_ENABLE=1 BLD_*_NETWORK_IP6_ENABLE=1 BLD_*_NETWORK_HTTP_BOOT_ENABLE=1"
+ # Build.Target: "NOOPT"
+ # Run.Flags: $(run_flags)
+ # Run: $(should_run)
+
+ workspace:
+ clean: all
+
+ pool:
+ vmImage: $(vm_image)
+
+ steps:
+ - template: ../../.azurepipelines/templates/platform-build-run-steps.yml
+ parameters:
+ tool_chain_tag: VS2019
+ build_pkg: $(package)
+ build_target: $(Build.Target)
+ build_arch: $(Build.Arch)
+ build_file: $(Build.File)
+ build_flags: $(Build.Flags)
+ run_flags: $(Run.Flags)
+ extra_install_step:
+ - powershell: choco install qemu; Write-Host "##vso[task.prependpath]c:\Program Files\qemu"
+ displayName: Install QEMU and Set QEMU on path # friendly name displayed in the UI
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
diff --git a/OvmfPkg/OvmfPkg.ci.yaml b/OvmfPkg/OvmfPkg.ci.yaml
new file mode 100644
index 000000000000..98992f0429ff
--- /dev/null
+++ b/OvmfPkg/OvmfPkg.ci.yaml
@@ -0,0 +1,83 @@
+## @file
+# Core CI configuration for OvmfPkg
+#
+# OvmfPkg is part of Platform Ci for builds so this is only
+# used for code analysis.
+#
+# Copyright (c) Microsoft Corporation
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ ## options defined .pytool/Plugin/CompilerPlugin
+ "CompilerPlugin": {
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin
+ "HostUnitTestCompilerPlugin": {
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/CharEncodingCheck
+ "CharEncodingCheck": {
+ "IgnoreFiles": []
+ },
+
+ ## options defined .pytool/Plugin/DependencyCheck
+ "DependencyCheck": {
+ "AcceptableDependencies": [
+ "MdePkg/MdePkg.dec",
+ "MdeModulePkg/MdeModulePkg.dec",
+ "OvmfPkg/OvmfPkg.dec",
+ "NetworkPkg/NetworkPkg.dec",
+ "SecurityPkg/SecurityPkg.dec",
+ "UefiCpuPkg/UefiCpuPkg.dec",
+ "ShellPkg/ShellPkg.dec",
+ "EmbeddedPkg/EmbeddedPkg.dec",
+ "SourceLevelDebugPkg/SourceLevelDebugPkg.dec"
+ ],
+ # For host based unit tests
+ "AcceptableDependencies-HOST_APPLICATION":[
+ "UnitTestFrameworkPkg/UnitTestFrameworkPkg.dec"
+ ],
+ # For UEFI shell based apps
+ "AcceptableDependencies-UEFI_APPLICATION":[],
+ "IgnoreInf": []
+ },
+
+ ## options defined .pytool/Plugin/DscCompleteCheck
+ "DscCompleteCheck": {
+ "IgnoreInf": [""],
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/HostUnitTestDscCompleteCheck
+ "HostUnitTestDscCompleteCheck": {
+ "IgnoreInf": [""],
+ "DscPath": "" # Don't support this test
+ },
+
+ ## options defined .pytool/Plugin/GuidCheck
+ "GuidCheck": {
+ "IgnoreGuidName": ["ResetVector", "XenResetVector"], # Expected duplication for gEfiFirmwareVolumeTopFileGuid
+ "IgnoreGuidValue": [],
+ "IgnoreFoldersAndFiles": [],
+ "IgnoreDuplicates": [],
+ },
+
+ ## options defined .pytool/Plugin/LibraryClassCheck
+ "LibraryClassCheck": {
+ "IgnoreHeaderFile": []
+ },
+
+ ## options defined .pytool/Plugin/SpellCheck
+ "SpellCheck": {
+ "AuditOnly": True, # Fails right now with over 270 errors
+ "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files
+ "ExtendWords": [
+
+ ], # words to extend to the dictionary for this package
+ "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore
+ "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported)
+ }
+}
diff --git a/OvmfPkg/PlatformBuild.py b/OvmfPkg/PlatformBuild.py
new file mode 100644
index 000000000000..bd7b342b6c00
--- /dev/null
+++ b/OvmfPkg/PlatformBuild.py
@@ -0,0 +1,242 @@
+# @file
+# Script to Build OVMF UEFI firmware
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+import os
+import logging
+
+from edk2toolext.environment import shell_environment
+from edk2toolext.environment.uefi_build import UefiBuilder
+from edk2toolext.invocables.edk2_platform_build import BuildSettingsManager
+from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule
+from edk2toolext.invocables.edk2_update import UpdateSettingsManager
+from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager
+from edk2toollib.utility_functions import RunCmd
+
+
+ # ####################################################################################### #
+ # Common Configuration #
+ # ####################################################################################### #
+class CommonPlatform():
+ ''' Common settings for this platform. Define static data here and use
+ for the different parts of stuart
+ '''
+ PackagesSupported = ("OvmfPkg",)
+ ArchSupported = ("IA32", "X64")
+ TargetsSupported = ("DEBUG", "RELEASE", "NOOPT")
+ Scopes = ('ovmf', 'edk2-build')
+ WorkspaceRoot = os.path.realpath(os.path.join(
+ os.path.dirname(os.path.abspath(__file__)), ".."))
+
+ @classmethod
+ def GetDscName(cls, ArchCsv: str) -> str:
+ ''' return the DSC given the architectures requested.
+
+ ArchCsv: csv string containing all architectures to build
+ '''
+ dsc = "OvmfPkg"
+ if "IA32" in ArchCsv.upper().split(","):
+ dsc += "Ia32"
+ if "X64" in ArchCsv.upper().split(","):
+ dsc += "X64"
+ dsc += ".dsc"
+ return dsc
+
+
+ # ####################################################################################### #
+ # Configuration for Update & Setup #
+ # ####################################################################################### #
+class SettingsManager(UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager):
+
+ def GetPackagesSupported(self):
+ ''' return iterable of edk2 packages supported by this build.
+ These should be edk2 workspace relative paths '''
+ return CommonPlatform.PackagesSupported
+
+ def GetArchitecturesSupported(self):
+ ''' return iterable of edk2 architectures supported by this build '''
+ return CommonPlatform.ArchSupported
+
+ def GetTargetsSupported(self):
+ ''' return iterable of edk2 target tags supported by this build '''
+ return CommonPlatform.TargetsSupported
+
+ def GetRequiredSubmodules(self):
+ ''' return iterable containing RequiredSubmodule objects.
+ If no RequiredSubmodules return an empty iterable
+ '''
+ rs = []
+ rs.append(RequiredSubmodule(
+ "ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3", False))
+ rs.append(RequiredSubmodule(
+ "CryptoPkg/Library/OpensslLib/openssl", False))
+ return rs
+
+ def SetArchitectures(self, list_of_requested_architectures):
+ ''' Confirm the requests architecture list is valid and configure SettingsManager
+ to run only the requested architectures.
+
+ Raise Exception if a list_of_requested_architectures is not supported
+ '''
+ unsupported = set(list_of_requested_architectures) - set(self.GetArchitecturesSupported())
+ if(len(unsupported) > 0):
+ errorString = ( "Unsupported Architecture Requested: " + " ".join(unsupported))
+ logging.critical( errorString )
+ raise Exception( errorString )
+ self.ActualArchitectures = list_of_requested_architectures
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return CommonPlatform.WorkspaceRoot
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ return CommonPlatform.Scopes
+
+ def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list:
+ ''' Filter other cases that this package should be built
+ based on changed files. This should cover things that can't
+ be detected as dependencies. '''
+ build_these_packages = []
+ possible_packages = potentialPackagesList.copy()
+ for f in changedFilesList:
+ # BaseTools files that might change the build
+ if "BaseTools" in f:
+ if os.path.splitext(f) not in [".txt", ".md"]:
+ build_these_packages = possible_packages
+ break
+
+ # if the azure pipeline file changed
+ if ".azurepipelines" in f and "Ovmf" in f:
+ build_these_packages = possible_packages
+ break
+
+ # if the azure pipeline platform template file changed
+ if ".azurepipelines" in f and "platforms" in f and "templates" in f:
+ build_these_packages = possible_packages
+ break
+
+ return build_these_packages
+
+ def GetPlatformDscAndConfig(self) -> tuple:
+ ''' If a platform desires to provide its DSC then Policy 4 will evaluate if
+ any of the changes will be built in the dsc.
+
+ The tuple should be (<workspace relative path to dsc file>, <input dictionary of dsc key value pairs>)
+ '''
+ dsc = CommonPlatform.GetDscName(",".join(self.ActualArchitectures))
+ return (f"OvmfPkg/{dsc}", {})
+
+
+ # ####################################################################################### #
+ # Actual Configuration for Platform Build #
+ # ####################################################################################### #
+class PlatformBuilder( UefiBuilder, BuildSettingsManager):
+ def __init__(self):
+ UefiBuilder.__init__(self)
+
+ def AddCommandLineOptions(self, parserObj):
+ ''' Add command line options to the argparser '''
+ parserObj.add_argument('-a', "--arch", dest="build_arch", type=str, default="IA32,X64",
+ help="Optional - CSV of architecture to build. IA32 will use IA32 for Pei & Dxe. "
+ "X64 will use X64 for both PEI and DXE. IA32,X64 will use IA32 for PEI and "
+ "X64 for DXE. default is IA32,X64")
+
+ def RetrieveCommandLineOptions(self, args):
+ ''' Retrieve command line options from the argparser '''
+
+ shell_environment.GetBuildVars().SetValue("TARGET_ARCH"," ".join(args.build_arch.upper().split(",")), "From CmdLine")
+ dsc = CommonPlatform.GetDscName(args.build_arch)
+ shell_environment.GetBuildVars().SetValue("ACTIVE_PLATFORM", f"OvmfPkg/{dsc}", "From CmdLine")
+
+ def GetWorkspaceRoot(self):
+ ''' get WorkspacePath '''
+ return CommonPlatform.WorkspaceRoot
+
+ def GetPackagesPath(self):
+ ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath '''
+ return ()
+
+ def GetActiveScopes(self):
+ ''' return tuple containing scopes that should be active for this process '''
+ return CommonPlatform.Scopes
+
+ def GetName(self):
+ ''' Get the name of the repo, platform, or product being build '''
+ ''' Used for naming the log file, among others '''
+ # check the startup nsh flag and if set then rename the log file.
+ # this helps in CI so we don't overwrite the build log since running
+ # uses the stuart_build command.
+ if(shell_environment.GetBuildVars().GetValue("MAKE_STARTUP_NSH", "FALSE") == "TRUE"):
+ return "OvmfPkg_With_Run"
+ return "OvmfPkg"
+
+ def GetLoggingLevel(self, loggerType):
+ ''' Get the logging level for a given type
+ base == lowest logging level supported
+ con == Screen logging
+ txt == plain text file logging
+ md == markdown file logging
+ '''
+ return logging.DEBUG
+
+ def SetPlatformEnv(self):
+ logging.debug("PlatformBuilder SetPlatformEnv")
+ self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
+ self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
+ self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
+ return 0
+
+ def PlatformPreBuild(self):
+ return 0
+
+ def PlatformPostBuild(self):
+ return 0
+
+ def FlashRomImage(self):
+ VirtualDrive = os.path.join(self.env.GetValue("BUILD_OUTPUT_BASE"), "VirtualDrive")
+ os.makedirs(VirtualDrive, exist_ok=True)
+ OutputPath_FV = os.path.join(self.env.GetValue("BUILD_OUTPUT_BASE"), "FV")
+
+ #
+ # QEMU must be on the path
+ #
+ cmd = "qemu-system-x86_64"
+ args = "-debugcon stdio" # write messages to stdio
+ args += " -global isa-debugcon.iobase=0x402" # debug messages out thru virtual io port
+ args += " -net none" # turn off network
+ args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk" # Mount disk with startup.nsh
+
+ if (self.env.GetValue("QEMU_HEADLESS").upper() == "TRUE"):
+ args += " -display none" # no graphics
+
+ if (self.env.GetBuildValue("SMM_REQUIRE") == "1"):
+ args += " -machine q35,smm=on" #,accel=(tcg|kvm)"
+ #args += " -m ..."
+ #args += " -smp ..."
+ args += " -global driver=cfi.pflash01,property=secure,value=on"
+ args += " -drive if=pflash,format=raw,unit=0,file=" + os.path.join(OutputPath_FV, "OVMF_CODE.fd") + ",readonly=on"
+ args += " -drive if=pflash,format=raw,unit=1,file=" + os.path.join(OutputPath_FV, "OVMF_VARS.fd")
+ else:
+ args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd") # path to firmware
+
+
+ if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
+ f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
+ f.write("BOOT SUCCESS !!! \n")
+ ## add commands here
+ f.write("reset -s\n")
+ f.close()
+
+ ret = RunCmd(cmd, args)
+
+ if ret == 0xc0000005:
+ #for some reason getting a c0000005 on successful return
+ return 0
+
+ return ret
+
+
+
diff --git a/OvmfPkg/README-pytools.md b/OvmfPkg/README-pytools.md
new file mode 100644
index 000000000000..7d8689bc2336
--- /dev/null
+++ b/OvmfPkg/README-pytools.md
@@ -0,0 +1,238 @@
+# OvmfPkg
+
+This README-pytools.md summarizes the current state of Platform CI for OvmfPkg.
+It also describes how to _build_ OvmfPkg using the Pytools build system.
+For general documentation on OvmfPkg, refer to the [local README](./README).
+
+## Platform CI Current Status
+
+<table>
+ <tr>
+ <th>Config</th>
+ <th colspan="3">Build & Run</th>
+ <th>Notes</th>
+ </tr>
+ <tr>
+ <th></th>
+ <th>DEBUG</th>
+ <th>RELEASE</th>
+ <th>NOOPT</th>
+ <th></th>
+ </tr>
+ <tr>
+ <th colspan="5" align="left">
+ Windows VS2019
+ </th>
+ </tr>
+ <tr>
+ <td>IA32</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>X64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_X64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_X64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_X64_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>IA32X64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>IA32X64 Full</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_FULL_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=38&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Windows%20VS2019?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_FULL_RELEASE"/></a>
+ </td>
+ <td>
+ Image sizes too large. Skipping
+ </td>
+ <td>NOOPT build out of space in FD</td>
+ </tr>
+ <tr>
+ <th colspan="5" align="left">
+ Ubuntu 18.04 GCC5
+ </th>
+ </tr>
+ <tr>
+ <td>IA32</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>X64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_X64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_X64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_X64_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>IA32X64</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>IA32X64 Full</td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_FULL_DEBUG"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_FULL_RELEASE"/></a>
+ </td>
+ <td>
+ <a href="https://dev.azure.com/tianocore/edk2-ci-play/_build/latest?definitionId=37&branchName=master">
+ <img src="https://dev.azure.com/tianocore/edk2-ci-play/_apis/build/status/OVMF/OVMF%20Ubuntu%20GCC5?branchName=master&jobName=Platform_CI&configuration=Platform_CI%20OVMF_IA32X64_FULL_NOOPT"/></a>
+ </td>
+ <td></td>
+ </tr>
+</table>
+
+### Config Details
+
+| Config | Architectures |Additional Flags |
+| :---- | :----- | :---- |
+| IA32 | IA32 | None |
+| X64 | X64 | None |
+| IA32X64 | PEI: IA32 DXE: X64 | None |
+| IA32X64 FULL | PEI: IA32 DXE: X64 | SECURE_BOOT_ENABLE=1 SMM_REQUIRE=1 TPM_ENABLE=1 TPM_CONFIG_ENABLE=1 NETWORK_TLS_ENABLE=1 NETWORK_IP6_ENABLE=1 NETWORK_HTTP_BOOT_ENABLE=1 |
+
+## Setup
+
+### The Usual EDK2 Build Setup
+
+- [Python 3.8.x - Download & Install](https://www.python.org/downloads/)
+- [GIT - Download & Install](https://git-scm.com/download/)
+- [GIT - Configure for EDK II](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#github-help)
+- [QEMU - Download, Install, and add to your path](https://www.qemu.org/download/)
+- [EDKII Source - Download/Checkout from Github](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems#download)
+ - **NOTE:** Do _not_ follow the EDK II Compile Tools and Build instructions, see below...
+
+### Differences from EDK Classic Build Setup
+
+- Build BaseTools using "`C:\git\edk2>python BaseTools\Edk2ToolsBuild.py [-t <ToolChainTag>]`"
+ - This replaces "`edksetup Rebuild`" from the classic build system
+ - For Windows `<ToolChainTag>` examples, refer to [Windows ToolChain Matrix](https://github.com/tianocore/tianocore.github.io/wiki/Windows-systems-ToolChain-Matrix), defaults to `VS2017` if not specified
+- **No Action:** Submodule initialization and manual installation/setup of NASM and iASL is **not** required, it is handled by the Pytools build system
+
+### Install & Configure Pytools for OvmfPkg
+
+- Install Pytools
+ - `pip install --upgrade -r pip-requirements.txt`
+- Initialize & Update Submodules
+ - `stuart_setup -c OvmfPkg\PlatformBuild.py`
+- Initialize & Update Dependencies (e.g. iASL & NASM)
+ - `stuart_update -c OvmfPkg\PlatformBuild.py`
+
+## Building
+
+OVMF has [3 versions](https://github.com/tianocore/tianocore.github.io/wiki/How-to-build-OVMF#choosing-which-version-of-ovmf-to-build). To build them using Pytools:
+
+First set the `TOOL_CHAIN_TAG` via environment variable, Conf/target.txt, or pass it on the command-lines below using "`TOOL_CHAIN_TAG=<value>`" syntax.
+
+| Platform | Commandline |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| OvmfPkgIa32X64.dsc | `stuart_build -c OvmfPkg\PlatformBuild.py [TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG>]`<BR>**OR**<BR>`stuart_build -c OvmfPkg\PlatformBuild.py -a IA32,X64 [TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG>]` |
+| OvmfPkgIa32.dsc | `stuart_build -c OvmfPkg\PlatformBuild.py -a IA32 [TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG>]` |
+| OvmfPkgX64.dsc | `stuart_build -c OvmfPkg\PlatformBuild.py -a X64 [TOOL_CHAIN_TAG=<TOOL_CHAIN_TAG>]` |
+
+**NOTE:** configuring ACTIVE*PLATFORM and TARGET_ARCH in Conf/target.txt is *not* required. This environment is set by PlatformBuild.py based upon the "`[-a <TARGET_ARCH>]`" parameter.
+
+### Custom Build Options
+
+**MAKE_STARTUP_NSH=TRUE** will output a _startup.nsh_ file to the location mapped as fs0. This is used in CI in combination with the --FlashOnly feature to run QEMU to the UEFI shell and then execute the contents of startup.nsh.
+
+**QEMU_HEADLESS=TRUE** Since CI servers run headless QEMU must be told to run with no display otherwise an error occurs. Locally you don't need to set this.
+
+### Passing Build Defines
+
+To pass build defines through stuart*build, prepend `BLD_*_`to the define name and pass it on the commandline. stuart_build currently requires values to be assigned, so add an `=1` suffix for bare defines.
+For example, to enable the Intel E1000 NIC, instead of the traditional "-D E1000_ENABLE", the stuart_build command-line would be:
+
+`stuart_build -c OvmfPkg/PlatformBuild.py BLD_*_E1000_ENABLE=1`
+
+## Running QEMU Emulator
+
+QEMU can be automatically launched using stuart_build. This makes path management and quick verification easy.
+QEMU must be added to your path. On Windows this is a manual process and not part of the QEMU installer.
+
+1. To run as part of the build but after building add the `--FlashRom` parameter.
+2. To run after the build process standalone use your build command mentioned above plus `--FlashOnly`.
+
+**NOTE:** Logging the execution output will be in the normal stuart log as well as to your console.
+
+## References
+
+- [Installing Pytools](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md#installing)
+- For each workspace, consider creating & using a [Python Virtual Environment](https://docs.python.org/3/library/venv.html). For example <https://microsoft.github.io/mu/CodeDevelopment/prerequisites/#workspace-virtual-environment-setup-process>
+- [stuart_build commandline parser](https://github.com/tianocore/edk2-pytool-extensions/blob/56f6a7aee09995c2f22da4765e8b0a29c1cbf5de/edk2toolext/edk2_invocable.py#L109)
diff --git a/OvmfPkg/iasl_ext_dep.yaml b/OvmfPkg/iasl_ext_dep.yaml
new file mode 100644
index 000000000000..cbee0e5a5a92
--- /dev/null
+++ b/OvmfPkg/iasl_ext_dep.yaml
@@ -0,0 +1,21 @@
+## @file
+# Download iasl executable tool from a nuget.org package
+# - package contains different binaries based on host
+# Add the folder with the tool to the path
+#
+# This is only downloaded for scope ovmf thus
+# should have no impact on the asl compiler used by any
+# other given platform to build.
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+ "id": "iasl-ovmf-1",
+ "scope": "ovmf",
+ "type": "nuget",
+ "name": "iasl",
+ "source": "https://api.nuget.org/v3/index.json",
+ "version": "20190215.0.0",
+ "flags": ["set_path", "host_specific"],
+}
--
2.16.3.windows.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH v1 4/6] OvmfPkg: Add Platform CI and configuration for Core CI
2020-04-08 18:13 ` [PATCH v1 4/6] OvmfPkg: " Michael Kubacki
@ 2020-04-09 9:17 ` Laszlo Ersek
0 siblings, 0 replies; 29+ messages in thread
From: Laszlo Ersek @ 2020-04-09 9:17 UTC (permalink / raw)
To: michael.kubacki, devel; +Cc: Jordan Justen, Ard Biesheuvel
On 04/08/20 20:13, michael.kubacki@outlook.com wrote:
> From: Sean Brogan <sean.brogan@microsoft.com>
>
> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
>
> Add new Azure Pipeline definitions to build and run OvmfPkg with:
> * Ubuntu GCC5
> * Windows VS2019
> Add PyTool based build of OvmfPkg
> Add extdep for managing the iasl dependency
> Add OvmfPkg.ci.yaml for Core CI
> Add README-pytools for details and instructions
>
> Cc: Jordan Justen <jordan.l.justen@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
> ---
> OvmfPkg/.azurepipelines/Ubuntu-GCC5.yml | 133 +++++++++++
> OvmfPkg/.azurepipelines/Windows-VS2019.yml | 138 +++++++++++
> OvmfPkg/OvmfPkg.ci.yaml | 83 +++++++
> OvmfPkg/PlatformBuild.py | 242 ++++++++++++++++++++
> OvmfPkg/README-pytools.md | 238 +++++++++++++++++++
> OvmfPkg/iasl_ext_dep.yaml | 21 ++
> 6 files changed, 855 insertions(+)
Same comments as for ArmVirtPkg; then I'm ready to ACK. Thanks!
Laszlo
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v1 5/6] .pytool: Update CI Settings to support Emulator, ArmVirt, and Ovmf packages
[not found] <20200408181327.4324-1-michael.kubacki@outlook.com>
` (3 preceding siblings ...)
2020-04-08 18:13 ` [PATCH v1 4/6] OvmfPkg: " Michael Kubacki
@ 2020-04-08 18:13 ` Michael Kubacki
2020-04-08 18:13 ` [PATCH v1 6/6] .azurepipelines: Update Core CI build matrix to include platforms Michael Kubacki
5 siblings, 0 replies; 29+ messages in thread
From: Michael Kubacki @ 2020-04-08 18:13 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bret Barkelew, Michael D Kinney, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
Update pytools configuration to enable EmulatorPkg, ArmVirtPkg,
and OvmfPkg.
Update documentation for the status of those packages.
Remove future work items that are now complete.
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
---
.pytool/CISettings.py | 7 +++++--
.pytool/Plugin/SpellCheck/cspell.base.yaml | 14 ++++++++++++++
.pytool/Readme.md | 10 ++++------
3 files changed, 23 insertions(+), 8 deletions(-)
diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py
index 79593d9dc514..f0c60997d448 100644
--- a/.pytool/CISettings.py
+++ b/.pytool/CISettings.py
@@ -39,7 +39,9 @@ class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSettingsManag
''' return iterable of edk2 packages supported by this build.
These should be edk2 workspace relative paths '''
- return ("MdePkg",
+ return ("ArmVirtPkg",
+ "EmulatorPkg",
+ "MdePkg",
"MdeModulePkg",
"NetworkPkg",
"PcAtChipsetPkg",
@@ -49,7 +51,8 @@ class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSettingsManag
"ShellPkg",
"FatPkg",
"CryptoPkg",
- "UnitTestFrameworkPkg"
+ "UnitTestFrameworkPkg",
+ "OvmfPkg"
)
def GetArchitecturesSupported(self):
diff --git a/.pytool/Plugin/SpellCheck/cspell.base.yaml b/.pytool/Plugin/SpellCheck/cspell.base.yaml
index 5f4c174c77c0..fab3befb90f6 100644
--- a/.pytool/Plugin/SpellCheck/cspell.base.yaml
+++ b/.pytool/Plugin/SpellCheck/cspell.base.yaml
@@ -162,5 +162,19 @@
"Sdhci",
"inmodule",
"RISCV",
+ "edksetup",
+ "iscsi",
+ "nvdata",
+ "pytools",
+ "NTDDI",
+ "Wnonportable",
+ "CLANGPDB",
+ "nologo",
+ "lldmap",
+ "ASMLINK",
+ "NODEFAULTLIB",
+ "vcruntimed",
+ "ucrtd",
+ "msvcrtd",
]
}
diff --git a/.pytool/Readme.md b/.pytool/Readme.md
index 135d283fe223..c7dce3b64ca0 100644
--- a/.pytool/Readme.md
+++ b/.pytool/Readme.md
@@ -6,11 +6,11 @@
| :---- | :----- | :---- | :--- |
| ArmPkg |
| ArmPlatformPkg |
-| ArmVirtPkg |
+| ArmVirtPkg | SEE PACKAGE README | SEE PACKAGE README |
| CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
| DynamicTablesPkg |
| EmbeddedPkg |
-| EmulatorPkg |
+| EmulatorPkg | SEE PACKAGE README | SEE PACKAGE README | Spell checking in audit mode
| FatPkg | :heavy_check_mark: | :heavy_check_mark: |
| FmpDevicePkg | :heavy_check_mark: | :heavy_check_mark: |
| IntelFsp2Pkg |
@@ -18,7 +18,7 @@
| MdeModulePkg | :heavy_check_mark: | :heavy_check_mark: | DxeIpl dependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit mode
| MdePkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
| NetworkPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
-| OvmfPkg |
+| OvmfPkg | SEE PACKAGE README | SEE PACKAGE README | Spell checking in audit mode
| PcAtChipsetPkg | :heavy_check_mark: | :heavy_check_mark: |
| SecurityPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode
| ShellPkg | :heavy_check_mark: | :heavy_check_mark: | Spell checking in audit mode, 3 modules are not being built by DSC
@@ -77,7 +77,7 @@ per package configuration which comes from this file.
## Running CI locally
The EDKII Tools environment (and by extension the ci) is designed to support
-easily and consistantly running locally and in a cloud ci environment. To do
+easily and consistently running locally and in a cloud ci environment. To do
that a few steps should be followed. Details of EDKII Tools can be found in the
[docs folder here](https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs)
@@ -216,8 +216,6 @@ few standard scopes.
* Visual Studio AARCH64 and ARM support
* BaseTools C tools CI/PR and binary release process
* BaseTools Python tools CI/PR process
-* Host based unit testing
* Extensible private/closed source platform reporting
-* Platform builds, validation
* UEFI SCTs
* Other automation
--
2.16.3.windows.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH v1 6/6] .azurepipelines: Update Core CI build matrix to include platforms
[not found] <20200408181327.4324-1-michael.kubacki@outlook.com>
` (4 preceding siblings ...)
2020-04-08 18:13 ` [PATCH v1 5/6] .pytool: Update CI Settings to support Emulator, ArmVirt, and Ovmf packages Michael Kubacki
@ 2020-04-08 18:13 ` Michael Kubacki
5 siblings, 0 replies; 29+ messages in thread
From: Michael Kubacki @ 2020-04-08 18:13 UTC (permalink / raw)
To: devel; +Cc: Sean Brogan, Bret Barkelew, Michael D Kinney, Liming Gao
From: Sean Brogan <sean.brogan@microsoft.com>
REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570
Add ArmVirtPkg to Core CI matrix
Add EmulatorPkg to Core CI matrix
Add OvmfPkg to Core CI matrix
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
---
.azurepipelines/templates/pr-gate-build-job.yml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index 61868554d43c..a9f89aa68451 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -44,6 +44,11 @@ jobs:
TARGET_SECURITY:
Build.Pkgs: 'SecurityPkg'
Build.Targets: 'DEBUG,RELEASE,NO-TARGET'
+ TARGET_PLATFORMS:
+ # For Platforms only check code. Leave it to Platform CI
+ # to build them.
+ Build.Pkgs: 'ArmVirtPkg,EmulatorPkg,OvmfPkg'
+ Build.Targets: 'NO-TARGET'
workspace:
clean: all
--
2.16.3.windows.1
^ permalink raw reply related [flat|nested] 29+ messages in thread