From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (NAM02-SN1-obe.outbound.protection.outlook.com [40.92.5.75]) by mx.groups.io with SMTP id smtpd.web12.1041.1588022469871484095 for ; Mon, 27 Apr 2020 14:21:10 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@outlook.com header.s=selector1 header.b=Uv1A5pe2; spf=pass (domain: outlook.com, ip: 40.92.5.75, mailfrom: michael.kubacki@outlook.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Vj5VE7EkVjW3OtrV4XZSIhUsKdee6hiBBPKkFs4ge5wRauPwxTSbO13bOoOg0zhZv2Gaa19UVF1P+9bUSWc/iHo8c0ZT2cqZEa76XJstGMdL2o2nOY5RSYY6sKiMrLYBWSjyYmuFqqkbQ+AZpz4XXYClfaeW6qkthxdSe3WroEslnUa2aAXR1qvI81pJljr/7LtwH7sDR6/TYZRxm6YyxTqchhgeTxgqnXajCLs/cX9ZGNzGmiHTTrNqZIdCe83XRN0K60wyxcEbYj659g2SN9GAJNKEzVfATo1Owh1M0NQVRW5T3MRREuIx9VWkFemrEbb22bhJ8N/b8FvSQeqBPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XPS4FoKacg1425O63gZ1R1CCH0oSYb60Oe7NtmWMlpA=; b=loRXXKfcntQKBj9SSYT7LChnu9Ajk2d2SmsPhQWNbyj3BXMKaVeqqNLTayVfMxW5Oa1PTZbt/wEwk6ZdDjAHsdrmSX4glgx7grhir6zRX5EPmsRqVOAXTeDtsoCVYWpnpF8nOcEy4hA6undRMBbQ1SxRNVymQs4MlxVHKvc+TQ/gKQSYZdNWldcePKMGzP0cwJ24uWVMk/W8gPToxrCZdRAqq3gSjCiBvgeTXP6B/4Fz0s0udP//mcpHwhY6Un0TlU2zKjIlH7Ca64Tvj0n7L+vz5MfYGwoaMEfOr1yMeUc8SwLWDYGop2y5vrCmHk//TMGD7hEfQ5l0+c46WNQiJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=outlook.com; dmarc=pass action=none header.from=outlook.com; dkim=pass header.d=outlook.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XPS4FoKacg1425O63gZ1R1CCH0oSYb60Oe7NtmWMlpA=; b=Uv1A5pe2Vo7l8yh93EXsjotAcPb4Vke8xsM1+/AFI3cFSFj60vUSdSPplQ79h6U40MaIqtb2+qxadtp1J9VxZaGgPp6+e3+0OnMMty7Iaku1dYK/bqIUAr+rNHwpvBPAsiZV0EymV9veOeQT6c1Ve0Sb35Nl6u7ay7bVvuSANdqWYUEI5ugfw5O5GmF0iDkNeRKbSnCgF+QbP2igcJZJ7n2WbJ8Ow8CMS1UyCW9gVFdrFhLcd9EAgLXkdcWKuSiVllqnxwkUdEM8BmRpMlyodFgJujwIpqqU8kdBIylLE7tcvpdnxZ1THdJ7RhUJDiqAJeQqIp69zolTFhRT2tTsQw== Received: from BL2NAM02FT055.eop-nam02.prod.protection.outlook.com (10.152.76.54) by BL2NAM02HT172.eop-nam02.prod.protection.outlook.com (10.152.77.60) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2937.19; Mon, 27 Apr 2020 21:21:08 +0000 Received: from MWHPR07MB3440.namprd07.prod.outlook.com (2a01:111:e400:7e46::49) by BL2NAM02FT055.mail.protection.outlook.com (2a01:111:e400:7e46::382) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2937.19 via Frontend Transport; Mon, 27 Apr 2020 21:21:08 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:3EF17BD283B9C1209E14EF8C04D2B8002C0EF3C00DDDFACE766A4B497477D8E5;UpperCasedChecksum:DBD8B71A7835D442E781810573D8DC07E362828BF73C20E12F385FDA7E48705B;SizeAsReceived:9145;Count:50 Received: from MWHPR07MB3440.namprd07.prod.outlook.com ([fe80::bcc9:271b:20db:52e3]) by MWHPR07MB3440.namprd07.prod.outlook.com ([fe80::bcc9:271b:20db:52e3%6]) with mapi id 15.20.2937.023; Mon, 27 Apr 2020 21:21:08 +0000 Subject: Re: [PATCH v3 2/7] ArmVirtPkg: Add Platform CI and configuration for Core CI To: Leif Lindholm Cc: devel@edk2.groups.io, Laszlo Ersek , Ard Biesheuvel References: <20200424213108.19888-1-michael.kubacki@outlook.com> <20200427122252.GQ14075@vanye> From: "Michael Kubacki" Message-ID: Date: Mon, 27 Apr 2020 14:21:07 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 In-Reply-To: <20200427122252.GQ14075@vanye> X-ClientProxiedBy: MWHPR12CA0067.namprd12.prod.outlook.com (2603:10b6:300:103::29) To MWHPR07MB3440.namprd07.prod.outlook.com (2603:10b6:301:69::28) Return-Path: michael.kubacki@outlook.com X-Microsoft-Original-Message-ID: <203e2669-2618-864e-0067-c1f730227cf4@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2001:4898:d8:39:b5a9:735c:50d0:9ea6] (2001:4898:80e8:f:35c3:735c:50d0:9ea6) by MWHPR12CA0067.namprd12.prod.outlook.com (2603:10b6:300:103::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2937.13 via Frontend Transport; Mon, 27 Apr 2020 21:21:07 +0000 X-Microsoft-Original-Message-ID: <203e2669-2618-864e-0067-c1f730227cf4@outlook.com> X-TMN: [1EWpFKMveahJpzY9rabgtdTN53mBXOs4iIPpMdaiP/FjWs2PaM0ZA39qrWziXRHX] X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 50 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: a4539ddf-3c7c-42d9-b96e-08d7eaf0e667 X-MS-TrafficTypeDiagnostic: BL2NAM02HT172: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /wqOzfuMGv4tim3JbbcxMr9cLZyM3gTd4j0M9zimMYB9gZqQijIJe1EfWqoU9irCYPQV2hRGfnOvHfVRUKoAAVUPXGiVc1Mx61OeQFqGq0S12z2ywo3V5GLc1RN66lKWZ9oRoOZh6L9+Vjr2Z69xyJYcAGtUZxMANw4uFn161ateZb9e7vjmNfU2jiDm2TF4XGgXXV/RcYcdBsLg5KQRwA5u3sd+JSXmx+MSe8AV+JoIHbsOri52+qDkOK/WFwJm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:0;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR07MB3440.namprd07.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:;DIR:OUT;SFP:1901; X-MS-Exchange-AntiSpam-MessageData: qVMx+En2EMRhsg8cMs8zRNbpzO8FHuxwHkzx/r11Gb8btWZrxIj7/mog9lC5JiCXgR8x3S+f/XV5Rjl8K1XFSsMp3zFDQOVL6WN90HglX8I1nz7Q616iD0sOG1qH3bL54kX72k06dsj0qUOb0wdqWi6OWNK6PmusCX+oO6onaHGc+YFAZeAaMzG/U7B4yLMgZS37nKkLVlKBAjlpWeUwYQ== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a4539ddf-3c7c-42d9-b96e-08d7eaf0e667 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2020 21:21:08.1208 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2NAM02HT172 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit I believe I misunderstood the conclusion of which Signed-off-by to include in the v1 series discussion - https://edk2.groups.io/g/devel/message/57129. These patches are taken from a branch on Sean's fork of edk2 as noted in the cover letter - https://edk2.groups.io/g/devel/message/58114. Sean has given a Signed-off-by on these patches in that branch. Is the guidance in this case to only include my Signed-off-by? Thanks, Michael On 4/27/2020 5:22 AM, Leif Lindholm wrote: > On Fri, Apr 24, 2020 at 14:31:03 -0700, michael.kubacki@outlook.com wrote: >> From: Sean Brogan >> >> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2570 >> >> Add new Azure Pipeline definitions to build and run ArmVirtPkg with: >> * Ubuntu GCC5 >> Add PyTool based build of ArmVirtPkg >> Add extdep for managing the iasl dependency >> Add ArmVirtPkg.ci.yaml for Core CI >> Add ReadMe.md for details and instructions >> >> Cc: Laszlo Ersek >> Cc: Ard Biesheuvel >> Cc: Leif Lindholm >> Signed-off-by: Sean Brogan >> Signed-off-by: Michael Kubacki > > The contributor attests the compliance of the submission with > https://developercertificate.org/ at the point of contribution by > using Signed-off-by. > > Statements by others (including original authors) should not be copied > when copying code between projects. The From: tag preserves authorship > credit. > > / > Leif > >> Reviewed-by: Shenglei Zhang >> Acked-by: Laszlo Ersek >> Acked-by: Ard Biesheuvel >> Reviewed-by: Bret Barkelew >> --- >> ArmVirtPkg/ArmVirtPkg.ci.yaml | 103 ++++++++ >> ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml | 89 +++++++ >> ArmVirtPkg/PlatformCI/PlatformBuild.py | 276 ++++++++++++++++++++ >> ArmVirtPkg/PlatformCI/ReadMe.md | 125 +++++++++ >> ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml | 21 ++ >> 5 files changed, 614 insertions(+) >> >> diff --git a/ArmVirtPkg/ArmVirtPkg.ci.yaml b/ArmVirtPkg/ArmVirtPkg.ci.yaml >> new file mode 100644 >> index 000000000000..4553725ee528 >> --- /dev/null >> +++ b/ArmVirtPkg/ArmVirtPkg.ci.yaml >> @@ -0,0 +1,103 @@ >> +## @file >> +# Core CI configuration for ArmVirtPkg >> +# >> +# ArmVirtPkg is part of Platform Ci for builds so this is only >> +# used for code analysis. >> +# >> +# Copyright (c) Microsoft Corporation >> +# SPDX-License-Identifier: BSD-2-Clause-Patent >> +## >> +{ >> + ## options defined .pytool/Plugin/CompilerPlugin >> + "CompilerPlugin": { >> + "DscPath": "" # Don't support this test >> + }, >> + >> + ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin >> + "HostUnitTestCompilerPlugin": { >> + "DscPath": "" # Don't support this test >> + }, >> + >> + ## options defined .pytool/Plugin/CharEncodingCheck >> + "CharEncodingCheck": { >> + "IgnoreFiles": [] >> + }, >> + >> + ## options defined .pytool/Plugin/DependencyCheck >> + "DependencyCheck": { >> + "AcceptableDependencies": [ >> + "MdePkg/MdePkg.dec", >> + "MdeModulePkg/MdeModulePkg.dec", >> + "ArmVirtPkg/ArmVirtPkg.dec", >> + "NetworkPkg/NetworkPkg.dec", >> + "ArmPkg/ArmPkg.dec", >> + "OvmfPkg/OvmfPkg.dec", >> + "EmbeddedPkg/EmbeddedPkg.dec", >> + "ArmPlatformPkg/ArmPlatformPkg.dec", >> + "SecurityPkg/SecurityPkg.dec", >> + "ShellPkg/ShellPkg.dec" #Is this ok? >> + ], >> + # For host based unit tests >> + "AcceptableDependencies-HOST_APPLICATION":[ >> + "UnitTestFrameworkPkg/UnitTestFrameworkPkg.dec" >> + ], >> + # For UEFI shell based apps >> + "AcceptableDependencies-UEFI_APPLICATION":[ >> + >> + ], >> + "IgnoreInf": [] >> + }, >> + >> + ## options defined .pytool/Plugin/DscCompleteCheck >> + "DscCompleteCheck": { >> + "IgnoreInf": [""], >> + "DscPath": "" # Don't support this test >> + }, >> + >> + ## options defined .pytool/Plugin/HostUnitTestDscCompleteCheck >> + "HostUnitTestDscCompleteCheck": { >> + "IgnoreInf": [""], >> + "DscPath": "" # Don't support this test >> + }, >> + >> + ## options defined .pytool/Plugin/GuidCheck >> + "GuidCheck": { >> + "IgnoreGuidName": [], >> + "IgnoreGuidValue": [], >> + "IgnoreFoldersAndFiles": [], >> + "IgnoreDuplicates": [], >> + }, >> + >> + ## options defined .pytool/Plugin/LibraryClassCheck >> + "LibraryClassCheck": { >> + "IgnoreHeaderFile": [] >> + }, >> + >> + ## options defined .pytool/Plugin/SpellCheck >> + "SpellCheck": { >> + "AuditOnly": False, # Fails right now with over 270 errors >> + "IgnoreFiles": [], # use gitignore syntax to ignore errors in matching files >> + "ExtendWords": [ >> + "setjump", >> + "plong", >> + "lparam", >> + "lpdword", >> + "lpthread", >> + "lresult", >> + "bootable", >> + "bsymbolic", >> + "endiannness", >> + "fvmain", >> + "multiboot", >> + "qemu's", >> + "ramdisk", >> + "ramfb", >> + "unbootable", >> + "virt's", >> + "werror", >> + "xenio" >> + ], # words to extend to the dictionary for this package >> + "IgnoreStandardPaths": [], # Standard Plugin defined paths that should be ignore >> + "AdditionalIncludePaths": [] # Additional paths to spell check (wildcards supported) >> + } >> +} >> diff --git a/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml >> new file mode 100644 >> index 000000000000..b9452ec5bfaf >> --- /dev/null >> +++ b/ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml >> @@ -0,0 +1,89 @@ >> +## @file >> +# Azure Pipeline build file for building a platform. >> +# >> +# Platform: ArmVirtQemu >> +# OS: Ubuntu >> +# Toolchain: GCC5 >> +# >> +# Copyright (c) Microsoft Corporation. >> +# SPDX-License-Identifier: BSD-2-Clause-Patent >> +## >> +trigger: >> + - master >> +pr: >> + - master >> + >> +jobs: >> + - job: Platform_CI >> + variables: >> + package: 'ArmVirtPkg' >> + vm_image: 'ubuntu-latest' >> + should_run: true >> + run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE" >> + >> + #Use matrix to speed up the build process >> + strategy: >> + matrix: >> + QEMU_AARCH64_DEBUG: >> + Build.File: "$(package)/PlatformCI/PlatformBuild.py" >> + Build.Arch: "AARCH64" >> + Build.Flags: "" >> + Build.Target: "DEBUG" >> + Run.Flags: $(run_flags) >> + Run: $(should_run) >> + QEMU_AARCH64_RELEASE: >> + Build.File: "$(package)/PlatformCI/PlatformBuild.py" >> + Build.Arch: "AARCH64" >> + Build.Flags: "" >> + Build.Target: "RELEASE" >> + Run.Flags: $(run_flags) >> + Run: $(should_run) >> + QEMU_AARCH64_NOOPT: >> + Build.File: "$(package)/PlatformCI/PlatformBuild.py" >> + Build.Arch: "AARCH64" >> + Build.Flags: "" >> + Build.Target: "NOOPT" >> + Run.Flags: $(run_flags) >> + Run: $(should_run) >> + QEMU_ARM_DEBUG: >> + Build.File: "$(package)/PlatformCI/PlatformBuild.py" >> + Build.Arch: "ARM" >> + Build.Flags: "" >> + Build.Target: "DEBUG" >> + Run.Flags: $(run_flags) >> + Run: $(should_run) >> + QEMU_ARM_RELEASE: >> + Build.File: "$(package)/PlatformCI/PlatformBuild.py" >> + Build.Arch: "ARM" >> + Build.Flags: "" >> + Build.Target: "RELEASE" >> + Run.Flags: $(run_flags) >> + Run: $(should_run) >> + QEMU_ARM_NOOPT: >> + Build.File: "$(package)/PlatformCI/PlatformBuild.py" >> + Build.Arch: "ARM" >> + Build.Flags: "" >> + Build.Target: "NOOPT" >> + Run.Flags: $(run_flags) >> + Run: $(should_run) >> + >> + workspace: >> + clean: all >> + >> + pool: >> + vmImage: $(vm_image) >> + >> + steps: >> + - template: ../../../.azurepipelines/templates/platform-build-run-steps.yml >> + parameters: >> + tool_chain_tag: GCC5 >> + build_pkg: $(package) >> + build_target: $(Build.Target) >> + build_arch: $(Build.Arch) >> + build_file: $(Build.File) >> + build_flags: $(Build.Flags) >> + run_flags: $(Run.Flags) >> + extra_install_step: >> + - bash: sudo apt-get install qemu >> + displayName: Install qemu >> + condition: and(gt(variables.pkg_count, 0), succeeded()) >> diff --git a/ArmVirtPkg/PlatformCI/PlatformBuild.py b/ArmVirtPkg/PlatformCI/PlatformBuild.py >> new file mode 100644 >> index 000000000000..dff653e919eb >> --- /dev/null >> +++ b/ArmVirtPkg/PlatformCI/PlatformBuild.py >> @@ -0,0 +1,276 @@ >> +# @file >> +# Script to Build ArmVirtPkg UEFI firmware >> +# >> +# Copyright (c) Microsoft Corporation. >> +# SPDX-License-Identifier: BSD-2-Clause-Patent >> +## >> +import os >> +import logging >> +import io >> + >> +from edk2toolext.environment import shell_environment >> +from edk2toolext.environment.uefi_build import UefiBuilder >> +from edk2toolext.invocables.edk2_platform_build import BuildSettingsManager >> +from edk2toolext.invocables.edk2_setup import SetupSettingsManager, RequiredSubmodule >> +from edk2toolext.invocables.edk2_update import UpdateSettingsManager >> +from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager >> +from edk2toollib.utility_functions import RunCmd >> +from edk2toollib.utility_functions import GetHostInfo >> + >> +# ####################################################################################### # >> +# Common Configuration # >> +# ####################################################################################### # >> + >> + >> +class CommonPlatform(): >> + ''' Common settings for this platform. Define static data here and use >> + for the different parts of stuart >> + ''' >> + PackagesSupported = ("ArmVirtPkg",) >> + ArchSupported = ("AARCH64", "ARM") >> + TargetsSupported = ("DEBUG", "RELEASE", "NOOPT") >> + Scopes = ('armvirt', 'edk2-build') >> + WorkspaceRoot = os.path.realpath(os.path.join( >> + os.path.dirname(os.path.abspath(__file__)), "..", "..")) >> + >> + # ####################################################################################### # >> + # Configuration for Update & Setup # >> + # ####################################################################################### # >> + >> + >> +class SettingsManager(UpdateSettingsManager, SetupSettingsManager, PrEvalSettingsManager): >> + >> + def GetPackagesSupported(self): >> + ''' return iterable of edk2 packages supported by this build. >> + These should be edk2 workspace relative paths ''' >> + return CommonPlatform.PackagesSupported >> + >> + def GetArchitecturesSupported(self): >> + ''' return iterable of edk2 architectures supported by this build ''' >> + return CommonPlatform.ArchSupported >> + >> + def GetTargetsSupported(self): >> + ''' return iterable of edk2 target tags supported by this build ''' >> + return CommonPlatform.TargetsSupported >> + >> + def GetRequiredSubmodules(self): >> + ''' return iterable containing RequiredSubmodule objects. >> + If no RequiredSubmodules return an empty iterable >> + ''' >> + rs = [] >> + >> + # intentionally declare this one with recursive false to avoid overhead >> + rs.append(RequiredSubmodule( >> + "CryptoPkg/Library/OpensslLib/openssl", False)) >> + >> + # To avoid maintenance of this file for every new submodule >> + # lets just parse the .gitmodules and add each if not already in list. >> + # The GetRequiredSubmodules is designed to allow a build to optimize >> + # the desired submodules but it isn't necessary for this repository. >> + result = io.StringIO() >> + ret = RunCmd("git", "config --file .gitmodules --get-regexp path", workingdir=self.GetWorkspaceRoot(), outstream=result) >> + # Cmd output is expected to look like: >> + # submodule.CryptoPkg/Library/OpensslLib/openssl.path CryptoPkg/Library/OpensslLib/openssl >> + # submodule.SoftFloat.path ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 >> + if ret == 0: >> + for line in result.getvalue().splitlines(): >> + _, _, path = line.partition(" ") >> + if path is not None: >> + if path not in [x.path for x in rs]: >> + rs.append(RequiredSubmodule(path, True)) # add it with recursive since we don't know >> + return rs >> + >> + def SetArchitectures(self, list_of_requested_architectures): >> + ''' Confirm the requests architecture list is valid and configure SettingsManager >> + to run only the requested architectures. >> + >> + Raise Exception if a list_of_requested_architectures is not supported >> + ''' >> + unsupported = set(list_of_requested_architectures) - \ >> + set(self.GetArchitecturesSupported()) >> + if(len(unsupported) > 0): >> + errorString = ( >> + "Unsupported Architecture Requested: " + " ".join(unsupported)) >> + logging.critical(errorString) >> + raise Exception(errorString) >> + self.ActualArchitectures = list_of_requested_architectures >> + >> + def GetWorkspaceRoot(self): >> + ''' get WorkspacePath ''' >> + return CommonPlatform.WorkspaceRoot >> + >> + def GetActiveScopes(self): >> + ''' return tuple containing scopes that should be active for this process ''' >> + >> + scopes = CommonPlatform.Scopes >> + ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "") >> + >> + if GetHostInfo().os.upper() == "LINUX" and ActualToolChainTag.upper().startswith("GCC"): >> + if "AARCH64" in self.ActualArchitectures: >> + scopes += ("gcc_aarch64_linux",) >> + if "ARM" in self.ActualArchitectures: >> + scopes += ("gcc_arm_linux",) >> + return scopes >> + >> + def FilterPackagesToTest(self, changedFilesList: list, potentialPackagesList: list) -> list: >> + ''' Filter other cases that this package should be built >> + based on changed files. This should cover things that can't >> + be detected as dependencies. ''' >> + build_these_packages = [] >> + possible_packages = potentialPackagesList.copy() >> + for f in changedFilesList: >> + # BaseTools files that might change the build >> + if "BaseTools" in f: >> + if os.path.splitext(f) not in [".txt", ".md"]: >> + build_these_packages = possible_packages >> + break >> + >> + # if the azure pipeline platform template file changed >> + if "platform-build-run-steps.yml" in f: >> + build_these_packages = possible_packages >> + break >> + >> + >> + return build_these_packages >> + >> + def GetPlatformDscAndConfig(self) -> tuple: >> + ''' If a platform desires to provide its DSC then Policy 4 will evaluate if >> + any of the changes will be built in the dsc. >> + >> + The tuple should be (, ) >> + ''' >> + return (os.path.join("ArmVirtPkg", "ArmVirtQemu.dsc"), {}) >> + >> + >> + # ####################################################################################### # >> + # Actual Configuration for Platform Build # >> + # ####################################################################################### # >> + >> + >> +class PlatformBuilder(UefiBuilder, BuildSettingsManager): >> + def __init__(self): >> + UefiBuilder.__init__(self) >> + >> + def AddCommandLineOptions(self, parserObj): >> + ''' Add command line options to the argparser ''' >> + parserObj.add_argument('-a', "--arch", dest="build_arch", type=str, default="AARCH64", >> + help="Optional - Architecture to build. Default = AARCH64") >> + >> + def RetrieveCommandLineOptions(self, args): >> + ''' Retrieve command line options from the argparser ''' >> + >> + shell_environment.GetBuildVars().SetValue( >> + "TARGET_ARCH", args.build_arch.upper(), "From CmdLine") >> + >> + shell_environment.GetBuildVars().SetValue( >> + "ACTIVE_PLATFORM", "ArmVirtPkg/ArmVirtQemu.dsc", "From CmdLine") >> + >> + def GetWorkspaceRoot(self): >> + ''' get WorkspacePath ''' >> + return CommonPlatform.WorkspaceRoot >> + >> + def GetPackagesPath(self): >> + ''' Return a list of workspace relative paths that should be mapped as edk2 PackagesPath ''' >> + return () >> + >> + def GetActiveScopes(self): >> + ''' return tuple containing scopes that should be active for this process ''' >> + scopes = CommonPlatform.Scopes >> + ActualToolChainTag = shell_environment.GetBuildVars().GetValue("TOOL_CHAIN_TAG", "") >> + Arch = shell_environment.GetBuildVars().GetValue("TARGET_ARCH", "") >> + >> + if GetHostInfo().os.upper() == "LINUX" and ActualToolChainTag.upper().startswith("GCC"): >> + if "AARCH64" == Arch: >> + scopes += ("gcc_aarch64_linux",) >> + elif "ARM" == Arch: >> + scopes += ("gcc_arm_linux",) >> + return scopes >> + >> + def GetName(self): >> + ''' Get the name of the repo, platform, or product being build ''' >> + ''' Used for naming the log file, among others ''' >> + # check the startup nsh flag and if set then rename the log file. >> + # this helps in CI so we don't overwrite the build log since running >> + # uses the stuart_build command. >> + if(shell_environment.GetBuildVars().GetValue("MAKE_STARTUP_NSH", "FALSE") == "TRUE"): >> + return "ArmVirtPkg_With_Run" >> + return "ArmVirtPkg" >> + >> + def GetLoggingLevel(self, loggerType): >> + ''' Get the logging level for a given type >> + base == lowest logging level supported >> + con == Screen logging >> + txt == plain text file logging >> + md == markdown file logging >> + ''' >> + return logging.DEBUG >> + >> + def SetPlatformEnv(self): >> + logging.debug("PlatformBuilder SetPlatformEnv") >> + self.env.SetValue("PRODUCT_NAME", "ArmVirtQemu", "Platform Hardcoded") >> + self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false") >> + self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false") >> + return 0 >> + >> + def PlatformPreBuild(self): >> + return 0 >> + >> + def PlatformPostBuild(self): >> + return 0 >> + >> + def FlashRomImage(self): >> + VirtualDrive = os.path.join(self.env.GetValue( >> + "BUILD_OUTPUT_BASE"), "VirtualDrive") >> + os.makedirs(VirtualDrive, exist_ok=True) >> + OutputPath_FV = os.path.join( >> + self.env.GetValue("BUILD_OUTPUT_BASE"), "FV") >> + Built_FV = os.path.join(OutputPath_FV, "QEMU_EFI.fd") >> + >> + # pad fd to 64mb >> + with open(Built_FV, "ab") as fvfile: >> + fvfile.seek(0, os.SEEK_END) >> + additional = b'\0' * ((64 * 1024 * 1024)-fvfile.tell()) >> + fvfile.write(additional) >> + >> + # QEMU must be on that path >> + >> + # Unique Command and Args parameters per ARCH >> + if (self.env.GetValue("TARGET_ARCH").upper() == "AARCH64"): >> + cmd = "qemu-system-aarch64" >> + args = "-M virt" >> + args += " -cpu cortex-a57" # emulate cpu >> + elif(self.env.GetValue("TARGET_ARCH").upper() == "ARM"): >> + cmd = "qemu-system-arm" >> + args = "-M virt" >> + args += " -cpu cortex-a15" # emulate cpu >> + else: >> + raise NotImplementedError() >> + >> + # Common Args >> + args += " -pflash " + Built_FV # path to fw >> + args += " -m 1024" # 1gb memory >> + # turn off network >> + args += " -net none" >> + # Serial messages out >> + args += " -serial stdio" >> + # Mount disk with startup.nsh >> + args += f" -drive file=fat:rw:{VirtualDrive},format=raw,media=disk" >> + >> + # Conditional Args >> + if (self.env.GetValue("QEMU_HEADLESS").upper() == "TRUE"): >> + args += " -display none" # no graphics >> + >> + if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"): >> + f = open(os.path.join(VirtualDrive, "startup.nsh"), "w") >> + f.write("BOOT SUCCESS !!! \n") >> + # add commands here >> + f.write("reset -s\n") >> + f.close() >> + >> + ret = RunCmd(cmd, args) >> + >> + if ret == 0xc0000005: >> + # for some reason getting a c0000005 on successful return >> + return 0 >> + >> + return ret >> diff --git a/ArmVirtPkg/PlatformCI/ReadMe.md b/ArmVirtPkg/PlatformCI/ReadMe.md >> new file mode 100644 >> index 000000000000..7c11d925f59e >> --- /dev/null >> +++ b/ArmVirtPkg/PlatformCI/ReadMe.md >> @@ -0,0 +1,125 @@ >> +# ArmVirtPkg - Platform CI >> + >> +This Readme.md describes the Azure DevOps based Platform CI for ArmVirtPkg and how >> +to use the same Pytools based build infrastructure locally. >> + >> +## Supported Configuration Details >> + >> +This solution for building and running ArmVirtPkg has only been validated with Ubuntu >> +18.04 and the GCC5 toolchain. Two different firmware builds are supported and are >> +described below. >> + >> +| Configuration name | Architecture | DSC File |Additional Flags | >> +| :---------- | :----- | :----- | :---- | >> +| AARCH64 | AARCH64 | ArmVirtQemu.dsc | None | >> +| ARM | ARM | ArmVirtQemu.dsc | None | >> + >> +## EDK2 Developer environment >> + >> +- [Python 3.8.x - Download & Install](https://www.python.org/downloads/) >> +- [GIT - Download & Install](https://git-scm.com/download/) >> +- [QEMU - Download, Install, and add to your path](https://www.qemu.org/download/) >> +- [Edk2 Source](https://github.com/tianocore/edk2) >> +- Additional packages found necessary for Ubuntu 18.04 >> + - apt-get install gcc g++ make uuid-dev >> + >> +Note: edksetup, Submodule initialization and manual installation of NASM, iASL, or >> +the required cross-compiler toolchains are **not** required, this is handled by the >> +Pytools build system. >> + >> +## Building with Pytools for ArmVirtPkg >> + >> +1. [Optional] Create a Python Virtual Environment - generally once per workspace >> + >> + ``` bash >> + python -m venv >> + ``` >> + >> +2. [Optional] Activate Virtual Environment - each time new shell opened >> + - Windows >> + >> + ``` bash >> + /Scripts/activate.bat >> + ``` >> + >> + - Linux >> + >> + ```bash >> + source /bin/activate >> + ``` >> + >> +3. Install Pytools - generally once per virtual env or whenever pip-requirements.txt changes >> + >> + ``` bash >> + pip install --upgrade -r pip-requirements.txt >> + ``` >> + >> +4. Initialize & Update Submodules - only when submodules updated >> + >> + ``` bash >> + stuart_setup -c ArmVirtPkg/PlatformCI/PlatformBuild.py TOOL_CHAIN_TAG= -a >> + ``` >> + >> +5. Initialize & Update Dependencies - only as needed when ext_deps change >> + >> + ``` bash >> + stuart_update -c ArmVirtPkg/PlatformCI/PlatformBuild.py TOOL_CHAIN_TAG= -a >> + ``` >> + >> +6. Compile the basetools if necessary - only when basetools C source files change >> + >> + ``` bash >> + python BaseTools/Edk2ToolsBuild.py -t >> + ``` >> + >> +7. Compile Firmware >> + >> + ``` bash >> + stuart_build -c ArmVirtPkg/PlatformCI/PlatformBuild.py TOOL_CHAIN_TAG= -a >> + ``` >> + >> + - use `stuart_build -c ArmVirtPkg/PlatformCI/PlatformBuild.py -h` option to see additional >> + options like `--clean` >> + >> +8. Running Emulator >> + - You can add `--FlashRom` to the end of your build command and the emulator will run after the >> + build is complete. >> + - or use the `--FlashOnly` feature to just run the emulator. >> + >> + ``` bash >> + stuart_build -c ArmVirtPkg/PlatformCI/PlatformBuild.py TOOL_CHAIN_TAG= -a --FlashOnly >> + ``` >> + >> +### Notes >> + >> +1. Including the expected build architecture and toolchain to the _stuart_update_ command is critical. >> + This is because there are extra scopes and tools that will be resolved during the update step that >> + need to match your build step. >> +2. Configuring *ACTIVE_PLATFORM* and *TARGET_ARCH* in Conf/target.txt is **not** required. This >> + environment is set by PlatformBuild.py based upon the `[-a ]` parameter. >> +3. QEMU must be on your path. On Windows this is a manual process and not part of the QEMU installer. >> + >> +**NOTE:** Logging the execution output will be in the normal stuart log as well as to your console. >> + >> +### Custom Build Options >> + >> +**MAKE_STARTUP_NSH=TRUE** will output a *startup.nsh* file to the location mapped as fs0. This is >> +used in CI in combination with the `--FlashOnly` feature to run QEMU to the UEFI shell and then execute >> +the contents of *startup.nsh*. >> + >> +**QEMU_HEADLESS=TRUE** Since CI servers run headless QEMU must be told to run with no display otherwise >> +an error occurs. Locally you don't need to set this. >> + >> +### Passing Build Defines >> + >> +To pass build defines through _stuart_build_, prepend `BLD_*_`to the define name and pass it on the >> +command-line. _stuart_build_ currently requires values to be assigned, so add an`=1` suffix for bare defines. >> +For example, to enable the TPM2 support, instead of the traditional "-D TPM2_ENABLE=TRUE", the stuart_build >> +command-line would be: >> + >> +`stuart_build -c ArmVirtPkg/PlatformCI/PlatformBuild.py BLD_*_TPM2_ENABLE=TRUE` >> + >> +## References >> + >> +- [Installing and using Pytools](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md#installing) >> +- More on [python virtual environments](https://docs.python.org/3/library/venv.html) >> diff --git a/ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml b/ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml >> new file mode 100644 >> index 000000000000..8869ed3ecef1 >> --- /dev/null >> +++ b/ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml >> @@ -0,0 +1,21 @@ >> +## @file >> +# Download iasl executable tool from a nuget.org package >> +# - package contains different binaries based on host >> +# Add the folder with the tool to the path >> +# >> +# This is only downloaded for scope armvirt thus >> +# should have no impact on the asl compiler used by any >> +# other platform build >> +# >> +# Copyright (c) Microsoft Corporation. >> +# SPDX-License-Identifier: BSD-2-Clause-Patent >> +## >> +{ >> + "id": "iasl-armvirt-1", >> + "scope": "armvirt", >> + "type": "nuget", >> + "name": "iasl", >> + "source": "https://api.nuget.org/v3/index.json", >> + "version": "20190215.0.0", >> + "flags": ["set_path", "host_specific"], >> +} >> -- >> 2.16.3.windows.1 >>