From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (NAM11-CO1-obe.outbound.protection.outlook.com [40.92.18.92]) by mx.groups.io with SMTP id smtpd.web10.144255.1598045773734098089 for ; Fri, 21 Aug 2020 14:36:14 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@outlook.com header.s=selector1 header.b=S+cUQQqj; spf=pass (domain: outlook.com, ip: 40.92.18.92, mailfrom: spbrogan@outlook.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bJoC+tsU0Ec7BmN7SJBqsS6xQdJ1p+k7zr2yx9krvoBogLmwbF7x6E6h3wdhGVCCssgSQceIlG3dE5JayPN6YdI9b/YyA47vCc6rHjR2RD0gmn04s8H2iAxI0dtCz8VmrBsoQhBebfiDKbkaQHUmF2ghl8W6jQrvaT8F8BcjXOcGBy+jcQTzkNvVPysKMoDM+8kvnSubhMbQTaXhAFSZXSiYEZ9Eq2cvB2wxlUw+olqB+sKVbTodnNt79yDyOstNMJtMx21p4zaCiHnZMRBwvUWHaUjwW8JAAOWrdCHWkVWSCToSvZtivtoxXFm2ne6ZXCYsJkeE1CshOcTWbuyGxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OB9nzxAiOtosdzstxDRtie8a/0zA8p8lZlaXIF6GWDE=; b=QzJAWvrjZP6h0nSp0ZlS+st2BYFcROfUkM9xsWid2ejznk8NPZcx5AvuabA7jy74jScmoGJSDlrdpDZzxdFbICymPNCudyEl5EwVxa6EKAX7c4d8YwTuHiFltpI1E+A3oGjXL6Xl8qgGa7pwRWu4u07s4hSC63Csinq+tnXqeSPRh/FiEeh51hoLHp5lwalB0GwgSzWU6w4Z9n+gDZ/70sANjJeesE5+KvNZRicLMuA9eMLm7yHaeijg3MlTM5Kas6hjelG+1hcqmYNxh1Hoatc69/qZMLnk9RCfKws7Z8O+mIJxoUchKGVHdHvPmPzJCTHVVOQrINH83aRUWh0y0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OB9nzxAiOtosdzstxDRtie8a/0zA8p8lZlaXIF6GWDE=; b=S+cUQQqjGzGL9rW5h/aK+nwx4/QZkI5SwEdbomkjEh6LY3AlnAQVanz/ALa3JatQOd34QX40kgP/IhC/7MoSmlMWVcYeuGa+RccXDd4pjjDCTT5nHp+F91ZSLB3uYjNYYoD7iI76UjaJLptpum4Kl7Z3+BmtcfP2AmYuYexIEM2XJpHJJZ7keRDyjJISdC4KSHsFdfUuInRohDk+VdTzDeX+1fZcJDW1GvS43BqTd/LG+6kYMaejaMKpc8L668vU2GTR8Q3CAj3WrRY+4GEXThdx/KVKGuMvPu/jA5Z6l0w07uR9LwxCnTGWioy4MuXRysaYD6Bs9jW+30fl99deLA== Received: from BN8NAM11FT024.eop-nam11.prod.protection.outlook.com (2a01:111:e400:fc4b::4f) by BN8NAM11HT044.eop-nam11.prod.protection.outlook.com (2a01:111:e400:fc4b::127) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3305.24; Fri, 21 Aug 2020 21:36:12 +0000 Received: from BN8PR07MB6962.namprd07.prod.outlook.com (2a01:111:e400:fc4b::52) by BN8NAM11FT024.mail.protection.outlook.com (2a01:111:e400:fc4b::294) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3305.24 via Frontend Transport; Fri, 21 Aug 2020 21:36:12 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:1834FAB84B019B51B347ABEC738217F6E961D95C500519D361B717A9818491AA;UpperCasedChecksum:6B1CE429690A1A623C36F7D3659A1F28F06116894BEEF76D626EAA9DBCC2C192;SizeAsReceived:9245;Count:49 Received: from BN8PR07MB6962.namprd07.prod.outlook.com ([fe80::b1be:f3e4:f6e:66c3]) by BN8PR07MB6962.namprd07.prod.outlook.com ([fe80::b1be:f3e4:f6e:66c3%4]) with mapi id 15.20.3305.026; Fri, 21 Aug 2020 21:36:12 +0000 Subject: Re: [edk2-devel] running CI locally To: Laszlo Ersek , devel@edk2.groups.io, michael.d.kinney@intel.com, Sean Brogan , Bret Barkelew Cc: Vladimir Olovyannikov , Rebecca Cran , Tom Lendacky References: <2d9572c6-7549-d0ae-3ec9-95158cfef484@redhat.com> <5ca9187c-96f3-53f1-0ac0-809bc1156ece@redhat.com> From: "Sean" Message-ID: Date: Fri, 21 Aug 2020 14:36:09 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 In-Reply-To: <5ca9187c-96f3-53f1-0ac0-809bc1156ece@redhat.com> X-ClientProxiedBy: MWHPR18CA0060.namprd18.prod.outlook.com (2603:10b6:300:39::22) To BN8PR07MB6962.namprd07.prod.outlook.com (2603:10b6:408:d6::11) Return-Path: spbrogan@outlook.com X-Microsoft-Original-Message-ID: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from 255.255.255.255 (255.255.255.255) by MWHPR18CA0060.namprd18.prod.outlook.com (2603:10b6:300:39::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3305.24 via Frontend Transport; Fri, 21 Aug 2020 21:36:11 +0000 X-Microsoft-Original-Message-ID: X-TMN: [t7CdA83SyUCzx7Q6SzEh1/mt14Jf2CEO] X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 49 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: 1d447433-454c-4fb8-6930-08d8461a397f X-MS-TrafficTypeDiagnostic: BN8NAM11HT044: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wR9DhOFkPhxKP1mCmZrdKcVcexHMNLtqsb8l2blczyrEMsEkGtdV7Ypq5xpPwx9YSagfPrsZ3q3Pz8obw8mMgNpeSSRhwmrvxTgoMVEED1okzeBev9OsUDhl7FFCZzLXWb81I8BE2h848k+Czjj1rBrxs0mAbx8V1TJlh3UF1QiMZ/140RIpzbjuKtaVoacHYtA2nBiW5ycw7jb+eVVniUcEMaO36vbNmILTTMD1snre/JytWC2WbyHStbIaBoYa X-MS-Exchange-AntiSpam-MessageData: pgnrTEsYOKIuxd2LgZjVX26KO4f8LPggJ0rs/KNlPrOx8BdsOy/s1Tra09H6urJLRd4ljUP27OKZApMoSXYVFJy94fbsnwnZF99CquX7OpcVxAPTQ+eoAJi54CmuhvHknjd013gGshJJSp/IMORfFw== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1d447433-454c-4fb8-6930-08d8461a397f X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Aug 2020 21:36:12.3428 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT024.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8NAM11HT044 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Laszlo, Idea being if the package supports host based unit tests it will scan package to make sure all host based unit tests and libraries are listed in the DSC. As to why you see a different set of packages...that is interesting. If you can include your full build logs i can look a little closer. I know on the Azure CI servers we don't run the On 8/21/2020 12:23 AM, Laszlo Ersek wrote: > Hi Mike, Sean; > > On 08/19/20 19:59, Laszlo Ersek wrote: > >> I'll report back with more results. > > I installed a Fedora 32 Server VM. > > Installed the "mono-complete" package (+its deps). > > Installed the "nodejs" package. > > Installed "cspell" (+deps) with "npm" (with the latter coming from the > nodejs package). > > Edk2 checked out at current HEAD > (5a6d764e1d073d28e8f398289ccb5592bf9a72ba). > > > * Platform CI results: > > - : > > Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5). > > - : > > Built successfully for ARM and AARCH64 (toolchain: GCC5). > > > * Core CI results: > > - > > Complete run successful (toolchain: GCC5); no package / arch / target > / test restrictions. > > I got the following WARNINGs: > > - ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not > Core CI -- these are expected: > >> PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG -- >> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG >> PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE -- >> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE > >> PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG -- >> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG >> PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE -- >> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE > >> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG -- >> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG >> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE -- >> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE > > - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg, > NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are > not covered by host-based unit tests -- I think also expected: > >> PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > >> PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > > - I'm not really sure about the following warnings. They were emitted > for a subset of the above packages. I read the docs at > > but I still don't understand :) Each plugin/test has a readme as well. Not sure if this helps explain the HostUnitTestDscCompleteCheck more. https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck As to why you don't see it skipped for ArmVirtPkg or a few others: I think this is a bug. In Azure CI i see it as test passed. PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET When i look at the code https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59 I see that how those packages *.ci.yaml have it configured is different than those that show skipped below. It avoids the skip conditions but since those packages don't have any host unit test libraries or modules there is no failure. This should probably be a bugzilla on the HostUnitTestDscCompleteCheck.py as the reporting is not correct. Test should show up as skipped. You can see them here showing up as success https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=11309&view=logs&j=780eaa17-fc2d-5131-b9d2-21b19bc9bec5&t=0791c90f-e7d8-5372-c9c5-edbe30bf6aeb&l=32 > >> PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET > >> PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET > >> PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET > >> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET > >> PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET > >> PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET > > - Still related to host-based unit tests, I believe the following > warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and > UnitTestFrameworkPkg, report that running the tests on the build > host (understandably) prevents the tests from covering all edk2 > architectures: Yes this message is because we override the arch for NOOPT to match that of the build system since we are going to run these executables in the host environment. > >> PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - Allowing Override for key TARGET_ARCH > >> PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - Allowing Override for key TARGET_ARCH > >> PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - Allowing Override for key TARGET_ARCH > >> PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - Allowing Override for key TARGET_ARCH > >> PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler Plugin NOOPT -- >> WARNING - Allowing Override for key TARGET_ARCH > > - The below seems to be related to the "spell checking in audit > mode" known issues at > . > > What does "audit mode" mean? see readme here. https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration Basically runs the tests and reports all the errors as warnings and then reports the test as skipped. It was hopefully to encourage package maintainers to fix their spelling issues but not breaking the build on day 1. > > FWIW, the packages listed below (EmulatorPkg, MdeModulePkg, > MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper > subset of the packges noted in ".pytool/Readme.md" -- the latter > mentions CryptoPkg and SecurityPkg in addition. > >> PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > >> PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > >> PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > >> PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > >> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > >> PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > >> PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > > - Not sure about the last one: > >> PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET -- >> WARNING - --->Test Skipped: in plugin! Library Class Check Test NO-TARGET > > > * One suggestion (in addition to the others made in this thread): > > - Please add "BaseToolsBuild" to "BaseTools/.gitignore". There is a bugzilla for this already. https://bugzilla.tianocore.org/show_bug.cgi?id=2581 > > > * A question: > > - On github, there is some logic that restricts the full Core CI build > to packages that may be affected by a patch series. Is this available > locally? > > According to the Core CI documentation, the "stuart_ci_build" command > can be limited with "-p" / "-a" / "-t" flags; given a commit range, > how do we calculate the "tightest" flag values? A couple things here. You can always look at the azurepipelines files to get a better understanding of the CI process. Hopefully there is no magic (besides being able to read yaml). You can see here: https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35 It is actually another stuart tool. stuart_pr_eval. We have bug to document it. https://github.com/tianocore/edk2-pytool-extensions/issues/89 But the python file is pretty well documented. There are 4 policies that can trigger the need for a change to build a package. See here https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117 > > Thank you very much! > Laszlo >