From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (NAM10-BN7-obe.outbound.protection.outlook.com [40.92.40.20]) by mx.groups.io with SMTP id smtpd.web11.1563.1591998361849574058 for ; Fri, 12 Jun 2020 14:46:02 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@outlook.com header.s=selector1 header.b=R4hJuqjn; spf=pass (domain: outlook.com, ip: 40.92.40.20, mailfrom: spbrogan@outlook.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZAPXPjpSp8PhQT0RZY8mG0Jb+KMkGdrHbw5+Y6GlXAa3V7KCxPiC1qbChciAhHg7yQpCgI2jq3Pyqd8m095kyX69lYSKwz4B5xiDFU82QcREtbNrJ0IaZ45Ktcvjj5J0qjXNZYRRBDOBoe4MDFvK+vzYn0bWeZaSYoRWwa9sR+kHWZfqhuKrAW7b8ST6fteNoPeEiFt0qQO7cTmcUpXwLzQ3YtAl5hkS/nt+ZL7s+0ix0qXHac+eOf4mviMfvrUZYNHlfly4uvPIE9CWyRiU1oNXYPB0QB0Ht/ic2w2gqfJdmx1dramNTTWaXdGnJNztinH3qU9Tdxbfd5zjn/70fA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V8brZqdOCV+QrAJ+o33fzRoMHpKYq0/Vv9UyB0c1/9s=; b=WzsYsIJZeD3IrCtILyEpqqhd7ZsPsGirBlM+LX7MlYcSUFyE+KStQlx9mzRCWfq/pdNfMDji4aEVaV6BB4Sfv4JQcTiFyFkUEljazAq/7vwlDJn/MsZMm4jBw7PyUwvieOnwXy7n2doirxS6pgZBIwDDCs3wme6w3eeFrfHGFG0+mtl895mWT05K2YAn5+xAoJrGqTS5Yrz6mSKcifVWbEIEbITVL3iwGJ3exhFmdMN3HpgNxXm6E0NL3udWMAJfBXlHCxN64TyENuvHBDKtGSSsGHs7DsjGS/FzZ3BVNsdx9BtCdE5xSJvvWT35Ks25qctC83zBfES7w3LsaXFNyw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V8brZqdOCV+QrAJ+o33fzRoMHpKYq0/Vv9UyB0c1/9s=; b=R4hJuqjnDitKUUM+ciRl+ujauScyKcdjOdhITdfJWRWkvhcyd0LQFTvxZ85ybrEmZp7mGM9crImrSYKPQK3d2UPDdTLV9N4cdm/Q2Vmv3eRP4L7e2TEVZ/pPZpPynKkCZwDOtMKG5a/L++j3wEt8owGcYgjL4RXbQ+rcZB1DFq5ca2yhSRZCTdQ1WlXSml2YeWKNSSr+rWoh06o3e6mnSWf2xzlqAOfvCe8H32VEOer7KygJGxGm4yMz0fGylkFxbhLLNzHTZosTlf/3wqPPMrZEnITjJBJaXOsHx4cTGUIG0FLBYbDU98JiO98WwX8ZbSXwS+TAbAK8HhmLqvuFOA== Received: from BN7NAM10FT064.eop-nam10.prod.protection.outlook.com (2a01:111:e400:7e8f::49) by BN7NAM10HT069.eop-nam10.prod.protection.outlook.com (2a01:111:e400:7e8f::145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.18; Fri, 12 Jun 2020 21:46:00 +0000 Received: from BN8PR07MB6962.namprd07.prod.outlook.com (2a01:111:e400:7e8f::50) by BN7NAM10FT064.mail.protection.outlook.com (2a01:111:e400:7e8f::340) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.18 via Frontend Transport; Fri, 12 Jun 2020 21:46:00 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:9E420A1B45927B1D477D4C745A5005B0C917AF952EE81E40681792560064EB9C;UpperCasedChecksum:C34A9885A005642DCA9663DFC7F256A043A5EC8DA6D07F4AF8E820793CBAE78E;SizeAsReceived:8711;Count:50 Received: from BN8PR07MB6962.namprd07.prod.outlook.com ([fe80::edbc:7122:7dd9:1403]) by BN8PR07MB6962.namprd07.prod.outlook.com ([fe80::edbc:7122:7dd9:1403%8]) with mapi id 15.20.3088.025; Fri, 12 Jun 2020 21:46:00 +0000 Subject: Re: [edk2-devel] ci problems again From: "Sean" To: devel@edk2.groups.io, afish@apple.com, Mike Kinney CC: Ard Biesheuvel , "lersek@redhat.com" , Bret Barkelew Reply-To: devel@edk2.groups.io, spbrogan@outlook.com References: <08ef07fe-67cd-7fad-a43c-2483136772af@arm.com> <9498c58c-4a77-5ecf-43ff-f218e4987bfc@arm.com> <8DE5146D-9BA1-446C-AC98-0EBEEE1EBAB4@apple.com> <19F172A4-4FE8-428D-9503-4C1D9E890749@apple.com> <1617E440CF758AC6.1224@groups.io> Message-ID: Date: Fri, 12 Jun 2020 14:45:58 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 In-Reply-To: <1617E440CF758AC6.1224@groups.io> X-ClientProxiedBy: MWHPR10CA0063.namprd10.prod.outlook.com (2603:10b6:300:2c::25) To BN8PR07MB6962.namprd07.prod.outlook.com (2603:10b6:408:d6::11) Return-Path: spbrogan@outlook.com X-Microsoft-Original-Message-ID: <1c8061f0-0582-416c-552c-a54e97840a24@outlook.com> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.78] (50.47.113.221) by MWHPR10CA0063.namprd10.prod.outlook.com (2603:10b6:300:2c::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.21 via Frontend Transport; Fri, 12 Jun 2020 21:45:59 +0000 X-Microsoft-Original-Message-ID: <1c8061f0-0582-416c-552c-a54e97840a24@outlook.com> X-TMN: [qsPsoWsxskE3d8OjtzhSR/8iBZttRVo7] X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 50 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: 809e4aa2-ba0a-4394-053d-08d80f19ff3b X-MS-TrafficTypeDiagnostic: BN7NAM10HT069: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: B2cGVpygW9H7mCgLo28+ZAoQz2IH+YdxZddbTWK3MerD6yV80wQyu0erAv7n3brLkNCSnv9H2qeDG78+LX06jp/p5PlgiZQnEH7WTBpRfSyBxXS6rwQrnldoInXzzRuvTZxApAG/Z/uDi013vr7I7hUIOp4h6AX1VSvk6TIVxJceHfppjVg9MUhghEDWZTZEJ+TMkQwAE0Ayrd4eavaGIqsfj4GV9MLFfg65TyxlXzehIoRYpx1H1qJZp4fk4Zxw X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:0;SRV:;IPV:NLI;SFV:NSPM;H:BN8PR07MB6962.namprd07.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:;DIR:OUT;SFP:1901; X-MS-Exchange-AntiSpam-MessageData: PPW6DvSmlepdczLpfzLK5f3t3xdZtdaZJ+z5u1TulIc3wHE7XOlOnd3RYGx6dXqJpl5RvxrnbEeXZHOhK20lERbdfRLAldm0ztYTPukmsw5FIzHJHa0km53o7QsVcYw+SmmpDneNHa+ccpWDxxGOKQ== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 809e4aa2-ba0a-4394-053d-08d80f19ff3b X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jun 2020 21:46:00.5473 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7NAM10HT069 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable Bug fixed. Release 0.13.7 created and published. @ard - You should be able to get past your current issue on PR 689. Thanks Sean On 6/12/2020 12:59 PM, Sean wrote: > A pytool bug is opened that provides details. > https://github.com/tianocore/edk2-pytool-extensions/issues/201 >=20 > A PR for edk2-pytool-extensions and a release will be coming soon. > Since we are using loose version tracking in edk2 for the CI=20 > dependencies it will be automatically picked up and CI builds and push= =20 > builds should be able to continue. >=20 > I'll send a follow up once the release is made. >=20 > Thanks > Sean >=20 >=20 >=20 > On 6/12/2020 12:27 PM, Andrew Fish via groups.io wrote: >> Mike, >> >> Thanks so different encase. >> >> Thanks, >> >> Andrew Fish >> >>> On Jun 12, 2020, at 12:25 PM, Kinney, Michael D=20 >>> wrote: >>> >>> Hi Andrew, >>> >>> Not related to patch vs patch set.=A0 All commits in a pr are being=20 >>> evaluated. >>> >>> After further evaluation, it appears it may be related to a patch set= =20 >>> that deletes the last file from a directory.=A0=A0 When that happens, = the=20 >>> parent directory is also deleted, and the missing dir is what is=20 >>> generating the python exception.=A0 Need to add logic to cover that=20 >>> corner case and avoid exception. >>> >>> The edk2-pytools owners are working on it now. >>> >>> Mike >>> >>> From: Andrew Fish > >>> Sent: Friday, June 12, 2020 11:43 AM >>> To: devel@edk2.groups.io ; Kinney,=20 >>> Michael D >> > >>> Cc: Ard Biesheuvel >> >; Sean Brogan >> >; lersek@redhat.com=20 >>> ; Bret Barkelew=20 >>> > >>> Subject: Re: [edk2-devel] ci problems again >>> >>> Mike, >>> >>> Is this an issue of assuming a patch vs. a patch set? Are there other= =20 >>> places we could run into this? >>> >>> Thanks, >>> >>> Andrew Fish >>> >>> >>> On Jun 12, 2020, at 11:19 AM, Michael D Kinney=20 >>> > wrote= : >>> >>> Hi Ard, >>> >>> I think this is related to a patch series that deletes >>> some files. >>> >>> The following code in the edk2-pytool-extensions is collecting >>> the files that change in a pr to determine the components that >>> need to be built. >>> >>> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c79= 7d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271=20 >>> = =20 >>> >>> >>> It does not filter out files that have been deleted. >>> >>> I tried the following change that filters out files that >>> have been deleted and reverses the diff to be from the >>> base branch to HEAD so the files deleted filter works as >>> expected.=A0 The stuart command completes without errors. >>> >>> diff --git a/edk2toolext/invocables/edk2_pr_eval.py=20 >>> b/edk2toolext/invocables/edk2_pr_eval.py >>> index 00ac282..a3f55b7 100644 >>> --- a/edk2toolext/invocables/edk2_pr_eval.py >>> +++ b/edk2toolext/invocables/edk2_pr_eval.py >>> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self,=20 >>> base_branch) -> tuple: >>> >>> =A0=A0=A0=A0=A0=A0=A0=A0 # get file differences between pr and base >>> =A0=A0=A0=A0=A0=A0=A0=A0 output =3D StringIO() >>> -=A0=A0=A0=A0=A0=A0=A0 cmd_params =3D f"diff --name-only HEAD..{base_b= ranch}" >>> +=A0=A0=A0=A0=A0=A0=A0 cmd_params =3D f"diff --name-only --diff-filter= = =3Dd=20 >>> {base_branch}..HEAD" >>> =A0=A0=A0=A0=A0=A0=A0=A0 rc =3D RunCmd("git", cmd_params, outstream=3D= output) >>> >>> =A0=A0=A0=A0=A0=A0=A0=A0 if(rc =3D=3D 0): >>> >>> I do not know if this change has other unintended side effects.=A0 I >>> need edk2-pytools-extensions to provide input. >>> >>> Mike >>> >>> >>> -----Original Message----- >>> From: Kinney, Michael D >> > >>> Sent: Friday, June 12, 2020 9:49 AM >>> To: Ard Biesheuvel >> >; >>> devel@edk2.groups.io ; Sean Brogan >>> >;=20 >>> lersek@redhat.com ; Kinney, >>> Michael D >> > >>> Subject: RE: [edk2-devel] ci problems again >>> >>> Hi Ard, >>> >>> The general instructions to setup env are in the repo >>> here: >>> >>> >>> https://github.com/tianocore/edk2/tree/master/.pytool=20 >>> >>> >>> I did this and pulled PR 689 and ran the command from >>> the >>> Azure pipelines log and reproduced the same failure on >>> my >>> laptop.=A0 Here is the log from my laptop.=A0 I will debug a >>> bit more to see what the issue is.=A0 This command passes >>> with edk2/master.=A0 It fails with my pr_689 branch. >>> >>> >>> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689 >>> Switched to branch 'pr_689' >>> >>> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c >>> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 -- >>> pr-target origin/master >>> SECTION - Init SDE >>> SECTION - Loading Plugins >>> SECTION - Start Invocable Tool >>> ERROR - Failed to get package for file >>> ArmVirtPkg/Include/Platform/Hidden.h.=A0 Exception >>> [WinError 3] The system cannot find the path specified: >>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >>> \Platform' >>> ERROR - Failed to get package for file >>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.=A0 Exception >>> [WinError 3] The system cannot find the path specified: >>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S >>> cripts' >>> ERROR - Failed to get package for file >>> ArmVirtPkg/Include/Platform/Hidden.h.=A0 Exception >>> [WinError 3] The system cannot find the path specified: >>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >>> \Platform' >>> ERROR - Failed to get package for file >>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.=A0 Exception >>> [WinError 3] The system cannot find the path specified: >>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S >>> cripts' >>> Traceback (most recent call last): >>> =A0 File "C:\Program Files (x86)\Microsoft Visual >>> Studio\Shared\Python37_64\lib\runpy.py", line 193, in >>> _run_module_as_main >>> =A0=A0=A0 "__main__", mod_spec) >>> =A0 File "C:\Program Files (x86)\Microsoft Visual >>> Studio\Shared\Python37_64\lib\runpy.py", line 85, in >>> _run_code >>> =A0=A0=A0 exec(code, run_globals) >>> =A0 File >>> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva >>> l.exe\__main__.py", line 7, in >>> =A0 File "c:\work\github\tianocore\edk2\env\lib\site- >>> packages\edk2toolext\invocables\edk2_pr_eval.py", line >>> 389, in main >>> =A0=A0=A0 Edk2PrEval().Invoke() >>> =A0 File "c:\work\github\tianocore\edk2\env\lib\site- >>> packages\edk2toolext\base_abstract_invocable.py", line >>> 141, in Invoke >>> =A0=A0=A0 retcode =3D self.Go() >>> =A0 File "c:\work\github\tianocore\edk2\env\lib\site- >>> packages\edk2toolext\invocables\edk2_pr_eval.py", line >>> 82, in Go >>> =A0=A0=A0 actualPackagesDict =3D >>> self.get_packages_to_build(self.requested_package_list) >>> =A0 File "c:\work\github\tianocore\edk2\env\lib\site- >>> packages\edk2toolext\invocables\edk2_pr_eval.py", line >>> 202, in get_packages_to_build >>> =A0=A0=A0 changed_modules =3D >>> self._get_unique_module_infs_changed(files) >>> =A0 File "c:\work\github\tianocore\edk2\env\lib\site- >>> packages\edk2toolext\invocables\edk2_pr_eval.py", line >>> 235, in _get_unique_module_infs_changed >>> =A0=A0=A0 infs =3D >>> self.edk2_path_obj.GetContainingModules(os.path.abspath( >>> f)) >>> =A0 File "c:\work\github\tianocore\edk2\env\lib\site- >>> packages\edk2toollib\uefi\edk2\path_utilities.py", line >>> 211, in GetContainingModules >>> =A0=A0=A0 for f in os.listdir(dirpath): >>> FileNotFoundError: [WinError 3] The system cannot find >>> the path specified: >>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >>> \Platform' >>> >>> Best regards, >>> >>> Mike >>> >>> >>> -----Original Message----- >>> From: Ard Biesheuvel >> > >>> Sent: Friday, June 12, 2020 9:16 AM >>> To: Kinney, Michael D >> >; >>> devel@edk2.groups.io ; Sean Brogan >>> >;=20 >>> lersek@redhat.com >>> Subject: Re: [edk2-devel] ci problems again >>> >>> On 6/12/20 6:12 PM, Kinney, Michael D wrote: >>> >>> Hi Ard, >>> >>> The CI agents are stateless, so it can not be stale >>> content. >>> >>> >>> So why is it looking for >>> ArmVirtPkg/PrePi/Scripts/PrePi- >>> >>> PIE.lds even >>> though not a single reference to it remains anywhere >>> in >>> >>> the tree? >>> >>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0 >>> hits) >>> >>> >>> >>> Have you tried the pytools command that is being >>> used >>> >>> locally for this patch series? >>> >>> >>> No. Are there any instructions on how to use that? >>> >>> >>> >>> >>> >>> >>> -----Original Message----- >>> From: devel@edk2.groups.io =20 >>> > >>> On >>> >>> Behalf Of Ard Biesheuvel >>> Sent: Friday, June 12, 2020 8:47 AM >>> To: Kinney, Michael D >> >; >>> devel@edk2.groups.io ; Sean Brogan >>> >;=20 >>> lersek@redhat.com >>> Subject: Re: [edk2-devel] ci problems again >>> >>> On 6/12/20 5:36 PM, Kinney, Michael D wrote: >>> >>> Hi Ard, >>> >>> Please look at this more detailed log from Azure. >>> >>> https://dev.azure.com/tianocore/edk2-=20 >>> >>> >>> >>> ci/_build/results?buildId=3D8170&view=3Dlogs&jobId=3D680b79ca- >>> >>> 2ce9-56d4-127b-ee6e95685750&j=3D680b79ca-2ce9-56d4- >>> 127b- >>> >>> ee6e95685750&t=3D1e12dbf9-b3ef-52bb-1019-4c2f443aef72 >>> >>> >>> It says it can not find some paths in ArmVirtPkg. >>> >>> >>> It is complaining about files that are being >>> removed >>> >>> by >>> >>> the series: >>> >>> Failed to get package for file >>> ArmVirtPkg/Include/Platform/Hidden.h. >>> Failed to get package for file >>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. >>> >>> but I cannot find any remaining references to those >>> files in anywhere in >>> the tree, so it seems the CI has remembered the >>> existence of those files >>> in some way, and is now expecting them to be still >>> there. >>> >>> So I don't think there is any way I can fix this >>> myself. >>> >>> Someone needs >>> to wipe the build directory for these CI jobs. >>> >>> >>> >>> >>> >>> >> >> >> >> >> >=20 >=20 >=20