From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ma1-aaemail-dr-lapp01.apple.com (ma1-aaemail-dr-lapp01.apple.com [17.171.2.60]) by mx.groups.io with SMTP id smtpd.web12.1374.1591987399395088088 for ; Fri, 12 Jun 2020 11:43:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=jV4Vu2nD; spf=pass (domain: apple.com, ip: 17.171.2.60, mailfrom: afish@apple.com) Received: from pps.filterd (ma1-aaemail-dr-lapp01.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp01.apple.com (8.16.0.42/8.16.0.42) with SMTP id 05CIelqp038027; Fri, 12 Jun 2020 11:43:15 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=4LkU2UVbWWowHp5ymKKGKSSV1p9reP++sSGQn5IDZzs=; b=jV4Vu2nDe6/NpogTRsheVO4rF0k1CdCv5ZGOnNyrK5CVWMsbSlQwGRr3L2SvSe2BIvM7 LgxCeJowTB7hX5TY9HqcswUGr3m6teQ7yo/2vsSm3o1Ru44BVZ1UJm+LBuxFkS76ojkC kvrg+NsGYpEU9e4RhiymM7LUwVuuVvAuXqCBe7+Ae5r8lWSBRJPs9jV9OYeEFkSnrxiR EKEll9hyyKHMFghvfeBK4fRPPjOIPutYLR+I9UjEP3+Zhqaw4yRICnrPfEInmfaEFuAl JlKumkYnAGxi4BgKI5Sw6v2k/YKCyFdPNC2YBLY52ohtCCqUajKPdOTDNS6zdqZCwY7o sg== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by ma1-aaemail-dr-lapp01.apple.com with ESMTP id 31g9t5dcvq-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 12 Jun 2020 11:43:15 -0700 Received: from rn-mailsvcp-mmp-lapp03.rno.apple.com (rn-mailsvcp-mmp-lapp03.rno.apple.com [17.179.253.16]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) with ESMTPS id <0QBT00Y5NTC1BYB0@rn-mailsvcp-mta-lapp01.rno.apple.com>; Fri, 12 Jun 2020 11:43:15 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp03.rno.apple.com by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) id <0QBT00E00STDYY00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Fri, 12 Jun 2020 11:43:14 -0700 (PDT) X-Va-A: X-Va-T-CD: 678bf7de5df0d9ff994f556fd1b44182 X-Va-E-CD: 68d21cd46f473ac90dd6cdf0d9bb8156 X-Va-R-CD: 34acb5f341b51477fe719d0983a65db1 X-Va-CD: 0 X-Va-ID: 00588d46-0d22-4082-8c1d-67ba20616128 X-V-A: X-V-T-CD: 678bf7de5df0d9ff994f556fd1b44182 X-V-E-CD: 68d21cd46f473ac90dd6cdf0d9bb8156 X-V-R-CD: 34acb5f341b51477fe719d0983a65db1 X-V-CD: 0 X-V-ID: 1f68e245-2aad-41ad-bdd9-e222b9f3468b X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216,18.0.687 definitions=2020-06-12_16:2020-06-12,2020-06-12 signatures=0 Received: from [17.235.13.21] (unknown [17.235.13.21]) by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) with ESMTPSA id <0QBT00LFFTBZAR00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Fri, 12 Jun 2020 11:43:13 -0700 (PDT) From: "Andrew Fish" Message-id: <8DE5146D-9BA1-446C-AC98-0EBEEE1EBAB4@apple.com> MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: [edk2-devel] ci problems again Date: Fri, 12 Jun 2020 11:43:11 -0700 In-reply-to: Cc: Ard Biesheuvel , Sean Brogan , "lersek@redhat.com" , Bret Barkelew To: devel@edk2.groups.io, Mike Kinney References: <08ef07fe-67cd-7fad-a43c-2483136772af@arm.com> <9498c58c-4a77-5ecf-43ff-f218e4987bfc@arm.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216,18.0.687 definitions=2020-06-12_16:2020-06-12,2020-06-12 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_F15AC1BE-B25E-4A63-A406-D26CAA5D1133" --Apple-Mail=_F15AC1BE-B25E-4A63-A406-D26CAA5D1133 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Mike, Is this an issue of assuming a patch vs. a patch set? Are there other plac= es we could run into this? Thanks, Andrew Fish > On Jun 12, 2020, at 11:19 AM, Michael D Kinney wrote: >=20 > Hi Ard, >=20 > I think this is related to a patch series that deletes > some files. >=20 > The following code in the edk2-pytool-extensions is collecting > the files that change in a pr to determine the components that > need to be built. >=20 > https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d= 6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 >=20 > It does not filter out files that have been deleted. >=20 > I tried the following change that filters out files that > have been deleted and reverses the diff to be from the > base branch to HEAD so the files deleted filter works as > expected. The stuart command completes without errors. >=20 > diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invoca= bles/edk2_pr_eval.py > index 00ac282..a3f55b7 100644 > --- a/edk2toolext/invocables/edk2_pr_eval.py > +++ b/edk2toolext/invocables/edk2_pr_eval.py > @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_br= anch) -> tuple: >=20 > # get file differences between pr and base > output =3D StringIO() > - cmd_params =3D f"diff --name-only HEAD..{base_branch}" > + cmd_params =3D f"diff --name-only --diff-filter=3Dd {base_branc= h}..HEAD" > rc =3D RunCmd("git", cmd_params, outstream=3Doutput) >=20 > if(rc =3D=3D 0): >=20 > I do not know if this change has other unintended side effects. I > need edk2-pytools-extensions to provide input. >=20 > Mike >=20 >> -----Original Message----- >> From: Kinney, Michael D > >> Sent: Friday, June 12, 2020 9:49 AM >> To: Ard Biesheuvel >; >> devel@edk2.groups.io ; Sean Brogan >> >; lersek@redhat.com= ; Kinney, >> Michael D > >> Subject: RE: [edk2-devel] ci problems again >>=20 >> Hi Ard, >>=20 >> The general instructions to setup env are in the repo >> here: >>=20 >>=20 >> https://github.com/tianocore/edk2/tree/master/.pytool >>=20 >> I did this and pulled PR 689 and ran the command from >> the >> Azure pipelines log and reproduced the same failure on >> my >> laptop. Here is the log from my laptop. I will debug a >> bit more to see what the issue is. This command passes >> with edk2/master. It fails with my pr_689 branch. >>=20 >>=20 >> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689 >> Switched to branch 'pr_689' >>=20 >> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c >> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 -- >> pr-target origin/master >> SECTION - Init SDE >> SECTION - Loading Plugins >> SECTION - Start Invocable Tool >> ERROR - Failed to get package for file >> ArmVirtPkg/Include/Platform/Hidden.h. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >> \Platform' >> ERROR - Failed to get package for file >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S >> cripts' >> ERROR - Failed to get package for file >> ArmVirtPkg/Include/Platform/Hidden.h. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >> \Platform' >> ERROR - Failed to get package for file >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S >> cripts' >> Traceback (most recent call last): >> File "C:\Program Files (x86)\Microsoft Visual >> Studio\Shared\Python37_64\lib\runpy.py", line 193, in >> _run_module_as_main >> "__main__", mod_spec) >> File "C:\Program Files (x86)\Microsoft Visual >> Studio\Shared\Python37_64\lib\runpy.py", line 85, in >> _run_code >> exec(code, run_globals) >> File >> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva >> l.exe\__main__.py", line 7, in >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 389, in main >> Edk2PrEval().Invoke() >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\base_abstract_invocable.py", line >> 141, in Invoke >> retcode =3D self.Go() >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 82, in Go >> actualPackagesDict =3D >> self.get_packages_to_build(self.requested_package_list) >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 202, in get_packages_to_build >> changed_modules =3D >> self._get_unique_module_infs_changed(files) >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 235, in _get_unique_module_infs_changed >> infs =3D >> self.edk2_path_obj.GetContainingModules(os.path.abspath( >> f)) >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toollib\uefi\edk2\path_utilities.py", line >> 211, in GetContainingModules >> for f in os.listdir(dirpath): >> FileNotFoundError: [WinError 3] The system cannot find >> the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >> \Platform' >>=20 >> Best regards, >>=20 >> Mike >>=20 >>> -----Original Message----- >>> From: Ard Biesheuvel >>> Sent: Friday, June 12, 2020 9:16 AM >>> To: Kinney, Michael D ; >>> devel@edk2.groups.io; Sean Brogan >>> ; lersek@redhat.com >>> Subject: Re: [edk2-devel] ci problems again >>>=20 >>> On 6/12/20 6:12 PM, Kinney, Michael D wrote: >>>> Hi Ard, >>>>=20 >>>> The CI agents are stateless, so it can not be stale >>>> content. >>>>=20 >>>=20 >>> So why is it looking for >> ArmVirtPkg/PrePi/Scripts/PrePi- >>> PIE.lds even >>> though not a single reference to it remains anywhere >> in >>> the tree? >>>=20 >>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0 >>> hits) >>>=20 >>>=20 >>>> Have you tried the pytools command that is being >> used >>>> locally for this patch series? >>>>=20 >>>=20 >>> No. Are there any instructions on how to use that? >>>=20 >>>=20 >>>=20 >>>>=20 >>>>> -----Original Message----- >>>>> From: devel@edk2.groups.io >> On >>>>> Behalf Of Ard Biesheuvel >>>>> Sent: Friday, June 12, 2020 8:47 AM >>>>> To: Kinney, Michael D ; >>>>> devel@edk2.groups.io; Sean Brogan >>>>> ; lersek@redhat.com >>>>> Subject: Re: [edk2-devel] ci problems again >>>>>=20 >>>>> On 6/12/20 5:36 PM, Kinney, Michael D wrote: >>>>>> Hi Ard, >>>>>>=20 >>>>>> Please look at this more detailed log from Azure. >>>>>>=20 >>>>>> https://dev.azure.com/tianocore/edk2- >>>>>=20 >>>=20 >> ci/_build/results?buildId=3D8170&view=3Dlogs&jobId=3D680b79ca- >>>>> 2ce9-56d4-127b-ee6e95685750&j=3D680b79ca-2ce9-56d4- >>> 127b- >>>>> ee6e95685750&t=3D1e12dbf9-b3ef-52bb-1019-4c2f443aef72 >>>>>>=20 >>>>>> It says it can not find some paths in ArmVirtPkg. >>>>>>=20 >>>>>=20 >>>>> It is complaining about files that are being >> removed >>> by >>>>> the series: >>>>>=20 >>>>> Failed to get package for file >>>>> ArmVirtPkg/Include/Platform/Hidden.h. >>>>> Failed to get package for file >>>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. >>>>>=20 >>>>> but I cannot find any remaining references to those >>>>> files in anywhere in >>>>> the tree, so it seems the CI has remembered the >>>>> existence of those files >>>>> in some way, and is now expecting them to be still >>>>> there. >>>>>=20 >>>>> So I don't think there is any way I can fix this >>> myself. >>>>> Someone needs >>>>> to wipe the build directory for these CI jobs. >>>>>=20 >>>>>=20 >>>>=20 >=20 >=20 >=20 --Apple-Mail=_F15AC1BE-B25E-4A63-A406-D26CAA5D1133 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Mike,
<= br class=3D"">
Is this an issue of assuming a patch vs= . a patch set? Are there other places we could run into this?

Thanks,
<= br class=3D"">
Andrew Fish

On Jun 12, 2020= , at 11:19 AM, Michael D Kinney <michael.d.kinney@intel.com> wrote:

Hi Ard,

I think this is related to a patch series that deletes
some files.

The foll= owing code in the edk2-pytool-extensions is collecting
the files that change in a pr to determine = the components that
nee= d to be built.

h= ttps://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1a= a4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271

It does not filter out files that have been deleted.

I tried the following change that filters out files that
have been deleted and reverses the di= ff to be from the
bas= e branch to HEAD so the files deleted filter works as
expected.  The stuart command completes= without errors.

diff --git a/edk2toolext/invocables/edk2_p= r_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
index 00ac282..a3f55b7 100644
--- a/edk2toolext/invocables/edk2_pr_= eval.py
+++ b/edk2toole= xt/invocables/edk2_pr_eval.py
= @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, bas= e_branch) -> tuple:

      =   # get file differences between pr and base
       &= nbsp;output =3D StringIO()
-        cmd_params =3D f"diff --= name-only HEAD..{base_branch}"
+        cmd_params =3D f"diff -= -name-only --diff-filter=3Dd {base_branch}..HEAD"
        = rc =3D RunCmd("git", cmd_params, outstream=3Doutput)

&nb= sp;       if(rc =3D=3D 0):

I do not know if this change has other unintended side effects.  I<= /span>
need edk2-pytools-exten= sions to provide input.

Mike

-= ----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.c= om>
Sent: Friday, June 12, 2020 9:49 AM
= To: Ard Biesheuvel <ard.biesheuvel@arm.com>;
devel@edk2.groups.io; Sean Brogan
= <spbrogan@outlook.com= >; lersek@redhat.com; Kinney,
Michael D <m= ichael.d.kinney@intel.com>
Subject: RE: [edk2-devel] c= i problems again

Hi Ard,

The general instructions to setup env are in the repo
here:


https://github.com/tia= nocore/edk2/tree/master/.pytool

I did this= and pulled PR 689 and ran the command from
the
Azure pipelines log and reproduced the same failure on
mylaptop.  Here is the log from my laptop.  I will debu= g a
bit more to see what the issue is.  This command pas= ses
with edk2/master.  It fails with my pr_689 branch.

(env) c:\work\GitHub\tianocore\e= dk2>git checkout pr_689
Switched to branch 'pr_689'

(env) c:\work\GitHub\tianocore\edk2>stuart_pr_eva= l -c
OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --<= br class=3D"">pr-target origin/master
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable= Tool
ERROR - Failed to get package for file
Ar= mVirtPkg/Include/Platform/Hidden.h.  Exception
[WinError= 3] The system cannot find the path specified:
'c:\\work\\Git= Hub\\tianocore\\edk2\\ArmVirtPkg\\Include\
\Platform'
ERROR - Failed to get package for file
ArmVirtPkg/PreP= i/Scripts/PrePi-PIE.lds.  Exception
[WinError 3] The sys= tem cannot find the path specified:
'c:\\work\\GitHub\\tianoc= ore\\edk2\\ArmVirtPkg\\PrePi\\S
cripts'
ERROR -= Failed to get package for file
ArmVirtPkg/Include/Platform/H= idden.h.  Exception
[WinError 3] The system cannot find = the path specified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVi= rtPkg\\Include\
\Platform'
ERROR - Failed to ge= t package for file
ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. &n= bsp;Exception
[WinError 3] The system cannot find the path sp= ecified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PreP= i\\S
cripts'
Traceback (most recent call last):=
 File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python37_64\lib\runpy.py", line 193, in
= _run_module_as_main
   "__main__", mod_spec) File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python37_64\lib\runpy.py", line 85, in
_= run_code
   exec(code, run_globals)
 File
"c:\work\GitHub\tianocore\edk2\env\Scripts\= stuart_pr_eva
l.exe\__main__.py", line 7, in <module> File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\invocables\edk2_pr_eval.py", line
389, in main
   Edk2PrEval().Invoke()
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\base_abstract_invocable.py", line
141= , in Invoke
   retcode =3D self.Go()
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\invocables\edk2_pr_eval.py", line
82, = in Go
   actualPackagesDict =3D
= self.get_packages_to_build(self.requested_package_list)
 = ;File "c:\work\github\tianocore\edk2\env\lib\site-
packages\e= dk2toolext\invocables\edk2_pr_eval.py", line
202, in get_pack= ages_to_build
   changed_modules =3D
self._get_unique_module_infs_changed(files)
 File= "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2to= olext\invocables\edk2_pr_eval.py", line
235, in _get_unique_m= odule_infs_changed
   infs =3D
s= elf.edk2_path_obj.GetContainingModules(os.path.abspath(
f)) File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toollib\uefi\edk2\path_utilities.py", line
211, in GetContainingModules
   for f in os= .listdir(dirpath):
FileNotFoundError: [WinError 3] The system= cannot find
the path specified:
'c:\\work\\Git= Hub\\tianocore\\edk2\\ArmVirtPkg\\Include\
\Platform'

Best regards,

Mike

-----Original = Message-----
From: Ard Biesheuvel <ard.biesheuvel@arm.com&= gt;
Sent: Friday, June 12, 2020 9:16 AM
To: Kin= ney, Michael D <michael.d.kinney@intel.com>;
devel@edk2= .groups.io; Sean Brogan
<spbrogan@outlook.com>; lersek@= redhat.com
Subject: Re: [edk2-devel] ci problems again

On 6/12/20 6:12 PM, Kinney, Michael D wrote:
Hi Ard,

The CI agents are stateless, so it can not be stale
co= ntent.


So why is i= t looking for
ArmVirtPkg/PrePi/Scripts/PrePi-
PIE.lds even
though not a single reference to it remains anywhere
in
the tree?

(grep'ing for PrePi-PIE.lds in the edk2 tree gives = 0
hits)


Have you tried the pytools command that is bein= g
used
locally for thi= s patch series?


No= . Are there any instructions on how to use that?




-----Original Message-----From: devel@edk2.groups.io <devel@edk2.groups.io>
On
Behalf Of Ard Biesheuvel
Sent: Frida= y, June 12, 2020 8:47 AM
To: Kinney, Michael D <michael.d.= kinney@intel.com>;
devel@edk2.groups.io; Sean Brogan
<spbrogan@outlook.com>; lersek@redhat.com
Subj= ect: Re: [edk2-devel] ci problems again

On 6/1= 2/20 5:36 PM, Kinney, Michael D wrote:
Hi Ard,

Please look at this mor= e detailed log from Azure.

https://dev.azure.c= om/tianocore/edk2-

<= /blockquote>
ci/_build/results?buildId=3D8170&am= p;view=3Dlogs&jobId=3D680b79ca-
2ce9-56d4-127b-ee6e95685750&j=3D680b79ca-2ce9-56d4-
127b-
ee6e95685750&t=3D1e= 12dbf9-b3ef-52bb-1019-4c2f443aef72

It says it can not find some paths in ArmVirtPkg.=


It is complaining= about files that are being
removed
by
the series:

Failed to get package for f= ile
ArmVirtPkg/Include/Platform/Hidden.h.
Faile= d to get package for file
ArmVirtPkg/PrePi/Scripts/PrePi-PIE.= lds.

but I cannot find any remaining reference= s to those
files in anywhere in
the tree, so it= seems the CI has remembered the
existence of those files
in some way, and is now expecting them to be still
there.

So I don't think there is any way I ca= n fix this
myself.
Som= eone needs
to wipe the build directory for these CI jobs.




=
--Apple-Mail=_F15AC1BE-B25E-4A63-A406-D26CAA5D1133--