Re: [edk2-devel] ci problems again

["Michael D Kinney" <michael.d.kinney@intel.com>]

Hi Ard,

I think this is related to a patch series that deletes
some files.

The following code in the edk2-pytool-extensions is collecting
the files that change in a pr to determine the components that
need to be built.

https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271

It does not filter out files that have been deleted.

I tried the following change that filters out files that
have been deleted and reverses the diff to be from the
base branch to HEAD so the files deleted filter works as
expected.  The stuart command completes without errors.

diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
index 00ac282..a3f55b7 100644
--- a/edk2toolext/invocables/edk2_pr_eval.py
+++ b/edk2toolext/invocables/edk2_pr_eval.py
@@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:

         # get file differences between pr and base
         output = StringIO()
-        cmd_params = f"diff --name-only HEAD..{base_branch}"
+        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
         rc = RunCmd("git", cmd_params, outstream=output)

         if(rc == 0):

I do not know if this change has other unintended side effects.  I
need edk2-pytools-extensions to provide input.

Mike

> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Friday, June 12, 2020 9:49 AM
> To: Ard Biesheuvel <ard.biesheuvel@arm.com>;
> devel@edk2.groups.io; Sean Brogan
> <spbrogan@outlook.com>; lersek@redhat.com; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Subject: RE: [edk2-devel] ci problems again
> 
> Hi Ard,
> 
> The general instructions to setup env are in the repo
> here:
> 
> 
> https://github.com/tianocore/edk2/tree/master/.pytool
> 
> I did this and pulled PR 689 and ran the command from
> the
> Azure pipelines log and reproduced the same failure on
> my
> laptop.  Here is the log from my laptop.  I will debug a
> bit more to see what the issue is.  This command passes
> with edk2/master.  It fails with my pr_689 branch.
> 
> 
> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
> Switched to branch 'pr_689'
> 
> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
> pr-target origin/master
> SECTION - Init SDE
> SECTION - Loading Plugins
> SECTION - Start Invocable Tool
> ERROR - Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> ERROR - Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
> cripts'
> ERROR - Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> ERROR - Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
> cripts'
> Traceback (most recent call last):
>   File "C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
> _run_module_as_main
>     "__main__", mod_spec)
>   File "C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
> _run_code
>     exec(code, run_globals)
>   File
> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
> l.exe\__main__.py", line 7, in <module>
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 389, in main
>     Edk2PrEval().Invoke()
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\base_abstract_invocable.py", line
> 141, in Invoke
>     retcode = self.Go()
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 82, in Go
>     actualPackagesDict =
> self.get_packages_to_build(self.requested_package_list)
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 202, in get_packages_to_build
>     changed_modules =
> self._get_unique_module_infs_changed(files)
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 235, in _get_unique_module_infs_changed
>     infs =
> self.edk2_path_obj.GetContainingModules(os.path.abspath(
> f))
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toollib\uefi\edk2\path_utilities.py", line
> 211, in GetContainingModules
>     for f in os.listdir(dirpath):
> FileNotFoundError: [WinError 3] The system cannot find
> the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> 
> Best regards,
> 
> Mike
> 
> > -----Original Message-----
> > From: Ard Biesheuvel <ard.biesheuvel@arm.com>
> > Sent: Friday, June 12, 2020 9:16 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > devel@edk2.groups.io; Sean Brogan
> > <spbrogan@outlook.com>; lersek@redhat.com
> > Subject: Re: [edk2-devel] ci problems again
> >
> > On 6/12/20 6:12 PM, Kinney, Michael D wrote:
> > > Hi Ard,
> > >
> > > The CI agents are stateless, so it can not be stale
> > > content.
> > >
> >
> > So why is it looking for
> ArmVirtPkg/PrePi/Scripts/PrePi-
> > PIE.lds even
> > though not a single reference to it remains anywhere
> in
> > the tree?
> >
> > (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
> > hits)
> >
> >
> > > Have you tried the pytools command that is being
> used
> > > locally for this patch series?
> > >
> >
> > No. Are there any instructions on how to use that?
> >
> >
> >
> > >
> > >> -----Original Message-----
> > >> From: devel@edk2.groups.io <devel@edk2.groups.io>
> On
> > >> Behalf Of Ard Biesheuvel
> > >> Sent: Friday, June 12, 2020 8:47 AM
> > >> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > >> devel@edk2.groups.io; Sean Brogan
> > >> <spbrogan@outlook.com>; lersek@redhat.com
> > >> Subject: Re: [edk2-devel] ci problems again
> > >>
> > >> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
> > >>> Hi Ard,
> > >>>
> > >>> Please look at this more detailed log from Azure.
> > >>>
> > >>> https://dev.azure.com/tianocore/edk2-
> > >>
> >
> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
> > >> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
> > 127b-
> > >> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
> > >>>
> > >>> It says it can not find some paths in ArmVirtPkg.
> > >>>
> > >>
> > >> It is complaining about files that are being
> removed
> > by
> > >> the series:
> > >>
> > >> Failed to get package for file
> > >> ArmVirtPkg/Include/Platform/Hidden.h.
> > >> Failed to get package for file
> > >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
> > >>
> > >> but I cannot find any remaining references to those
> > >> files in anywhere in
> > >> the tree, so it seems the CI has remembered the
> > >> existence of those files
> > >> in some way, and is now expecting them to be still
> > >> there.
> > >>
> > >> So I don't think there is any way I can fix this
> > myself.
> > >> Someone needs
> > >> to wipe the build directory for these CI jobs.
> > >>
> > >> 
> > >