ci problems again

ci problems again

[Ard Biesheuvel]

Hi all,

I am seeing weird CI failures again:

https://github.com/tianocore/edk2/pull/688
https://github.com/tianocore/edk2/pull/689

with errors like

Unhandled: Not found SourceFolder: 
/home/vsts/work/1/s/BaseTools/BaseToolsBuild

or

/opt/hostedtoolcache/Python/3.8.3/x64/bin/stuart_pr_eval failed with 
return code: 1

and the pull request being rejected.

Re: [edk2-devel] ci problems again

[Michael D Kinney]

Hi Ard,

Please look at this more detailed log from Azure.

https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-127b-ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72

It says it can not find some paths in ArmVirtPkg.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Ard Biesheuvel
> Sent: Friday, June 12, 2020 2:54 AM
> To: devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; Sean Brogan
> <spbrogan@outlook.com>; lersek@redhat.com
> Subject: [edk2-devel] ci problems again
> 
> Hi all,
> 
> I am seeing weird CI failures again:
> 
> https://github.com/tianocore/edk2/pull/688
> https://github.com/tianocore/edk2/pull/689
> 
> with errors like
> 
> Unhandled: Not found SourceFolder:
> /home/vsts/work/1/s/BaseTools/BaseToolsBuild
> 
> or
> 
> /opt/hostedtoolcache/Python/3.8.3/x64/bin/stuart_pr_eval
> failed with
> return code: 1
> 
> and the pull request being rejected.
> 
> 
> 

Re: [edk2-devel] ci problems again

[Ard Biesheuvel]

On 6/12/20 5:36 PM, Kinney, Michael D wrote:
> Hi Ard,
> 
> Please look at this more detailed log from Azure.
> 
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-127b-ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
> 
> It says it can not find some paths in ArmVirtPkg.
> 

It is complaining about files that are being removed by the series:

Failed to get package for file ArmVirtPkg/Include/Platform/Hidden.h.
Failed to get package for file ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.

but I cannot find any remaining references to those files in anywhere in 
the tree, so it seems the CI has remembered the existence of those files 
in some way, and is now expecting them to be still there.

So I don't think there is any way I can fix this myself. Someone needs 
to wipe the build directory for these CI jobs.

Re: [edk2-devel] ci problems again

[Michael D Kinney]

Hi Ard,

The CI agents are stateless, so it can not be stale
content.

Have you tried the pytools command that is being used
locally for this patch series?

Thanks,

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Ard Biesheuvel
> Sent: Friday, June 12, 2020 8:47 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io; Sean Brogan
> <spbrogan@outlook.com>; lersek@redhat.com
> Subject: Re: [edk2-devel] ci problems again
> 
> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
> > Hi Ard,
> >
> > Please look at this more detailed log from Azure.
> >
> > https://dev.azure.com/tianocore/edk2-
> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-127b-
> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
> >
> > It says it can not find some paths in ArmVirtPkg.
> >
> 
> It is complaining about files that are being removed by
> the series:
> 
> Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.
> Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
> 
> but I cannot find any remaining references to those
> files in anywhere in
> the tree, so it seems the CI has remembered the
> existence of those files
> in some way, and is now expecting them to be still
> there.
> 
> So I don't think there is any way I can fix this myself.
> Someone needs
> to wipe the build directory for these CI jobs.
> 
> 

Re: [edk2-devel] ci problems again

[Ard Biesheuvel]

On 6/12/20 6:12 PM, Kinney, Michael D wrote:
> Hi Ard,
> 
> The CI agents are stateless, so it can not be stale
> content.
> 

So why is it looking for ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds even 
though not a single reference to it remains anywhere in the tree?

(grep'ing for PrePi-PIE.lds in the edk2 tree gives 0 hits)


> Have you tried the pytools command that is being used
> locally for this patch series?
> 

No. Are there any instructions on how to use that?



> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On
>> Behalf Of Ard Biesheuvel
>> Sent: Friday, June 12, 2020 8:47 AM
>> To: Kinney, Michael D <michael.d.kinney@intel.com>;
>> devel@edk2.groups.io; Sean Brogan
>> <spbrogan@outlook.com>; lersek@redhat.com
>> Subject: Re: [edk2-devel] ci problems again
>>
>> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
>>> Hi Ard,
>>>
>>> Please look at this more detailed log from Azure.
>>>
>>> https://dev.azure.com/tianocore/edk2-
>> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-127b-
>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
>>>
>>> It says it can not find some paths in ArmVirtPkg.
>>>
>>
>> It is complaining about files that are being removed by
>> the series:
>>
>> Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.
>> Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
>>
>> but I cannot find any remaining references to those
>> files in anywhere in
>> the tree, so it seems the CI has remembered the
>> existence of those files
>> in some way, and is now expecting them to be still
>> there.
>>
>> So I don't think there is any way I can fix this myself.
>> Someone needs
>> to wipe the build directory for these CI jobs.
>>
>> 
> 

Re: [edk2-devel] ci problems again

[Michael D Kinney]

Hi Ard,

The general instructions to setup env are in the repo here:

    https://github.com/tianocore/edk2/tree/master/.pytool

I did this and pulled PR 689 and ran the command from the 
Azure pipelines log and reproduced the same failure on my
laptop.  Here is the log from my laptop.  I will debug a 
bit more to see what the issue is.  This command passes
with edk2/master.  It fails with my pr_689 branch.


(env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
Switched to branch 'pr_689'

(env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --pr-target origin/master
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
ERROR - Failed to get package for file ArmVirtPkg/Include/Platform/Hidden.h.  Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\\Platform'
ERROR - Failed to get package for file ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\Scripts'
ERROR - Failed to get package for file ArmVirtPkg/Include/Platform/Hidden.h.  Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\\Platform'
ERROR - Failed to get package for file ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\Scripts'
Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eval.exe\__main__.py", line 7, in <module>
  File "c:\work\github\tianocore\edk2\env\lib\site-packages\edk2toolext\invocables\edk2_pr_eval.py", line 389, in main
    Edk2PrEval().Invoke()
  File "c:\work\github\tianocore\edk2\env\lib\site-packages\edk2toolext\base_abstract_invocable.py", line 141, in Invoke
    retcode = self.Go()
  File "c:\work\github\tianocore\edk2\env\lib\site-packages\edk2toolext\invocables\edk2_pr_eval.py", line 82, in Go
    actualPackagesDict = self.get_packages_to_build(self.requested_package_list)
  File "c:\work\github\tianocore\edk2\env\lib\site-packages\edk2toolext\invocables\edk2_pr_eval.py", line 202, in get_packages_to_build
    changed_modules = self._get_unique_module_infs_changed(files)
  File "c:\work\github\tianocore\edk2\env\lib\site-packages\edk2toolext\invocables\edk2_pr_eval.py", line 235, in _get_unique_module_infs_changed
    infs = self.edk2_path_obj.GetContainingModules(os.path.abspath(f))
  File "c:\work\github\tianocore\edk2\env\lib\site-packages\edk2toollib\uefi\edk2\path_utilities.py", line 211, in GetContainingModules
    for f in os.listdir(dirpath):
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\\Platform'

Best regards,

Mike

> -----Original Message-----
> From: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Sent: Friday, June 12, 2020 9:16 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io; Sean Brogan
> <spbrogan@outlook.com>; lersek@redhat.com
> Subject: Re: [edk2-devel] ci problems again
> 
> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
> > Hi Ard,
> >
> > The CI agents are stateless, so it can not be stale
> > content.
> >
> 
> So why is it looking for ArmVirtPkg/PrePi/Scripts/PrePi-
> PIE.lds even
> though not a single reference to it remains anywhere in
> the tree?
> 
> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
> hits)
> 
> 
> > Have you tried the pytools command that is being used
> > locally for this patch series?
> >
> 
> No. Are there any instructions on how to use that?
> 
> 
> 
> >
> >> -----Original Message-----
> >> From: devel@edk2.groups.io <devel@edk2.groups.io> On
> >> Behalf Of Ard Biesheuvel
> >> Sent: Friday, June 12, 2020 8:47 AM
> >> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> >> devel@edk2.groups.io; Sean Brogan
> >> <spbrogan@outlook.com>; lersek@redhat.com
> >> Subject: Re: [edk2-devel] ci problems again
> >>
> >> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
> >>> Hi Ard,
> >>>
> >>> Please look at this more detailed log from Azure.
> >>>
> >>> https://dev.azure.com/tianocore/edk2-
> >>
> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
> >> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
> 127b-
> >> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
> >>>
> >>> It says it can not find some paths in ArmVirtPkg.
> >>>
> >>
> >> It is complaining about files that are being removed
> by
> >> the series:
> >>
> >> Failed to get package for file
> >> ArmVirtPkg/Include/Platform/Hidden.h.
> >> Failed to get package for file
> >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
> >>
> >> but I cannot find any remaining references to those
> >> files in anywhere in
> >> the tree, so it seems the CI has remembered the
> >> existence of those files
> >> in some way, and is now expecting them to be still
> >> there.
> >>
> >> So I don't think there is any way I can fix this
> myself.
> >> Someone needs
> >> to wipe the build directory for these CI jobs.
> >>
> >> 
> >

Re: [edk2-devel] ci problems again

[Michael D Kinney]

Hi Ard,

I think this is related to a patch series that deletes
some files.

The following code in the edk2-pytool-extensions is collecting
the files that change in a pr to determine the components that
need to be built.

https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271

It does not filter out files that have been deleted.

I tried the following change that filters out files that
have been deleted and reverses the diff to be from the
base branch to HEAD so the files deleted filter works as
expected.  The stuart command completes without errors.

diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
index 00ac282..a3f55b7 100644
--- a/edk2toolext/invocables/edk2_pr_eval.py
+++ b/edk2toolext/invocables/edk2_pr_eval.py
@@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:

         # get file differences between pr and base
         output = StringIO()
-        cmd_params = f"diff --name-only HEAD..{base_branch}"
+        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
         rc = RunCmd("git", cmd_params, outstream=output)

         if(rc == 0):

I do not know if this change has other unintended side effects.  I
need edk2-pytools-extensions to provide input.

Mike

> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Friday, June 12, 2020 9:49 AM
> To: Ard Biesheuvel <ard.biesheuvel@arm.com>;
> devel@edk2.groups.io; Sean Brogan
> <spbrogan@outlook.com>; lersek@redhat.com; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Subject: RE: [edk2-devel] ci problems again
> 
> Hi Ard,
> 
> The general instructions to setup env are in the repo
> here:
> 
> 
> https://github.com/tianocore/edk2/tree/master/.pytool
> 
> I did this and pulled PR 689 and ran the command from
> the
> Azure pipelines log and reproduced the same failure on
> my
> laptop.  Here is the log from my laptop.  I will debug a
> bit more to see what the issue is.  This command passes
> with edk2/master.  It fails with my pr_689 branch.
> 
> 
> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
> Switched to branch 'pr_689'
> 
> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
> pr-target origin/master
> SECTION - Init SDE
> SECTION - Loading Plugins
> SECTION - Start Invocable Tool
> ERROR - Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> ERROR - Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
> cripts'
> ERROR - Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> ERROR - Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
> cripts'
> Traceback (most recent call last):
>   File "C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
> _run_module_as_main
>     "__main__", mod_spec)
>   File "C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
> _run_code
>     exec(code, run_globals)
>   File
> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
> l.exe\__main__.py", line 7, in <module>
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 389, in main
>     Edk2PrEval().Invoke()
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\base_abstract_invocable.py", line
> 141, in Invoke
>     retcode = self.Go()
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 82, in Go
>     actualPackagesDict =
> self.get_packages_to_build(self.requested_package_list)
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 202, in get_packages_to_build
>     changed_modules =
> self._get_unique_module_infs_changed(files)
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 235, in _get_unique_module_infs_changed
>     infs =
> self.edk2_path_obj.GetContainingModules(os.path.abspath(
> f))
>   File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toollib\uefi\edk2\path_utilities.py", line
> 211, in GetContainingModules
>     for f in os.listdir(dirpath):
> FileNotFoundError: [WinError 3] The system cannot find
> the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> 
> Best regards,
> 
> Mike
> 
> > -----Original Message-----
> > From: Ard Biesheuvel <ard.biesheuvel@arm.com>
> > Sent: Friday, June 12, 2020 9:16 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > devel@edk2.groups.io; Sean Brogan
> > <spbrogan@outlook.com>; lersek@redhat.com
> > Subject: Re: [edk2-devel] ci problems again
> >
> > On 6/12/20 6:12 PM, Kinney, Michael D wrote:
> > > Hi Ard,
> > >
> > > The CI agents are stateless, so it can not be stale
> > > content.
> > >
> >
> > So why is it looking for
> ArmVirtPkg/PrePi/Scripts/PrePi-
> > PIE.lds even
> > though not a single reference to it remains anywhere
> in
> > the tree?
> >
> > (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
> > hits)
> >
> >
> > > Have you tried the pytools command that is being
> used
> > > locally for this patch series?
> > >
> >
> > No. Are there any instructions on how to use that?
> >
> >
> >
> > >
> > >> -----Original Message-----
> > >> From: devel@edk2.groups.io <devel@edk2.groups.io>
> On
> > >> Behalf Of Ard Biesheuvel
> > >> Sent: Friday, June 12, 2020 8:47 AM
> > >> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > >> devel@edk2.groups.io; Sean Brogan
> > >> <spbrogan@outlook.com>; lersek@redhat.com
> > >> Subject: Re: [edk2-devel] ci problems again
> > >>
> > >> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
> > >>> Hi Ard,
> > >>>
> > >>> Please look at this more detailed log from Azure.
> > >>>
> > >>> https://dev.azure.com/tianocore/edk2-
> > >>
> >
> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
> > >> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
> > 127b-
> > >> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
> > >>>
> > >>> It says it can not find some paths in ArmVirtPkg.
> > >>>
> > >>
> > >> It is complaining about files that are being
> removed
> > by
> > >> the series:
> > >>
> > >> Failed to get package for file
> > >> ArmVirtPkg/Include/Platform/Hidden.h.
> > >> Failed to get package for file
> > >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
> > >>
> > >> but I cannot find any remaining references to those
> > >> files in anywhere in
> > >> the tree, so it seems the CI has remembered the
> > >> existence of those files
> > >> in some way, and is now expecting them to be still
> > >> there.
> > >>
> > >> So I don't think there is any way I can fix this
> > myself.
> > >> Someone needs
> > >> to wipe the build directory for these CI jobs.
> > >>
> > >> 
> > >

Re: [edk2-devel] ci problems again

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 8486 bytes --]

Mike,

Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this?

Thanks,

Andrew Fish

> On Jun 12, 2020, at 11:19 AM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> 
> Hi Ard,
> 
> I think this is related to a patch series that deletes
> some files.
> 
> The following code in the edk2-pytool-extensions is collecting
> the files that change in a pr to determine the components that
> need to be built.
> 
> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 <https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271>
> 
> It does not filter out files that have been deleted.
> 
> I tried the following change that filters out files that
> have been deleted and reverses the diff to be from the
> base branch to HEAD so the files deleted filter works as
> expected.  The stuart command completes without errors.
> 
> diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
> index 00ac282..a3f55b7 100644
> --- a/edk2toolext/invocables/edk2_pr_eval.py
> +++ b/edk2toolext/invocables/edk2_pr_eval.py
> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:
> 
>         # get file differences between pr and base
>         output = StringIO()
> -        cmd_params = f"diff --name-only HEAD..{base_branch}"
> +        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
>         rc = RunCmd("git", cmd_params, outstream=output)
> 
>         if(rc == 0):
> 
> I do not know if this change has other unintended side effects.  I
> need edk2-pytools-extensions to provide input.
> 
> Mike
> 
>> -----Original Message-----
>> From: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Sent: Friday, June 12, 2020 9:49 AM
>> To: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>;
>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>; Kinney,
>> Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Subject: RE: [edk2-devel] ci problems again
>> 
>> Hi Ard,
>> 
>> The general instructions to setup env are in the repo
>> here:
>> 
>> 
>> https://github.com/tianocore/edk2/tree/master/.pytool
>> 
>> I did this and pulled PR 689 and ran the command from
>> the
>> Azure pipelines log and reproduced the same failure on
>> my
>> laptop.  Here is the log from my laptop.  I will debug a
>> bit more to see what the issue is.  This command passes
>> with edk2/master.  It fails with my pr_689 branch.
>> 
>> 
>> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
>> Switched to branch 'pr_689'
>> 
>> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
>> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
>> pr-target origin/master
>> SECTION - Init SDE
>> SECTION - Loading Plugins
>> SECTION - Start Invocable Tool
>> ERROR - Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>> cripts'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>> cripts'
>> Traceback (most recent call last):
>>  File "C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
>> _run_module_as_main
>>    "__main__", mod_spec)
>>  File "C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
>> _run_code
>>    exec(code, run_globals)
>>  File
>> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
>> l.exe\__main__.py", line 7, in <module>
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 389, in main
>>    Edk2PrEval().Invoke()
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\base_abstract_invocable.py", line
>> 141, in Invoke
>>    retcode = self.Go()
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 82, in Go
>>    actualPackagesDict =
>> self.get_packages_to_build(self.requested_package_list)
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 202, in get_packages_to_build
>>    changed_modules =
>> self._get_unique_module_infs_changed(files)
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 235, in _get_unique_module_infs_changed
>>    infs =
>> self.edk2_path_obj.GetContainingModules(os.path.abspath(
>> f))
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toollib\uefi\edk2\path_utilities.py", line
>> 211, in GetContainingModules
>>    for f in os.listdir(dirpath):
>> FileNotFoundError: [WinError 3] The system cannot find
>> the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> 
>> Best regards,
>> 
>> Mike
>> 
>>> -----Original Message-----
>>> From: Ard Biesheuvel <ard.biesheuvel@arm.com>
>>> Sent: Friday, June 12, 2020 9:16 AM
>>> To: Kinney, Michael D <michael.d.kinney@intel.com>;
>>> devel@edk2.groups.io; Sean Brogan
>>> <spbrogan@outlook.com>; lersek@redhat.com
>>> Subject: Re: [edk2-devel] ci problems again
>>> 
>>> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
>>>> Hi Ard,
>>>> 
>>>> The CI agents are stateless, so it can not be stale
>>>> content.
>>>> 
>>> 
>>> So why is it looking for
>> ArmVirtPkg/PrePi/Scripts/PrePi-
>>> PIE.lds even
>>> though not a single reference to it remains anywhere
>> in
>>> the tree?
>>> 
>>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
>>> hits)
>>> 
>>> 
>>>> Have you tried the pytools command that is being
>> used
>>>> locally for this patch series?
>>>> 
>>> 
>>> No. Are there any instructions on how to use that?
>>> 
>>> 
>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io>
>> On
>>>>> Behalf Of Ard Biesheuvel
>>>>> Sent: Friday, June 12, 2020 8:47 AM
>>>>> To: Kinney, Michael D <michael.d.kinney@intel.com>;
>>>>> devel@edk2.groups.io; Sean Brogan
>>>>> <spbrogan@outlook.com>; lersek@redhat.com
>>>>> Subject: Re: [edk2-devel] ci problems again
>>>>> 
>>>>> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
>>>>>> Hi Ard,
>>>>>> 
>>>>>> Please look at this more detailed log from Azure.
>>>>>> 
>>>>>> https://dev.azure.com/tianocore/edk2-
>>>>> 
>>> 
>> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
>>>>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
>>> 127b-
>>>>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
>>>>>> 
>>>>>> It says it can not find some paths in ArmVirtPkg.
>>>>>> 
>>>>> 
>>>>> It is complaining about files that are being
>> removed
>>> by
>>>>> the series:
>>>>> 
>>>>> Failed to get package for file
>>>>> ArmVirtPkg/Include/Platform/Hidden.h.
>>>>> Failed to get package for file
>>>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
>>>>> 
>>>>> but I cannot find any remaining references to those
>>>>> files in anywhere in
>>>>> the tree, so it seems the CI has remembered the
>>>>> existence of those files
>>>>> in some way, and is now expecting them to be still
>>>>> there.
>>>>> 
>>>>> So I don't think there is any way I can fix this
>>> myself.
>>>>> Someone needs
>>>>> to wipe the build directory for these CI jobs.
>>>>> 
>>>>> 
>>>> 
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 35964 bytes --]

Re: [edk2-devel] ci problems again

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 8690 bytes --]

Hi Andrew,

Not related to patch vs patch set.  All commits in a pr are being evaluated.

After further evaluation, it appears it may be related to a patch set that deletes the last file from a directory.   When that happens, the parent directory is also deleted, and the missing dir is what is generating the python exception.  Need to add logic to cover that corner case and avoid exception.

The edk2-pytools owners are working on it now.

Mike

From: Andrew Fish <afish@apple.com>
Sent: Friday, June 12, 2020 11:43 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>; Sean Brogan <spbrogan@outlook.com>; lersek@redhat.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] ci problems again

Mike,

Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this?

Thanks,

Andrew Fish


On Jun 12, 2020, at 11:19 AM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Ard,

I think this is related to a patch series that deletes
some files.

The following code in the edk2-pytool-extensions is collecting
the files that change in a pr to determine the components that
need to be built.

https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271

It does not filter out files that have been deleted.

I tried the following change that filters out files that
have been deleted and reverses the diff to be from the
base branch to HEAD so the files deleted filter works as
expected.  The stuart command completes without errors.

diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
index 00ac282..a3f55b7 100644
--- a/edk2toolext/invocables/edk2_pr_eval.py
+++ b/edk2toolext/invocables/edk2_pr_eval.py
@@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:

        # get file differences between pr and base
        output = StringIO()
-        cmd_params = f"diff --name-only HEAD..{base_branch}"
+        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
        rc = RunCmd("git", cmd_params, outstream=output)

        if(rc == 0):

I do not know if this change has other unintended side effects.  I
need edk2-pytools-extensions to provide input.

Mike


-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Sent: Friday, June 12, 2020 9:49 AM
To: Ard Biesheuvel <ard.biesheuvel@arm.com<mailto:ard.biesheuvel@arm.com>>;
devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Sean Brogan
<spbrogan@outlook.com<mailto:spbrogan@outlook.com>>; lersek@redhat.com<mailto:lersek@redhat.com>; Kinney,
Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: RE: [edk2-devel] ci problems again

Hi Ard,

The general instructions to setup env are in the repo
here:


https://github.com/tianocore/edk2/tree/master/.pytool

I did this and pulled PR 689 and ran the command from
the
Azure pipelines log and reproduced the same failure on
my
laptop.  Here is the log from my laptop.  I will debug a
bit more to see what the issue is.  This command passes
with edk2/master.  It fails with my pr_689 branch.


(env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
Switched to branch 'pr_689'

(env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
pr-target origin/master
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
ERROR - Failed to get package for file
ArmVirtPkg/Include/Platform/Hidden.h.  Exception
[WinError 3] The system cannot find the path specified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
\Platform'
ERROR - Failed to get package for file
ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
[WinError 3] The system cannot find the path specified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
cripts'
ERROR - Failed to get package for file
ArmVirtPkg/Include/Platform/Hidden.h.  Exception
[WinError 3] The system cannot find the path specified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
\Platform'
ERROR - Failed to get package for file
ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
[WinError 3] The system cannot find the path specified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
cripts'
Traceback (most recent call last):
 File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python37_64\lib\runpy.py", line 193, in
_run_module_as_main
   "__main__", mod_spec)
 File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python37_64\lib\runpy.py", line 85, in
_run_code
   exec(code, run_globals)
 File
"c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
l.exe\__main__.py", line 7, in <module>
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\invocables\edk2_pr_eval.py", line
389, in main
   Edk2PrEval().Invoke()
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\base_abstract_invocable.py", line
141, in Invoke
   retcode = self.Go()
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\invocables\edk2_pr_eval.py", line
82, in Go
   actualPackagesDict =
self.get_packages_to_build(self.requested_package_list)
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\invocables\edk2_pr_eval.py", line
202, in get_packages_to_build
   changed_modules =
self._get_unique_module_infs_changed(files)
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toolext\invocables\edk2_pr_eval.py", line
235, in _get_unique_module_infs_changed
   infs =
self.edk2_path_obj.GetContainingModules(os.path.abspath(
f))
 File "c:\work\github\tianocore\edk2\env\lib\site-
packages\edk2toollib\uefi\edk2\path_utilities.py", line
211, in GetContainingModules
   for f in os.listdir(dirpath):
FileNotFoundError: [WinError 3] The system cannot find
the path specified:
'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
\Platform'

Best regards,

Mike


-----Original Message-----
From: Ard Biesheuvel <ard.biesheuvel@arm.com<mailto:ard.biesheuvel@arm.com>>
Sent: Friday, June 12, 2020 9:16 AM
To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;
devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Sean Brogan
<spbrogan@outlook.com<mailto:spbrogan@outlook.com>>; lersek@redhat.com<mailto:lersek@redhat.com>
Subject: Re: [edk2-devel] ci problems again

On 6/12/20 6:12 PM, Kinney, Michael D wrote:

Hi Ard,

The CI agents are stateless, so it can not be stale
content.

So why is it looking for
ArmVirtPkg/PrePi/Scripts/PrePi-

PIE.lds even
though not a single reference to it remains anywhere
in

the tree?

(grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
hits)



Have you tried the pytools command that is being
used

locally for this patch series?

No. Are there any instructions on how to use that?






-----Original Message-----
From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>
On

Behalf Of Ard Biesheuvel
Sent: Friday, June 12, 2020 8:47 AM
To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;
devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Sean Brogan
<spbrogan@outlook.com<mailto:spbrogan@outlook.com>>; lersek@redhat.com<mailto:lersek@redhat.com>
Subject: Re: [edk2-devel] ci problems again

On 6/12/20 5:36 PM, Kinney, Michael D wrote:

Hi Ard,

Please look at this more detailed log from Azure.

https://dev.azure.com/tianocore/edk2-


ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-

2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
127b-

ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72


It says it can not find some paths in ArmVirtPkg.

It is complaining about files that are being
removed

by

the series:

Failed to get package for file
ArmVirtPkg/Include/Platform/Hidden.h.
Failed to get package for file
ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.

but I cannot find any remaining references to those
files in anywhere in
the tree, so it seems the CI has remembered the
existence of those files
in some way, and is now expecting them to be still
there.

So I don't think there is any way I can fix this
myself.

Someone needs
to wipe the build directory for these CI jobs.







[-- Attachment #2: Type: text/html, Size: 62777 bytes --]

Re: [edk2-devel] ci problems again

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 9867 bytes --]

Mike,

Thanks so different encase. 

Thanks,

Andrew Fish

> On Jun 12, 2020, at 12:25 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
> 
> Hi Andrew,
>
> Not related to patch vs patch set.  All commits in a pr are being evaluated.
>
> After further evaluation, it appears it may be related to a patch set that deletes the last file from a directory.   When that happens, the parent directory is also deleted, and the missing dir is what is generating the python exception.  Need to add logic to cover that corner case and avoid exception.
>
> The edk2-pytools owners are working on it now.
>
> Mike
>
> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>> 
> Sent: Friday, June 12, 2020 11:43 AM
> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>; Sean Brogan <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
> Subject: Re: [edk2-devel] ci problems again
>
> Mike,
>
> Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this?
>
> Thanks,
>
> Andrew Fish
> 
> 
> On Jun 12, 2020, at 11:19 AM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>
> Hi Ard,
> 
> I think this is related to a patch series that deletes
> some files.
> 
> The following code in the edk2-pytool-extensions is collecting
> the files that change in a pr to determine the components that
> need to be built.
> 
> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 <https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271>
> 
> It does not filter out files that have been deleted.
> 
> I tried the following change that filters out files that
> have been deleted and reverses the diff to be from the
> base branch to HEAD so the files deleted filter works as
> expected.  The stuart command completes without errors.
> 
> diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
> index 00ac282..a3f55b7 100644
> --- a/edk2toolext/invocables/edk2_pr_eval.py
> +++ b/edk2toolext/invocables/edk2_pr_eval.py
> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:
> 
>         # get file differences between pr and base
>         output = StringIO()
> -        cmd_params = f"diff --name-only HEAD..{base_branch}"
> +        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
>         rc = RunCmd("git", cmd_params, outstream=output)
> 
>         if(rc == 0):
> 
> I do not know if this change has other unintended side effects.  I
> need edk2-pytools-extensions to provide input.
> 
> Mike
> 
> 
> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
> Sent: Friday, June 12, 2020 9:49 AM
> To: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>;
> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>; Kinney,
> Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
> Subject: RE: [edk2-devel] ci problems again
> 
> Hi Ard,
> 
> The general instructions to setup env are in the repo
> here:
> 
> 
> https://github.com/tianocore/edk2/tree/master/.pytool <https://github.com/tianocore/edk2/tree/master/.pytool>
> 
> I did this and pulled PR 689 and ran the command from
> the
> Azure pipelines log and reproduced the same failure on
> my
> laptop.  Here is the log from my laptop.  I will debug a
> bit more to see what the issue is.  This command passes
> with edk2/master.  It fails with my pr_689 branch.
> 
> 
> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
> Switched to branch 'pr_689'
> 
> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
> pr-target origin/master
> SECTION - Init SDE
> SECTION - Loading Plugins
> SECTION - Start Invocable Tool
> ERROR - Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> ERROR - Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
> cripts'
> ERROR - Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> ERROR - Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
> [WinError 3] The system cannot find the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
> cripts'
> Traceback (most recent call last):
>  File "C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
> _run_module_as_main
>    "__main__", mod_spec)
>  File "C:\Program Files (x86)\Microsoft Visual
> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
> _run_code
>    exec(code, run_globals)
>  File
> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
> l.exe\__main__.py", line 7, in <module>
>  File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 389, in main
>    Edk2PrEval().Invoke()
>  File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\base_abstract_invocable.py", line
> 141, in Invoke
>    retcode = self.Go()
>  File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 82, in Go
>    actualPackagesDict =
> self.get_packages_to_build(self.requested_package_list)
>  File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 202, in get_packages_to_build
>    changed_modules =
> self._get_unique_module_infs_changed(files)
>  File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toolext\invocables\edk2_pr_eval.py", line
> 235, in _get_unique_module_infs_changed
>    infs =
> self.edk2_path_obj.GetContainingModules(os.path.abspath(
> f))
>  File "c:\work\github\tianocore\edk2\env\lib\site-
> packages\edk2toollib\uefi\edk2\path_utilities.py", line
> 211, in GetContainingModules
>    for f in os.listdir(dirpath):
> FileNotFoundError: [WinError 3] The system cannot find
> the path specified:
> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
> \Platform'
> 
> Best regards,
> 
> Mike
> 
> 
> -----Original Message-----
> From: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>
> Sent: Friday, June 12, 2020 9:16 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>;
> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>
> Subject: Re: [edk2-devel] ci problems again
> 
> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
> 
> Hi Ard,
> 
> The CI agents are stateless, so it can not be stale
> content.
> 
> 
> So why is it looking for
> ArmVirtPkg/PrePi/Scripts/PrePi-
> 
> PIE.lds even
> though not a single reference to it remains anywhere
> in
> 
> the tree?
> 
> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
> hits)
> 
> 
> 
> Have you tried the pytools command that is being
> used
> 
> locally for this patch series?
> 
> 
> No. Are there any instructions on how to use that?
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>
> On
> 
> Behalf Of Ard Biesheuvel
> Sent: Friday, June 12, 2020 8:47 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>;
> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>
> Subject: Re: [edk2-devel] ci problems again
> 
> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
> 
> Hi Ard,
> 
> Please look at this more detailed log from Azure.
> 
> https://dev.azure.com/tianocore/edk2- <https://dev.azure.com/tianocore/edk2->
>
>
> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
> 
> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
> 127b-
> 
> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
> 
> 
> It says it can not find some paths in ArmVirtPkg.
> 
> 
> It is complaining about files that are being
> removed
> 
> by
> 
> the series:
> 
> Failed to get package for file
> ArmVirtPkg/Include/Platform/Hidden.h.
> Failed to get package for file
> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
> 
> but I cannot find any remaining references to those
> files in anywhere in
> the tree, so it seems the CI has remembered the
> existence of those files
> in some way, and is now expecting them to be still
> there.
> 
> So I don't think there is any way I can fix this
> myself.
> 
> Someone needs
> to wipe the build directory for these CI jobs.
> 
> 
>
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 29509 bytes --]

Re: [edk2-devel] ci problems again

[Sean]

A pytool bug is opened that provides details.
https://github.com/tianocore/edk2-pytool-extensions/issues/201

A PR for edk2-pytool-extensions and a release will be coming soon.
Since we are using loose version tracking in edk2 for the CI 
dependencies it will be automatically picked up and CI builds and push 
builds should be able to continue.

I'll send a follow up once the release is made.

Thanks
Sean



On 6/12/2020 12:27 PM, Andrew Fish via groups.io wrote:
> Mike,
> 
> Thanks so different encase.
> 
> Thanks,
> 
> Andrew Fish
> 
>> On Jun 12, 2020, at 12:25 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
>>
>> Hi Andrew,
>>
>> Not related to patch vs patch set.  All commits in a pr are being evaluated.
>>
>> After further evaluation, it appears it may be related to a patch set that deletes the last file from a directory.   When that happens, the parent directory is also deleted, and the missing dir is what is generating the python exception.  Need to add logic to cover that corner case and avoid exception.
>>
>> The edk2-pytools owners are working on it now.
>>
>> Mike
>>
>> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>>
>> Sent: Friday, June 12, 2020 11:43 AM
>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>; Sean Brogan <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
>> Subject: Re: [edk2-devel] ci problems again
>>
>> Mike,
>>
>> Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this?
>>
>> Thanks,
>>
>> Andrew Fish
>>
>>
>> On Jun 12, 2020, at 11:19 AM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>>
>> Hi Ard,
>>
>> I think this is related to a patch series that deletes
>> some files.
>>
>> The following code in the edk2-pytool-extensions is collecting
>> the files that change in a pr to determine the components that
>> need to be built.
>>
>> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 <https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271>
>>
>> It does not filter out files that have been deleted.
>>
>> I tried the following change that filters out files that
>> have been deleted and reverses the diff to be from the
>> base branch to HEAD so the files deleted filter works as
>> expected.  The stuart command completes without errors.
>>
>> diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
>> index 00ac282..a3f55b7 100644
>> --- a/edk2toolext/invocables/edk2_pr_eval.py
>> +++ b/edk2toolext/invocables/edk2_pr_eval.py
>> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:
>>
>>          # get file differences between pr and base
>>          output = StringIO()
>> -        cmd_params = f"diff --name-only HEAD..{base_branch}"
>> +        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
>>          rc = RunCmd("git", cmd_params, outstream=output)
>>
>>          if(rc == 0):
>>
>> I do not know if this change has other unintended side effects.  I
>> need edk2-pytools-extensions to provide input.
>>
>> Mike
>>
>>
>> -----Original Message-----
>> From: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Sent: Friday, June 12, 2020 9:49 AM
>> To: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>;
>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>; Kinney,
>> Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Subject: RE: [edk2-devel] ci problems again
>>
>> Hi Ard,
>>
>> The general instructions to setup env are in the repo
>> here:
>>
>>
>> https://github.com/tianocore/edk2/tree/master/.pytool <https://github.com/tianocore/edk2/tree/master/.pytool>
>>
>> I did this and pulled PR 689 and ran the command from
>> the
>> Azure pipelines log and reproduced the same failure on
>> my
>> laptop.  Here is the log from my laptop.  I will debug a
>> bit more to see what the issue is.  This command passes
>> with edk2/master.  It fails with my pr_689 branch.
>>
>>
>> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
>> Switched to branch 'pr_689'
>>
>> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
>> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
>> pr-target origin/master
>> SECTION - Init SDE
>> SECTION - Loading Plugins
>> SECTION - Start Invocable Tool
>> ERROR - Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>> cripts'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>> cripts'
>> Traceback (most recent call last):
>>   File "C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
>> _run_module_as_main
>>     "__main__", mod_spec)
>>   File "C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
>> _run_code
>>     exec(code, run_globals)
>>   File
>> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
>> l.exe\__main__.py", line 7, in <module>
>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 389, in main
>>     Edk2PrEval().Invoke()
>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\base_abstract_invocable.py", line
>> 141, in Invoke
>>     retcode = self.Go()
>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 82, in Go
>>     actualPackagesDict =
>> self.get_packages_to_build(self.requested_package_list)
>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 202, in get_packages_to_build
>>     changed_modules =
>> self._get_unique_module_infs_changed(files)
>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 235, in _get_unique_module_infs_changed
>>     infs =
>> self.edk2_path_obj.GetContainingModules(os.path.abspath(
>> f))
>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toollib\uefi\edk2\path_utilities.py", line
>> 211, in GetContainingModules
>>     for f in os.listdir(dirpath):
>> FileNotFoundError: [WinError 3] The system cannot find
>> the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>>
>> Best regards,
>>
>> Mike
>>
>>
>> -----Original Message-----
>> From: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>
>> Sent: Friday, June 12, 2020 9:16 AM
>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>;
>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>
>> Subject: Re: [edk2-devel] ci problems again
>>
>> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
>>
>> Hi Ard,
>>
>> The CI agents are stateless, so it can not be stale
>> content.
>>
>>
>> So why is it looking for
>> ArmVirtPkg/PrePi/Scripts/PrePi-
>>
>> PIE.lds even
>> though not a single reference to it remains anywhere
>> in
>>
>> the tree?
>>
>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
>> hits)
>>
>>
>>
>> Have you tried the pytools command that is being
>> used
>>
>> locally for this patch series?
>>
>>
>> No. Are there any instructions on how to use that?
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>
>> On
>>
>> Behalf Of Ard Biesheuvel
>> Sent: Friday, June 12, 2020 8:47 AM
>> To: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>;
>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>
>> Subject: Re: [edk2-devel] ci problems again
>>
>> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
>>
>> Hi Ard,
>>
>> Please look at this more detailed log from Azure.
>>
>> https://dev.azure.com/tianocore/edk2- <https://dev.azure.com/tianocore/edk2->
>>
>>
>> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
>>
>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
>> 127b-
>>
>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
>>
>>
>> It says it can not find some paths in ArmVirtPkg.
>>
>>
>> It is complaining about files that are being
>> removed
>>
>> by
>>
>> the series:
>>
>> Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.
>> Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
>>
>> but I cannot find any remaining references to those
>> files in anywhere in
>> the tree, so it seems the CI has remembered the
>> existence of those files
>> in some way, and is now expecting them to be still
>> there.
>>
>> So I don't think there is any way I can fix this
>> myself.
>>
>> Someone needs
>> to wipe the build directory for these CI jobs.
>>
>>
>>
>>
>>
>>
> 
> 
> 
> 
> 

Re: [edk2-devel] ci problems again

[Sean]

Bug fixed.  Release 0.13.7 created and published.

@ard - You should be able to get past your current issue on PR 689.

Thanks
Sean


On 6/12/2020 12:59 PM, Sean wrote:
> A pytool bug is opened that provides details.
> https://github.com/tianocore/edk2-pytool-extensions/issues/201
> 
> A PR for edk2-pytool-extensions and a release will be coming soon.
> Since we are using loose version tracking in edk2 for the CI 
> dependencies it will be automatically picked up and CI builds and push 
> builds should be able to continue.
> 
> I'll send a follow up once the release is made.
> 
> Thanks
> Sean
> 
> 
> 
> On 6/12/2020 12:27 PM, Andrew Fish via groups.io wrote:
>> Mike,
>>
>> Thanks so different encase.
>>
>> Thanks,
>>
>> Andrew Fish
>>
>>> On Jun 12, 2020, at 12:25 PM, Kinney, Michael D 
>>> <michael.d.kinney@intel.com> wrote:
>>>
>>> Hi Andrew,
>>>
>>> Not related to patch vs patch set.  All commits in a pr are being 
>>> evaluated.
>>>
>>> After further evaluation, it appears it may be related to a patch set 
>>> that deletes the last file from a directory.   When that happens, the 
>>> parent directory is also deleted, and the missing dir is what is 
>>> generating the python exception.  Need to add logic to cover that 
>>> corner case and avoid exception.
>>>
>>> The edk2-pytools owners are working on it now.
>>>
>>> Mike
>>>
>>> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>>
>>> Sent: Friday, June 12, 2020 11:43 AM
>>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, 
>>> Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>
>>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com 
>>> <mailto:ard.biesheuvel@arm.com>>; Sean Brogan <spbrogan@outlook.com 
>>> <mailto:spbrogan@outlook.com>>; lersek@redhat.com 
>>> <mailto:lersek@redhat.com>; Bret Barkelew 
>>> <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
>>> Subject: Re: [edk2-devel] ci problems again
>>>
>>> Mike,
>>>
>>> Is this an issue of assuming a patch vs. a patch set? Are there other 
>>> places we could run into this?
>>>
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>> On Jun 12, 2020, at 11:19 AM, Michael D Kinney 
>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>>>
>>> Hi Ard,
>>>
>>> I think this is related to a patch series that deletes
>>> some files.
>>>
>>> The following code in the edk2-pytool-extensions is collecting
>>> the files that change in a pr to determine the components that
>>> need to be built.
>>>
>>> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 
>>> <https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271> 
>>>
>>>
>>> It does not filter out files that have been deleted.
>>>
>>> I tried the following change that filters out files that
>>> have been deleted and reverses the diff to be from the
>>> base branch to HEAD so the files deleted filter works as
>>> expected.  The stuart command completes without errors.
>>>
>>> diff --git a/edk2toolext/invocables/edk2_pr_eval.py 
>>> b/edk2toolext/invocables/edk2_pr_eval.py
>>> index 00ac282..a3f55b7 100644
>>> --- a/edk2toolext/invocables/edk2_pr_eval.py
>>> +++ b/edk2toolext/invocables/edk2_pr_eval.py
>>> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, 
>>> base_branch) -> tuple:
>>>
>>>          # get file differences between pr and base
>>>          output = StringIO()
>>> -        cmd_params = f"diff --name-only HEAD..{base_branch}"
>>> +        cmd_params = f"diff --name-only --diff-filter=d 
>>> {base_branch}..HEAD"
>>>          rc = RunCmd("git", cmd_params, outstream=output)
>>>
>>>          if(rc == 0):
>>>
>>> I do not know if this change has other unintended side effects.  I
>>> need edk2-pytools-extensions to provide input.
>>>
>>> Mike
>>>
>>>
>>> -----Original Message-----
>>> From: Kinney, Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>
>>> Sent: Friday, June 12, 2020 9:49 AM
>>> To: Ard Biesheuvel <ard.biesheuvel@arm.com 
>>> <mailto:ard.biesheuvel@arm.com>>;
>>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; 
>>> lersek@redhat.com <mailto:lersek@redhat.com>; Kinney,
>>> Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>
>>> Subject: RE: [edk2-devel] ci problems again
>>>
>>> Hi Ard,
>>>
>>> The general instructions to setup env are in the repo
>>> here:
>>>
>>>
>>> https://github.com/tianocore/edk2/tree/master/.pytool 
>>> <https://github.com/tianocore/edk2/tree/master/.pytool>
>>>
>>> I did this and pulled PR 689 and ran the command from
>>> the
>>> Azure pipelines log and reproduced the same failure on
>>> my
>>> laptop.  Here is the log from my laptop.  I will debug a
>>> bit more to see what the issue is.  This command passes
>>> with edk2/master.  It fails with my pr_689 branch.
>>>
>>>
>>> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
>>> Switched to branch 'pr_689'
>>>
>>> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
>>> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
>>> pr-target origin/master
>>> SECTION - Init SDE
>>> SECTION - Loading Plugins
>>> SECTION - Start Invocable Tool
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>>> \Platform'
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>>> cripts'
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>>> \Platform'
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>>> cripts'
>>> Traceback (most recent call last):
>>>   File "C:\Program Files (x86)\Microsoft Visual
>>> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
>>> _run_module_as_main
>>>     "__main__", mod_spec)
>>>   File "C:\Program Files (x86)\Microsoft Visual
>>> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
>>> _run_code
>>>     exec(code, run_globals)
>>>   File
>>> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
>>> l.exe\__main__.py", line 7, in <module>
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 389, in main
>>>     Edk2PrEval().Invoke()
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\base_abstract_invocable.py", line
>>> 141, in Invoke
>>>     retcode = self.Go()
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 82, in Go
>>>     actualPackagesDict =
>>> self.get_packages_to_build(self.requested_package_list)
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 202, in get_packages_to_build
>>>     changed_modules =
>>> self._get_unique_module_infs_changed(files)
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 235, in _get_unique_module_infs_changed
>>>     infs =
>>> self.edk2_path_obj.GetContainingModules(os.path.abspath(
>>> f))
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toollib\uefi\edk2\path_utilities.py", line
>>> 211, in GetContainingModules
>>>     for f in os.listdir(dirpath):
>>> FileNotFoundError: [WinError 3] The system cannot find
>>> the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>>> \Platform'
>>>
>>> Best regards,
>>>
>>> Mike
>>>
>>>
>>> -----Original Message-----
>>> From: Ard Biesheuvel <ard.biesheuvel@arm.com 
>>> <mailto:ard.biesheuvel@arm.com>>
>>> Sent: Friday, June 12, 2020 9:16 AM
>>> To: Kinney, Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>;
>>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; 
>>> lersek@redhat.com <mailto:lersek@redhat.com>
>>> Subject: Re: [edk2-devel] ci problems again
>>>
>>> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
>>>
>>> Hi Ard,
>>>
>>> The CI agents are stateless, so it can not be stale
>>> content.
>>>
>>>
>>> So why is it looking for
>>> ArmVirtPkg/PrePi/Scripts/PrePi-
>>>
>>> PIE.lds even
>>> though not a single reference to it remains anywhere
>>> in
>>>
>>> the tree?
>>>
>>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
>>> hits)
>>>
>>>
>>>
>>> Have you tried the pytools command that is being
>>> used
>>>
>>> locally for this patch series?
>>>
>>>
>>> No. Are there any instructions on how to use that?
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
>>> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>
>>> On
>>>
>>> Behalf Of Ard Biesheuvel
>>> Sent: Friday, June 12, 2020 8:47 AM
>>> To: Kinney, Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>;
>>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; 
>>> lersek@redhat.com <mailto:lersek@redhat.com>
>>> Subject: Re: [edk2-devel] ci problems again
>>>
>>> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
>>>
>>> Hi Ard,
>>>
>>> Please look at this more detailed log from Azure.
>>>
>>> https://dev.azure.com/tianocore/edk2- 
>>> <https://dev.azure.com/tianocore/edk2->
>>>
>>>
>>> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
>>>
>>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
>>> 127b-
>>>
>>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
>>>
>>>
>>> It says it can not find some paths in ArmVirtPkg.
>>>
>>>
>>> It is complaining about files that are being
>>> removed
>>>
>>> by
>>>
>>> the series:
>>>
>>> Failed to get package for file
>>> ArmVirtPkg/Include/Platform/Hidden.h.
>>> Failed to get package for file
>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
>>>
>>> but I cannot find any remaining references to those
>>> files in anywhere in
>>> the tree, so it seems the CI has remembered the
>>> existence of those files
>>> in some way, and is now expecting them to be still
>>> there.
>>>
>>> So I don't think there is any way I can fix this
>>> myself.
>>>
>>> Someone needs
>>> to wipe the build directory for these CI jobs.
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
> 
> 
> 

Re: [edk2-devel] ci problems again

[Ard Biesheuvel]

On 6/12/20 11:45 PM, Sean Brogan wrote:
> Bug fixed.  Release 0.13.7 created and published.
> 
> @ard - You should be able to get past your current issue on PR 689.
> 

Thanks all for the diagnosis and the fix.