Re: [edk2-devel] ci problems again

["Sean" <spbrogan@outlook.com>]

Bug fixed.  Release 0.13.7 created and published.

@ard - You should be able to get past your current issue on PR 689.

Thanks
Sean


On 6/12/2020 12:59 PM, Sean wrote:
> A pytool bug is opened that provides details.
> https://github.com/tianocore/edk2-pytool-extensions/issues/201
> 
> A PR for edk2-pytool-extensions and a release will be coming soon.
> Since we are using loose version tracking in edk2 for the CI 
> dependencies it will be automatically picked up and CI builds and push 
> builds should be able to continue.
> 
> I'll send a follow up once the release is made.
> 
> Thanks
> Sean
> 
> 
> 
> On 6/12/2020 12:27 PM, Andrew Fish via groups.io wrote:
>> Mike,
>>
>> Thanks so different encase.
>>
>> Thanks,
>>
>> Andrew Fish
>>
>>> On Jun 12, 2020, at 12:25 PM, Kinney, Michael D 
>>> <michael.d.kinney@intel.com> wrote:
>>>
>>> Hi Andrew,
>>>
>>> Not related to patch vs patch set.  All commits in a pr are being 
>>> evaluated.
>>>
>>> After further evaluation, it appears it may be related to a patch set 
>>> that deletes the last file from a directory.   When that happens, the 
>>> parent directory is also deleted, and the missing dir is what is 
>>> generating the python exception.  Need to add logic to cover that 
>>> corner case and avoid exception.
>>>
>>> The edk2-pytools owners are working on it now.
>>>
>>> Mike
>>>
>>> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>>
>>> Sent: Friday, June 12, 2020 11:43 AM
>>> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, 
>>> Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>
>>> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com 
>>> <mailto:ard.biesheuvel@arm.com>>; Sean Brogan <spbrogan@outlook.com 
>>> <mailto:spbrogan@outlook.com>>; lersek@redhat.com 
>>> <mailto:lersek@redhat.com>; Bret Barkelew 
>>> <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
>>> Subject: Re: [edk2-devel] ci problems again
>>>
>>> Mike,
>>>
>>> Is this an issue of assuming a patch vs. a patch set? Are there other 
>>> places we could run into this?
>>>
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>> On Jun 12, 2020, at 11:19 AM, Michael D Kinney 
>>> <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>>>
>>> Hi Ard,
>>>
>>> I think this is related to a patch series that deletes
>>> some files.
>>>
>>> The following code in the edk2-pytool-extensions is collecting
>>> the files that change in a pr to determine the components that
>>> need to be built.
>>>
>>> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 
>>> <https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271> 
>>>
>>>
>>> It does not filter out files that have been deleted.
>>>
>>> I tried the following change that filters out files that
>>> have been deleted and reverses the diff to be from the
>>> base branch to HEAD so the files deleted filter works as
>>> expected.  The stuart command completes without errors.
>>>
>>> diff --git a/edk2toolext/invocables/edk2_pr_eval.py 
>>> b/edk2toolext/invocables/edk2_pr_eval.py
>>> index 00ac282..a3f55b7 100644
>>> --- a/edk2toolext/invocables/edk2_pr_eval.py
>>> +++ b/edk2toolext/invocables/edk2_pr_eval.py
>>> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, 
>>> base_branch) -> tuple:
>>>
>>>          # get file differences between pr and base
>>>          output = StringIO()
>>> -        cmd_params = f"diff --name-only HEAD..{base_branch}"
>>> +        cmd_params = f"diff --name-only --diff-filter=d 
>>> {base_branch}..HEAD"
>>>          rc = RunCmd("git", cmd_params, outstream=output)
>>>
>>>          if(rc == 0):
>>>
>>> I do not know if this change has other unintended side effects.  I
>>> need edk2-pytools-extensions to provide input.
>>>
>>> Mike
>>>
>>>
>>> -----Original Message-----
>>> From: Kinney, Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>
>>> Sent: Friday, June 12, 2020 9:49 AM
>>> To: Ard Biesheuvel <ard.biesheuvel@arm.com 
>>> <mailto:ard.biesheuvel@arm.com>>;
>>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; 
>>> lersek@redhat.com <mailto:lersek@redhat.com>; Kinney,
>>> Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>
>>> Subject: RE: [edk2-devel] ci problems again
>>>
>>> Hi Ard,
>>>
>>> The general instructions to setup env are in the repo
>>> here:
>>>
>>>
>>> https://github.com/tianocore/edk2/tree/master/.pytool 
>>> <https://github.com/tianocore/edk2/tree/master/.pytool>
>>>
>>> I did this and pulled PR 689 and ran the command from
>>> the
>>> Azure pipelines log and reproduced the same failure on
>>> my
>>> laptop.  Here is the log from my laptop.  I will debug a
>>> bit more to see what the issue is.  This command passes
>>> with edk2/master.  It fails with my pr_689 branch.
>>>
>>>
>>> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
>>> Switched to branch 'pr_689'
>>>
>>> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
>>> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
>>> pr-target origin/master
>>> SECTION - Init SDE
>>> SECTION - Loading Plugins
>>> SECTION - Start Invocable Tool
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>>> \Platform'
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>>> cripts'
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>>> \Platform'
>>> ERROR - Failed to get package for file
>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>>> [WinError 3] The system cannot find the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>>> cripts'
>>> Traceback (most recent call last):
>>>   File "C:\Program Files (x86)\Microsoft Visual
>>> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
>>> _run_module_as_main
>>>     "__main__", mod_spec)
>>>   File "C:\Program Files (x86)\Microsoft Visual
>>> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
>>> _run_code
>>>     exec(code, run_globals)
>>>   File
>>> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
>>> l.exe\__main__.py", line 7, in <module>
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 389, in main
>>>     Edk2PrEval().Invoke()
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\base_abstract_invocable.py", line
>>> 141, in Invoke
>>>     retcode = self.Go()
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 82, in Go
>>>     actualPackagesDict =
>>> self.get_packages_to_build(self.requested_package_list)
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 202, in get_packages_to_build
>>>     changed_modules =
>>> self._get_unique_module_infs_changed(files)
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>>> 235, in _get_unique_module_infs_changed
>>>     infs =
>>> self.edk2_path_obj.GetContainingModules(os.path.abspath(
>>> f))
>>>   File "c:\work\github\tianocore\edk2\env\lib\site-
>>> packages\edk2toollib\uefi\edk2\path_utilities.py", line
>>> 211, in GetContainingModules
>>>     for f in os.listdir(dirpath):
>>> FileNotFoundError: [WinError 3] The system cannot find
>>> the path specified:
>>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>>> \Platform'
>>>
>>> Best regards,
>>>
>>> Mike
>>>
>>>
>>> -----Original Message-----
>>> From: Ard Biesheuvel <ard.biesheuvel@arm.com 
>>> <mailto:ard.biesheuvel@arm.com>>
>>> Sent: Friday, June 12, 2020 9:16 AM
>>> To: Kinney, Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>;
>>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; 
>>> lersek@redhat.com <mailto:lersek@redhat.com>
>>> Subject: Re: [edk2-devel] ci problems again
>>>
>>> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
>>>
>>> Hi Ard,
>>>
>>> The CI agents are stateless, so it can not be stale
>>> content.
>>>
>>>
>>> So why is it looking for
>>> ArmVirtPkg/PrePi/Scripts/PrePi-
>>>
>>> PIE.lds even
>>> though not a single reference to it remains anywhere
>>> in
>>>
>>> the tree?
>>>
>>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
>>> hits)
>>>
>>>
>>>
>>> Have you tried the pytools command that is being
>>> used
>>>
>>> locally for this patch series?
>>>
>>>
>>> No. Are there any instructions on how to use that?
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
>>> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>
>>> On
>>>
>>> Behalf Of Ard Biesheuvel
>>> Sent: Friday, June 12, 2020 8:47 AM
>>> To: Kinney, Michael D <michael.d.kinney@intel.com 
>>> <mailto:michael.d.kinney@intel.com>>;
>>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; 
>>> lersek@redhat.com <mailto:lersek@redhat.com>
>>> Subject: Re: [edk2-devel] ci problems again
>>>
>>> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
>>>
>>> Hi Ard,
>>>
>>> Please look at this more detailed log from Azure.
>>>
>>> https://dev.azure.com/tianocore/edk2- 
>>> <https://dev.azure.com/tianocore/edk2->
>>>
>>>
>>> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
>>>
>>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
>>> 127b-
>>>
>>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
>>>
>>>
>>> It says it can not find some paths in ArmVirtPkg.
>>>
>>>
>>> It is complaining about files that are being
>>> removed
>>>
>>> by
>>>
>>> the series:
>>>
>>> Failed to get package for file
>>> ArmVirtPkg/Include/Platform/Hidden.h.
>>> Failed to get package for file
>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
>>>
>>> but I cannot find any remaining references to those
>>> files in anywhere in
>>> the tree, so it seems the CI has remembered the
>>> existence of those files
>>> in some way, and is now expecting them to be still
>>> there.
>>>
>>> So I don't think there is any way I can fix this
>>> myself.
>>>
>>> Someone needs
>>> to wipe the build directory for these CI jobs.
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
> 
> 
>