Re: [edk2-devel] ci problems again

["Andrew Fish" <afish@apple.com>]

[-- Attachment #1: Type: text/plain, Size: 8486 bytes --]

Mike,

Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this?

Thanks,

Andrew Fish

> On Jun 12, 2020, at 11:19 AM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> 
> Hi Ard,
> 
> I think this is related to a patch series that deletes
> some files.
> 
> The following code in the edk2-pytool-extensions is collecting
> the files that change in a pr to determine the components that
> need to be built.
> 
> https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 <https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271>
> 
> It does not filter out files that have been deleted.
> 
> I tried the following change that filters out files that
> have been deleted and reverses the diff to be from the
> base branch to HEAD so the files deleted filter works as
> expected.  The stuart command completes without errors.
> 
> diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py
> index 00ac282..a3f55b7 100644
> --- a/edk2toolext/invocables/edk2_pr_eval.py
> +++ b/edk2toolext/invocables/edk2_pr_eval.py
> @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple:
> 
>         # get file differences between pr and base
>         output = StringIO()
> -        cmd_params = f"diff --name-only HEAD..{base_branch}"
> +        cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD"
>         rc = RunCmd("git", cmd_params, outstream=output)
> 
>         if(rc == 0):
> 
> I do not know if this change has other unintended side effects.  I
> need edk2-pytools-extensions to provide input.
> 
> Mike
> 
>> -----Original Message-----
>> From: Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Sent: Friday, June 12, 2020 9:49 AM
>> To: Ard Biesheuvel <ard.biesheuvel@arm.com <mailto:ard.biesheuvel@arm.com>>;
>> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Sean Brogan
>> <spbrogan@outlook.com <mailto:spbrogan@outlook.com>>; lersek@redhat.com <mailto:lersek@redhat.com>; Kinney,
>> Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Subject: RE: [edk2-devel] ci problems again
>> 
>> Hi Ard,
>> 
>> The general instructions to setup env are in the repo
>> here:
>> 
>> 
>> https://github.com/tianocore/edk2/tree/master/.pytool
>> 
>> I did this and pulled PR 689 and ran the command from
>> the
>> Azure pipelines log and reproduced the same failure on
>> my
>> laptop.  Here is the log from my laptop.  I will debug a
>> bit more to see what the issue is.  This command passes
>> with edk2/master.  It fails with my pr_689 branch.
>> 
>> 
>> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689
>> Switched to branch 'pr_689'
>> 
>> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c
>> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 --
>> pr-target origin/master
>> SECTION - Init SDE
>> SECTION - Loading Plugins
>> SECTION - Start Invocable Tool
>> ERROR - Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>> cripts'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/Include/Platform/Hidden.h.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> ERROR - Failed to get package for file
>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.  Exception
>> [WinError 3] The system cannot find the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S
>> cripts'
>> Traceback (most recent call last):
>>  File "C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python37_64\lib\runpy.py", line 193, in
>> _run_module_as_main
>>    "__main__", mod_spec)
>>  File "C:\Program Files (x86)\Microsoft Visual
>> Studio\Shared\Python37_64\lib\runpy.py", line 85, in
>> _run_code
>>    exec(code, run_globals)
>>  File
>> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva
>> l.exe\__main__.py", line 7, in <module>
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 389, in main
>>    Edk2PrEval().Invoke()
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\base_abstract_invocable.py", line
>> 141, in Invoke
>>    retcode = self.Go()
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 82, in Go
>>    actualPackagesDict =
>> self.get_packages_to_build(self.requested_package_list)
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 202, in get_packages_to_build
>>    changed_modules =
>> self._get_unique_module_infs_changed(files)
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toolext\invocables\edk2_pr_eval.py", line
>> 235, in _get_unique_module_infs_changed
>>    infs =
>> self.edk2_path_obj.GetContainingModules(os.path.abspath(
>> f))
>>  File "c:\work\github\tianocore\edk2\env\lib\site-
>> packages\edk2toollib\uefi\edk2\path_utilities.py", line
>> 211, in GetContainingModules
>>    for f in os.listdir(dirpath):
>> FileNotFoundError: [WinError 3] The system cannot find
>> the path specified:
>> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\
>> \Platform'
>> 
>> Best regards,
>> 
>> Mike
>> 
>>> -----Original Message-----
>>> From: Ard Biesheuvel <ard.biesheuvel@arm.com>
>>> Sent: Friday, June 12, 2020 9:16 AM
>>> To: Kinney, Michael D <michael.d.kinney@intel.com>;
>>> devel@edk2.groups.io; Sean Brogan
>>> <spbrogan@outlook.com>; lersek@redhat.com
>>> Subject: Re: [edk2-devel] ci problems again
>>> 
>>> On 6/12/20 6:12 PM, Kinney, Michael D wrote:
>>>> Hi Ard,
>>>> 
>>>> The CI agents are stateless, so it can not be stale
>>>> content.
>>>> 
>>> 
>>> So why is it looking for
>> ArmVirtPkg/PrePi/Scripts/PrePi-
>>> PIE.lds even
>>> though not a single reference to it remains anywhere
>> in
>>> the tree?
>>> 
>>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0
>>> hits)
>>> 
>>> 
>>>> Have you tried the pytools command that is being
>> used
>>>> locally for this patch series?
>>>> 
>>> 
>>> No. Are there any instructions on how to use that?
>>> 
>>> 
>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: devel@edk2.groups.io <devel@edk2.groups.io>
>> On
>>>>> Behalf Of Ard Biesheuvel
>>>>> Sent: Friday, June 12, 2020 8:47 AM
>>>>> To: Kinney, Michael D <michael.d.kinney@intel.com>;
>>>>> devel@edk2.groups.io; Sean Brogan
>>>>> <spbrogan@outlook.com>; lersek@redhat.com
>>>>> Subject: Re: [edk2-devel] ci problems again
>>>>> 
>>>>> On 6/12/20 5:36 PM, Kinney, Michael D wrote:
>>>>>> Hi Ard,
>>>>>> 
>>>>>> Please look at this more detailed log from Azure.
>>>>>> 
>>>>>> https://dev.azure.com/tianocore/edk2-
>>>>> 
>>> 
>> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca-
>>>>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4-
>>> 127b-
>>>>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72
>>>>>> 
>>>>>> It says it can not find some paths in ArmVirtPkg.
>>>>>> 
>>>>> 
>>>>> It is complaining about files that are being
>> removed
>>> by
>>>>> the series:
>>>>> 
>>>>> Failed to get package for file
>>>>> ArmVirtPkg/Include/Platform/Hidden.h.
>>>>> Failed to get package for file
>>>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds.
>>>>> 
>>>>> but I cannot find any remaining references to those
>>>>> files in anywhere in
>>>>> the tree, so it seems the CI has remembered the
>>>>> existence of those files
>>>>> in some way, and is now expecting them to be still
>>>>> there.
>>>>> 
>>>>> So I don't think there is any way I can fix this
>>> myself.
>>>>> Someone needs
>>>>> to wipe the build directory for these CI jobs.
>>>>> 
>>>>> 
>>>> 
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 35964 bytes --]