Mike, Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this? Thanks, Andrew Fish > On Jun 12, 2020, at 11:19 AM, Michael D Kinney wrote: > > Hi Ard, > > I think this is related to a patch series that deletes > some files. > > The following code in the edk2-pytool-extensions is collecting > the files that change in a pr to determine the components that > need to be built. > > https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 > > It does not filter out files that have been deleted. > > I tried the following change that filters out files that > have been deleted and reverses the diff to be from the > base branch to HEAD so the files deleted filter works as > expected. The stuart command completes without errors. > > diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py > index 00ac282..a3f55b7 100644 > --- a/edk2toolext/invocables/edk2_pr_eval.py > +++ b/edk2toolext/invocables/edk2_pr_eval.py > @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple: > > # get file differences between pr and base > output = StringIO() > - cmd_params = f"diff --name-only HEAD..{base_branch}" > + cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD" > rc = RunCmd("git", cmd_params, outstream=output) > > if(rc == 0): > > I do not know if this change has other unintended side effects. I > need edk2-pytools-extensions to provide input. > > Mike > >> -----Original Message----- >> From: Kinney, Michael D > >> Sent: Friday, June 12, 2020 9:49 AM >> To: Ard Biesheuvel >; >> devel@edk2.groups.io ; Sean Brogan >> >; lersek@redhat.com ; Kinney, >> Michael D > >> Subject: RE: [edk2-devel] ci problems again >> >> Hi Ard, >> >> The general instructions to setup env are in the repo >> here: >> >> >> https://github.com/tianocore/edk2/tree/master/.pytool >> >> I did this and pulled PR 689 and ran the command from >> the >> Azure pipelines log and reproduced the same failure on >> my >> laptop. Here is the log from my laptop. I will debug a >> bit more to see what the issue is. This command passes >> with edk2/master. It fails with my pr_689 branch. >> >> >> (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689 >> Switched to branch 'pr_689' >> >> (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c >> OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 -- >> pr-target origin/master >> SECTION - Init SDE >> SECTION - Loading Plugins >> SECTION - Start Invocable Tool >> ERROR - Failed to get package for file >> ArmVirtPkg/Include/Platform/Hidden.h. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >> \Platform' >> ERROR - Failed to get package for file >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S >> cripts' >> ERROR - Failed to get package for file >> ArmVirtPkg/Include/Platform/Hidden.h. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >> \Platform' >> ERROR - Failed to get package for file >> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. Exception >> [WinError 3] The system cannot find the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S >> cripts' >> Traceback (most recent call last): >> File "C:\Program Files (x86)\Microsoft Visual >> Studio\Shared\Python37_64\lib\runpy.py", line 193, in >> _run_module_as_main >> "__main__", mod_spec) >> File "C:\Program Files (x86)\Microsoft Visual >> Studio\Shared\Python37_64\lib\runpy.py", line 85, in >> _run_code >> exec(code, run_globals) >> File >> "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva >> l.exe\__main__.py", line 7, in >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 389, in main >> Edk2PrEval().Invoke() >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\base_abstract_invocable.py", line >> 141, in Invoke >> retcode = self.Go() >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 82, in Go >> actualPackagesDict = >> self.get_packages_to_build(self.requested_package_list) >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 202, in get_packages_to_build >> changed_modules = >> self._get_unique_module_infs_changed(files) >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toolext\invocables\edk2_pr_eval.py", line >> 235, in _get_unique_module_infs_changed >> infs = >> self.edk2_path_obj.GetContainingModules(os.path.abspath( >> f)) >> File "c:\work\github\tianocore\edk2\env\lib\site- >> packages\edk2toollib\uefi\edk2\path_utilities.py", line >> 211, in GetContainingModules >> for f in os.listdir(dirpath): >> FileNotFoundError: [WinError 3] The system cannot find >> the path specified: >> 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ >> \Platform' >> >> Best regards, >> >> Mike >> >>> -----Original Message----- >>> From: Ard Biesheuvel >>> Sent: Friday, June 12, 2020 9:16 AM >>> To: Kinney, Michael D ; >>> devel@edk2.groups.io; Sean Brogan >>> ; lersek@redhat.com >>> Subject: Re: [edk2-devel] ci problems again >>> >>> On 6/12/20 6:12 PM, Kinney, Michael D wrote: >>>> Hi Ard, >>>> >>>> The CI agents are stateless, so it can not be stale >>>> content. >>>> >>> >>> So why is it looking for >> ArmVirtPkg/PrePi/Scripts/PrePi- >>> PIE.lds even >>> though not a single reference to it remains anywhere >> in >>> the tree? >>> >>> (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0 >>> hits) >>> >>> >>>> Have you tried the pytools command that is being >> used >>>> locally for this patch series? >>>> >>> >>> No. Are there any instructions on how to use that? >>> >>> >>> >>>> >>>>> -----Original Message----- >>>>> From: devel@edk2.groups.io >> On >>>>> Behalf Of Ard Biesheuvel >>>>> Sent: Friday, June 12, 2020 8:47 AM >>>>> To: Kinney, Michael D ; >>>>> devel@edk2.groups.io; Sean Brogan >>>>> ; lersek@redhat.com >>>>> Subject: Re: [edk2-devel] ci problems again >>>>> >>>>> On 6/12/20 5:36 PM, Kinney, Michael D wrote: >>>>>> Hi Ard, >>>>>> >>>>>> Please look at this more detailed log from Azure. >>>>>> >>>>>> https://dev.azure.com/tianocore/edk2- >>>>> >>> >> ci/_build/results?buildId=8170&view=logs&jobId=680b79ca- >>>>> 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4- >>> 127b- >>>>> ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72 >>>>>> >>>>>> It says it can not find some paths in ArmVirtPkg. >>>>>> >>>>> >>>>> It is complaining about files that are being >> removed >>> by >>>>> the series: >>>>> >>>>> Failed to get package for file >>>>> ArmVirtPkg/Include/Platform/Hidden.h. >>>>> Failed to get package for file >>>>> ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. >>>>> >>>>> but I cannot find any remaining references to those >>>>> files in anywhere in >>>>> the tree, so it seems the CI has remembered the >>>>> existence of those files >>>>> in some way, and is now expecting them to be still >>>>> there. >>>>> >>>>> So I don't think there is any way I can fix this >>> myself. >>>>> Someone needs >>>>> to wipe the build directory for these CI jobs. >>>>> >>>>> >>>> > > >