Hi Andrew, Not related to patch vs patch set. All commits in a pr are being evaluated. After further evaluation, it appears it may be related to a patch set that deletes the last file from a directory. When that happens, the parent directory is also deleted, and the missing dir is what is generating the python exception. Need to add logic to cover that corner case and avoid exception. The edk2-pytools owners are working on it now. Mike From: Andrew Fish Sent: Friday, June 12, 2020 11:43 AM To: devel@edk2.groups.io; Kinney, Michael D Cc: Ard Biesheuvel ; Sean Brogan ; lersek@redhat.com; Bret Barkelew Subject: Re: [edk2-devel] ci problems again Mike, Is this an issue of assuming a patch vs. a patch set? Are there other places we could run into this? Thanks, Andrew Fish On Jun 12, 2020, at 11:19 AM, Michael D Kinney > wrote: Hi Ard, I think this is related to a patch series that deletes some files. The following code in the edk2-pytool-extensions is collecting the files that change in a pr to determine the components that need to be built. https://github.com/tianocore/edk2-pytool-extensions/blob/e036937666c797d6d1aa4a8285088d3eae128d6c/edk2toolext/invocables/edk2_pr_eval.py#L271 It does not filter out files that have been deleted. I tried the following change that filters out files that have been deleted and reverses the diff to be from the base branch to HEAD so the files deleted filter works as expected. The stuart command completes without errors. diff --git a/edk2toolext/invocables/edk2_pr_eval.py b/edk2toolext/invocables/edk2_pr_eval.py index 00ac282..a3f55b7 100644 --- a/edk2toolext/invocables/edk2_pr_eval.py +++ b/edk2toolext/invocables/edk2_pr_eval.py @@ -268,7 +268,7 @@ def _get_files_that_changed_in_this_pr(self, base_branch) -> tuple: # get file differences between pr and base output = StringIO() - cmd_params = f"diff --name-only HEAD..{base_branch}" + cmd_params = f"diff --name-only --diff-filter=d {base_branch}..HEAD" rc = RunCmd("git", cmd_params, outstream=output) if(rc == 0): I do not know if this change has other unintended side effects. I need edk2-pytools-extensions to provide input. Mike -----Original Message----- From: Kinney, Michael D > Sent: Friday, June 12, 2020 9:49 AM To: Ard Biesheuvel >; devel@edk2.groups.io; Sean Brogan >; lersek@redhat.com; Kinney, Michael D > Subject: RE: [edk2-devel] ci problems again Hi Ard, The general instructions to setup env are in the repo here: https://github.com/tianocore/edk2/tree/master/.pytool I did this and pulled PR 689 and ran the command from the Azure pipelines log and reproduced the same failure on my laptop. Here is the log from my laptop. I will debug a bit more to see what the issue is. This command passes with edk2/master. It fails with my pr_689 branch. (env) c:\work\GitHub\tianocore\edk2>git checkout pr_689 Switched to branch 'pr_689' (env) c:\work\GitHub\tianocore\edk2>stuart_pr_eval -c OvmfPkg/PlatformCI/PlatformBuild.py -t DEBUG -a IA32 -- pr-target origin/master SECTION - Init SDE SECTION - Loading Plugins SECTION - Start Invocable Tool ERROR - Failed to get package for file ArmVirtPkg/Include/Platform/Hidden.h. Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ \Platform' ERROR - Failed to get package for file ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S cripts' ERROR - Failed to get package for file ArmVirtPkg/Include/Platform/Hidden.h. Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ \Platform' ERROR - Failed to get package for file ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. Exception [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\PrePi\\S cripts' Traceback (most recent call last): File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "c:\work\GitHub\tianocore\edk2\env\Scripts\stuart_pr_eva l.exe\__main__.py", line 7, in File "c:\work\github\tianocore\edk2\env\lib\site- packages\edk2toolext\invocables\edk2_pr_eval.py", line 389, in main Edk2PrEval().Invoke() File "c:\work\github\tianocore\edk2\env\lib\site- packages\edk2toolext\base_abstract_invocable.py", line 141, in Invoke retcode = self.Go() File "c:\work\github\tianocore\edk2\env\lib\site- packages\edk2toolext\invocables\edk2_pr_eval.py", line 82, in Go actualPackagesDict = self.get_packages_to_build(self.requested_package_list) File "c:\work\github\tianocore\edk2\env\lib\site- packages\edk2toolext\invocables\edk2_pr_eval.py", line 202, in get_packages_to_build changed_modules = self._get_unique_module_infs_changed(files) File "c:\work\github\tianocore\edk2\env\lib\site- packages\edk2toolext\invocables\edk2_pr_eval.py", line 235, in _get_unique_module_infs_changed infs = self.edk2_path_obj.GetContainingModules(os.path.abspath( f)) File "c:\work\github\tianocore\edk2\env\lib\site- packages\edk2toollib\uefi\edk2\path_utilities.py", line 211, in GetContainingModules for f in os.listdir(dirpath): FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\work\\GitHub\\tianocore\\edk2\\ArmVirtPkg\\Include\ \Platform' Best regards, Mike -----Original Message----- From: Ard Biesheuvel > Sent: Friday, June 12, 2020 9:16 AM To: Kinney, Michael D >; devel@edk2.groups.io; Sean Brogan >; lersek@redhat.com Subject: Re: [edk2-devel] ci problems again On 6/12/20 6:12 PM, Kinney, Michael D wrote: Hi Ard, The CI agents are stateless, so it can not be stale content. So why is it looking for ArmVirtPkg/PrePi/Scripts/PrePi- PIE.lds even though not a single reference to it remains anywhere in the tree? (grep'ing for PrePi-PIE.lds in the edk2 tree gives 0 hits) Have you tried the pytools command that is being used locally for this patch series? No. Are there any instructions on how to use that? -----Original Message----- From: devel@edk2.groups.io > On Behalf Of Ard Biesheuvel Sent: Friday, June 12, 2020 8:47 AM To: Kinney, Michael D >; devel@edk2.groups.io; Sean Brogan >; lersek@redhat.com Subject: Re: [edk2-devel] ci problems again On 6/12/20 5:36 PM, Kinney, Michael D wrote: Hi Ard, Please look at this more detailed log from Azure. https://dev.azure.com/tianocore/edk2- ci/_build/results?buildId=8170&view=logs&jobId=680b79ca- 2ce9-56d4-127b-ee6e95685750&j=680b79ca-2ce9-56d4- 127b- ee6e95685750&t=1e12dbf9-b3ef-52bb-1019-4c2f443aef72 It says it can not find some paths in ArmVirtPkg. It is complaining about files that are being removed by the series: Failed to get package for file ArmVirtPkg/Include/Platform/Hidden.h. Failed to get package for file ArmVirtPkg/PrePi/Scripts/PrePi-PIE.lds. but I cannot find any remaining references to those files in anywhere in the tree, so it seems the CI has remembered the existence of those files in some way, and is now expecting them to be still there. So I don't think there is any way I can fix this myself. Someone needs to wipe the build directory for these CI jobs.