* [PATCH] BaseTools: Fix --hash Package and Module hash value. [not found] <20180509073836.6924-1-derek.lin2@hpe.com> @ 2018-05-09 7:49 ` Lin, Derek (HPS UEFI Dev) 2018-05-09 8:35 ` Zhu, Yonghong 0 siblings, 1 reply; 7+ messages in thread From: Lin, Derek (HPS UEFI Dev) @ 2018-05-09 7:49 UTC (permalink / raw) To: edk2-devel@lists.01.org; +Cc: yonghong.zhu@intel.com, Lin, Derek (HPS UEFI Dev) From: Lin, Derek (HPS UEFI Dev) Sent: Wednesday, May 9, 2018 3:39 PM To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com> Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. The order of List enumeration is arbitrary. Need to be sorted while calculating Package/Module hash, otherwise it generate different hash value even nothing changes. --- BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index 54f6b1f173..90704dcae4 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -2,6 +2,7 @@ # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): return True def _GenPkgLevelHash(self, Pkg): + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: + return + PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) CreateDirectory(PkgDir) HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ -681,17 +685,16 @@ class WorkspaceAutoGen(AutoGen): m.update(Content) # Get include files hash value if Pkg.Includes: - for inc in Pkg.Includes: + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): for Root, Dirs, Files in os.walk(str(inc)): - for File in Files: + for File in sorted(Files): File_Path = os.path.join(Root, File) f = open(File_Path, 'r') Content = f.read() f.close() m.update(Content) SaveFileOnChange(HashFile, m.hexdigest(), True) - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest() + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = + m.hexdigest() def _GetMetaFiles(self, Target, Toolchain, Arch): AllWorkSpaceMetaFiles = set() @@ -4432,13 +4435,13 @@ class ModuleAutoGen(AutoGen): m.update(GlobalData.gPlatformHash) # Add Package level hash if self.DependentPackageList: - for Pkg in self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName): if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) # Add Library hash if self.LibraryAutoGenList: - for Lib in self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): if Lib.Name not in GlobalData.gModuleHash[self.Arch]: Lib.GenModuleHash() m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) # Add Module's source files if self.SourceFileList: - for File in self.SourceFileList: + for File in sorted(self.SourceFileList, key=lambda x: str(x)): f = open(str(File), 'r') Content = f.read() f.close() -- 2.15.1.windows.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] BaseTools: Fix --hash Package and Module hash value. 2018-05-09 7:49 ` [PATCH] BaseTools: Fix --hash Package and Module hash value Lin, Derek (HPS UEFI Dev) @ 2018-05-09 8:35 ` Zhu, Yonghong 2018-05-09 9:04 ` Lin, Derek (HPS UEFI Dev) 0 siblings, 1 reply; 7+ messages in thread From: Zhu, Yonghong @ 2018-05-09 8:35 UTC (permalink / raw) To: Lin, Derek (HPS UEFI Dev), edk2-devel@lists.01.org Hi Derek, Please refer below link to add Contributed-under and Signed-off-by info. We can use BaseTools\Scripts\PatchCheck.py to check the patch format. Thanks. https://github.com/tianocore/tianocore.github.io/wiki/Commit-Message-Format Best Regards, Zhu Yonghong -----Original Message----- From: Lin, Derek (HPS UEFI Dev) [mailto:derek.lin2@hpe.com] Sent: Wednesday, May 09, 2018 3:49 PM To: edk2-devel@lists.01.org Cc: Zhu, Yonghong <yonghong.zhu@intel.com>; Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com> Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. From: Lin, Derek (HPS UEFI Dev) Sent: Wednesday, May 9, 2018 3:39 PM To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com> Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. The order of List enumeration is arbitrary. Need to be sorted while calculating Package/Module hash, otherwise it generate different hash value even nothing changes. --- BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index 54f6b1f173..90704dcae4 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -2,6 +2,7 @@ # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): return True def _GenPkgLevelHash(self, Pkg): + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: + return + PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) CreateDirectory(PkgDir) HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ -681,17 +685,16 @@ class WorkspaceAutoGen(AutoGen): m.update(Content) # Get include files hash value if Pkg.Includes: - for inc in Pkg.Includes: + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): for Root, Dirs, Files in os.walk(str(inc)): - for File in Files: + for File in sorted(Files): File_Path = os.path.join(Root, File) f = open(File_Path, 'r') Content = f.read() f.close() m.update(Content) SaveFileOnChange(HashFile, m.hexdigest(), True) - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest() + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = + m.hexdigest() def _GetMetaFiles(self, Target, Toolchain, Arch): AllWorkSpaceMetaFiles = set() @@ -4432,13 +4435,13 @@ class ModuleAutoGen(AutoGen): m.update(GlobalData.gPlatformHash) # Add Package level hash if self.DependentPackageList: - for Pkg in self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName): if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) # Add Library hash if self.LibraryAutoGenList: - for Lib in self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): if Lib.Name not in GlobalData.gModuleHash[self.Arch]: Lib.GenModuleHash() m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) # Add Module's source files if self.SourceFileList: - for File in self.SourceFileList: + for File in sorted(self.SourceFileList, key=lambda x: str(x)): f = open(str(File), 'r') Content = f.read() f.close() -- 2.15.1.windows.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] BaseTools: Fix --hash Package and Module hash value. 2018-05-09 8:35 ` Zhu, Yonghong @ 2018-05-09 9:04 ` Lin, Derek (HPS UEFI Dev) 0 siblings, 0 replies; 7+ messages in thread From: Lin, Derek (HPS UEFI Dev) @ 2018-05-09 9:04 UTC (permalink / raw) To: Zhu, Yonghong, edk2-devel@lists.01.org Hi Yonghong, Thanks for notify about this. I resend the patch. Thanks, Derek -----Original Message----- From: Zhu, Yonghong [mailto:yonghong.zhu@intel.com] Sent: Wednesday, May 9, 2018 4:35 PM To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com>; edk2-devel@lists.01.org Cc: Zhu, Yonghong <yonghong.zhu@intel.com> Subject: RE: [PATCH] BaseTools: Fix --hash Package and Module hash value. Hi Derek, Please refer below link to add Contributed-under and Signed-off-by info. We can use BaseTools\Scripts\PatchCheck.py to check the patch format. Thanks. https://github.com/tianocore/tianocore.github.io/wiki/Commit-Message-Format Best Regards, Zhu Yonghong -----Original Message----- From: Lin, Derek (HPS UEFI Dev) [mailto:derek.lin2@hpe.com] Sent: Wednesday, May 09, 2018 3:49 PM To: edk2-devel@lists.01.org Cc: Zhu, Yonghong <yonghong.zhu@intel.com>; Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com> Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. From: Lin, Derek (HPS UEFI Dev) Sent: Wednesday, May 9, 2018 3:39 PM To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com> Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. The order of List enumeration is arbitrary. Need to be sorted while calculating Package/Module hash, otherwise it generate different hash value even nothing changes. --- BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index 54f6b1f173..90704dcae4 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -2,6 +2,7 @@ # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): return True def _GenPkgLevelHash(self, Pkg): + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: + return + PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) CreateDirectory(PkgDir) HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ -681,17 +685,16 @@ class WorkspaceAutoGen(AutoGen): m.update(Content) # Get include files hash value if Pkg.Includes: - for inc in Pkg.Includes: + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): for Root, Dirs, Files in os.walk(str(inc)): - for File in Files: + for File in sorted(Files): File_Path = os.path.join(Root, File) f = open(File_Path, 'r') Content = f.read() f.close() m.update(Content) SaveFileOnChange(HashFile, m.hexdigest(), True) - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest() + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = + m.hexdigest() def _GetMetaFiles(self, Target, Toolchain, Arch): AllWorkSpaceMetaFiles = set() @@ -4432,13 +4435,13 @@ class ModuleAutoGen(AutoGen): m.update(GlobalData.gPlatformHash) # Add Package level hash if self.DependentPackageList: - for Pkg in self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName): if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) # Add Library hash if self.LibraryAutoGenList: - for Lib in self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): if Lib.Name not in GlobalData.gModuleHash[self.Arch]: Lib.GenModuleHash() m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) # Add Module's source files if self.SourceFileList: - for File in self.SourceFileList: + for File in sorted(self.SourceFileList, key=lambda x: str(x)): f = open(str(File), 'r') Content = f.read() f.close() -- 2.15.1.windows.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
[parent not found: <20180509085924.13140-1-derek.lin2@hpe.com>]
* [PATCH] BaseTools: Fix --hash Package and Module hash value. [not found] <20180509085924.13140-1-derek.lin2@hpe.com> @ 2018-05-09 9:03 ` Lin, Derek (HPS UEFI Dev) 2018-05-16 7:12 ` Zhu, Yonghong 0 siblings, 1 reply; 7+ messages in thread From: Lin, Derek (HPS UEFI Dev) @ 2018-05-09 9:03 UTC (permalink / raw) To: edk2-devel@lists.01.org; +Cc: yonghong.zhu@intel.com, Lin, Derek (HPS UEFI Dev) The order of List enumeration is arbitrary. Need to be sorted while calculating Package/Module hash, otherwise it generate different hash value even nothing changes. Contributed-under: TianoCore Contribution Agreement 1.1 Signed-off-by: Derek Lin <derek.lin2@hpe.com> --- BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index 54f6b1f173..90704dcae4 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -2,6 +2,7 @@ # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): return True def _GenPkgLevelHash(self, Pkg): + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: + return + PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) CreateDirectory(PkgDir) HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ -681,17 +685,16 @@ class WorkspaceAutoGen(AutoGen): m.update(Content) # Get include files hash value if Pkg.Includes: - for inc in Pkg.Includes: + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): for Root, Dirs, Files in os.walk(str(inc)): - for File in Files: + for File in sorted(Files): File_Path = os.path.join(Root, File) f = open(File_Path, 'r') Content = f.read() f.close() m.update(Content) SaveFileOnChange(HashFile, m.hexdigest(), True) - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest() + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = + m.hexdigest() def _GetMetaFiles(self, Target, Toolchain, Arch): AllWorkSpaceMetaFiles = set() @@ -4432,13 +4435,13 @@ class ModuleAutoGen(AutoGen): m.update(GlobalData.gPlatformHash) # Add Package level hash if self.DependentPackageList: - for Pkg in self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName): if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) # Add Library hash if self.LibraryAutoGenList: - for Lib in self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): if Lib.Name not in GlobalData.gModuleHash[self.Arch]: Lib.GenModuleHash() m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) # Add Module's source files if self.SourceFileList: - for File in self.SourceFileList: + for File in sorted(self.SourceFileList, key=lambda x: str(x)): f = open(str(File), 'r') Content = f.read() f.close() -- 2.15.1.windows.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] BaseTools: Fix --hash Package and Module hash value. 2018-05-09 9:03 ` Lin, Derek (HPS UEFI Dev) @ 2018-05-16 7:12 ` Zhu, Yonghong 2018-05-16 16:20 ` Carsey, Jaben 0 siblings, 1 reply; 7+ messages in thread From: Zhu, Yonghong @ 2018-05-16 7:12 UTC (permalink / raw) To: Lin, Derek (HPS UEFI Dev), edk2-devel@lists.01.org Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com> Best Regards, Zhu Yonghong -----Original Message----- From: Lin, Derek (HPS UEFI Dev) [mailto:derek.lin2@hpe.com] Sent: Wednesday, May 09, 2018 5:03 PM To: edk2-devel@lists.01.org Cc: Zhu, Yonghong <yonghong.zhu@intel.com>; Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com> Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. The order of List enumeration is arbitrary. Need to be sorted while calculating Package/Module hash, otherwise it generate different hash value even nothing changes. Contributed-under: TianoCore Contribution Agreement 1.1 Signed-off-by: Derek Lin <derek.lin2@hpe.com> --- BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py index 54f6b1f173..90704dcae4 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -2,6 +2,7 @@ # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> # This program and the accompanying materials # are licensed and made available under the terms and conditions of the BSD License # which accompanies this distribution. The full text of the license may be found at @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): return True def _GenPkgLevelHash(self, Pkg): + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: + return + PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) CreateDirectory(PkgDir) HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ -681,17 +685,16 @@ class WorkspaceAutoGen(AutoGen): m.update(Content) # Get include files hash value if Pkg.Includes: - for inc in Pkg.Includes: + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): for Root, Dirs, Files in os.walk(str(inc)): - for File in Files: + for File in sorted(Files): File_Path = os.path.join(Root, File) f = open(File_Path, 'r') Content = f.read() f.close() m.update(Content) SaveFileOnChange(HashFile, m.hexdigest(), True) - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest() + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = + m.hexdigest() def _GetMetaFiles(self, Target, Toolchain, Arch): AllWorkSpaceMetaFiles = set() @@ -4432,13 +4435,13 @@ class ModuleAutoGen(AutoGen): m.update(GlobalData.gPlatformHash) # Add Package level hash if self.DependentPackageList: - for Pkg in self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName): if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) # Add Library hash if self.LibraryAutoGenList: - for Lib in self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): if Lib.Name not in GlobalData.gModuleHash[self.Arch]: Lib.GenModuleHash() m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) # Add Module's source files if self.SourceFileList: - for File in self.SourceFileList: + for File in sorted(self.SourceFileList, key=lambda x: str(x)): f = open(str(File), 'r') Content = f.read() f.close() -- 2.15.1.windows.2 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] BaseTools: Fix --hash Package and Module hash value. 2018-05-16 7:12 ` Zhu, Yonghong @ 2018-05-16 16:20 ` Carsey, Jaben 2018-05-17 2:14 ` Lin, Derek (HPS UEFI Dev) 0 siblings, 1 reply; 7+ messages in thread From: Carsey, Jaben @ 2018-05-16 16:20 UTC (permalink / raw) To: Zhu, Yonghong, Lin, Derek (HPS UEFI Dev), edk2-devel@lists.01.org Derek, The change is good. I have a comment/question on the message. Reviewed-by: Jaben Carsey <Jaben.carsey@intel.com> Order of list enumeration is not arbitrary, it's based on python lists being inherently insertion-ordered objects. If insertion order is irrelevant, should we consider changing those to a different type? (set?) -Jaben > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > Zhu, Yonghong > Sent: Wednesday, May 16, 2018 12:12 AM > To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com>; edk2- > devel@lists.01.org > Subject: Re: [edk2] [PATCH] BaseTools: Fix --hash Package and Module hash > value. > > Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com> > > Best Regards, > Zhu Yonghong > > > -----Original Message----- > From: Lin, Derek (HPS UEFI Dev) [mailto:derek.lin2@hpe.com] > Sent: Wednesday, May 09, 2018 5:03 PM > To: edk2-devel@lists.01.org > Cc: Zhu, Yonghong <yonghong.zhu@intel.com>; Lin, Derek (HPS UEFI Dev) > <derek.lin2@hpe.com> > Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. > > The order of List enumeration is arbitrary. > Need to be sorted while calculating Package/Module hash, otherwise it > generate different hash value even nothing changes. > > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Derek Lin <derek.lin2@hpe.com> > --- > BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- > 1 file changed, 10 insertions(+), 7 deletions(-) > > diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py > b/BaseTools/Source/Python/AutoGen/AutoGen.py > index 54f6b1f173..90704dcae4 100644 > --- a/BaseTools/Source/Python/AutoGen/AutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py > @@ -2,6 +2,7 @@ > # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) 2007 > - 2018, Intel Corporation. All rights reserved.<BR> > +# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> > # This program and the accompanying materials # are licensed and made > available under the terms and conditions of the BSD License # which > accompanies this distribution. The full text of the license may be found at > @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): > return True > > def _GenPkgLevelHash(self, Pkg): > + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: > + return > + > PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) > CreateDirectory(PkgDir) > HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ -681,17 > +685,16 @@ class WorkspaceAutoGen(AutoGen): > m.update(Content) > # Get include files hash value > if Pkg.Includes: > - for inc in Pkg.Includes: > + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): > for Root, Dirs, Files in os.walk(str(inc)): > - for File in Files: > + for File in sorted(Files): > File_Path = os.path.join(Root, File) > f = open(File_Path, 'r') > Content = f.read() > f.close() > m.update(Content) > SaveFileOnChange(HashFile, m.hexdigest(), True) > - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: > - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = > m.hexdigest() > + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = > + m.hexdigest() > > def _GetMetaFiles(self, Target, Toolchain, Arch): > AllWorkSpaceMetaFiles = set() > @@ -4432,13 +4435,13 @@ class ModuleAutoGen(AutoGen): > m.update(GlobalData.gPlatformHash) > # Add Package level hash > if self.DependentPackageList: > - for Pkg in self.DependentPackageList: > + for Pkg in sorted(self.DependentPackageList, key=lambda x: > x.PackageName): > if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: > > m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) > > # Add Library hash > if self.LibraryAutoGenList: > - for Lib in self.LibraryAutoGenList: > + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): > if Lib.Name not in GlobalData.gModuleHash[self.Arch]: > Lib.GenModuleHash() > m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) > @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): > m.update(Content) > # Add Module's source files > if self.SourceFileList: > - for File in self.SourceFileList: > + for File in sorted(self.SourceFileList, key=lambda x: str(x)): > f = open(str(File), 'r') > Content = f.read() > f.close() > -- > 2.15.1.windows.2 > > > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] BaseTools: Fix --hash Package and Module hash value. 2018-05-16 16:20 ` Carsey, Jaben @ 2018-05-17 2:14 ` Lin, Derek (HPS UEFI Dev) 0 siblings, 0 replies; 7+ messages in thread From: Lin, Derek (HPS UEFI Dev) @ 2018-05-17 2:14 UTC (permalink / raw) To: Carsey, Jaben, Zhu, Yonghong, edk2-devel@lists.01.org Jaben, You're right, Python list is ordered. So it's because of insertion order, hmm.. but the meta-file parser run in single thread, not sure why it generate different order in different build...never mind. However, I like the idea that changing list to set. I should improve performance. Thanks, Derek -----Original Message----- From: Carsey, Jaben [mailto:jaben.carsey@intel.com] Sent: Thursday, May 17, 2018 12:20 AM To: Zhu, Yonghong <yonghong.zhu@intel.com>; Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com>; edk2-devel@lists.01.org Subject: RE: [PATCH] BaseTools: Fix --hash Package and Module hash value. Derek, The change is good. I have a comment/question on the message. Reviewed-by: Jaben Carsey <Jaben.carsey@intel.com> Order of list enumeration is not arbitrary, it's based on python lists being inherently insertion-ordered objects. If insertion order is irrelevant, should we consider changing those to a different type? (set?) -Jaben > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > Zhu, Yonghong > Sent: Wednesday, May 16, 2018 12:12 AM > To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com>; edk2- > devel@lists.01.org > Subject: Re: [edk2] [PATCH] BaseTools: Fix --hash Package and Module > hash value. > > Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com> > > Best Regards, > Zhu Yonghong > > > -----Original Message----- > From: Lin, Derek (HPS UEFI Dev) [mailto:derek.lin2@hpe.com] > Sent: Wednesday, May 09, 2018 5:03 PM > To: edk2-devel@lists.01.org > Cc: Zhu, Yonghong <yonghong.zhu@intel.com>; Lin, Derek (HPS UEFI Dev) > <derek.lin2@hpe.com> > Subject: [PATCH] BaseTools: Fix --hash Package and Module hash value. > > The order of List enumeration is arbitrary. > Need to be sorted while calculating Package/Module hash, otherwise it > generate different hash value even nothing changes. > > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Derek Lin <derek.lin2@hpe.com> > --- > BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++------- > 1 file changed, 10 insertions(+), 7 deletions(-) > > diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py > b/BaseTools/Source/Python/AutoGen/AutoGen.py > index 54f6b1f173..90704dcae4 100644 > --- a/BaseTools/Source/Python/AutoGen/AutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py > @@ -2,6 +2,7 @@ > # Generate AutoGen.h, AutoGen.c and *.depex files # # Copyright (c) > 2007 > - 2018, Intel Corporation. All rights reserved.<BR> > +# Copyright (c) 2018, Hewlett Packard Enterprise Development, > +L.P.<BR> > # This program and the accompanying materials # are licensed and > made available under the terms and conditions of the BSD License # > which accompanies this distribution. The full text of the license may > be found at @@ -670,6 +671,9 @@ class WorkspaceAutoGen(AutoGen): > return True > > def _GenPkgLevelHash(self, Pkg): > + if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]: > + return > + > PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) > CreateDirectory(PkgDir) > HashFile = os.path.join(PkgDir, Pkg.PackageName + '.hash') @@ > -681,17 > +685,16 @@ class WorkspaceAutoGen(AutoGen): > m.update(Content) > # Get include files hash value > if Pkg.Includes: > - for inc in Pkg.Includes: > + for inc in sorted(Pkg.Includes, key=lambda x: str(x)): > for Root, Dirs, Files in os.walk(str(inc)): > - for File in Files: > + for File in sorted(Files): > File_Path = os.path.join(Root, File) > f = open(File_Path, 'r') > Content = f.read() > f.close() > m.update(Content) > SaveFileOnChange(HashFile, m.hexdigest(), True) > - if Pkg.PackageName not in GlobalData.gPackageHash[Pkg.Arch]: > - GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = > m.hexdigest() > + GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = > + m.hexdigest() > > def _GetMetaFiles(self, Target, Toolchain, Arch): > AllWorkSpaceMetaFiles = set() @@ -4432,13 +4435,13 @@ class > ModuleAutoGen(AutoGen): > m.update(GlobalData.gPlatformHash) > # Add Package level hash > if self.DependentPackageList: > - for Pkg in self.DependentPackageList: > + for Pkg in sorted(self.DependentPackageList, key=lambda x: > x.PackageName): > if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]: > > m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName]) > > # Add Library hash > if self.LibraryAutoGenList: > - for Lib in self.LibraryAutoGenList: > + for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name): > if Lib.Name not in GlobalData.gModuleHash[self.Arch]: > Lib.GenModuleHash() > m.update(GlobalData.gModuleHash[self.Arch][Lib.Name]) > @@ -4450,7 +4453,7 @@ class ModuleAutoGen(AutoGen): > m.update(Content) > # Add Module's source files > if self.SourceFileList: > - for File in self.SourceFileList: > + for File in sorted(self.SourceFileList, key=lambda x: str(x)): > f = open(str(File), 'r') > Content = f.read() > f.close() > -- > 2.15.1.windows.2 > > > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-05-17 2:14 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20180509073836.6924-1-derek.lin2@hpe.com>
2018-05-09 7:49 ` [PATCH] BaseTools: Fix --hash Package and Module hash value Lin, Derek (HPS UEFI Dev)
2018-05-09 8:35 ` Zhu, Yonghong
2018-05-09 9:04 ` Lin, Derek (HPS UEFI Dev)
[not found] <20180509085924.13140-1-derek.lin2@hpe.com>
2018-05-09 9:03 ` Lin, Derek (HPS UEFI Dev)
2018-05-16 7:12 ` Zhu, Yonghong
2018-05-16 16:20 ` Carsey, Jaben
2018-05-17 2:14 ` Lin, Derek (HPS UEFI Dev)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox