Re: [Patch] BaseTools: Enable component override functionality

["Gao, Liming" <liming.gao@intel.com>]

Ard:
  Could you let me know which platform do you use? We will verify it for the base tools change. 

Thanks
Liming
>-----Original Message-----
>From: Feng, Bob C
>Sent: Monday, January 14, 2019 3:19 PM
>To: Ard Biesheuvel <ard.biesheuvel@linaro.org>; Gao, Liming
><liming.gao@intel.com>
>Cc: edk2-devel@lists.01.org; Carsey, Jaben <jaben.carsey@intel.com>
>Subject: RE: [edk2] [Patch] BaseTools: Enable component override
>functionality
>
>Sorry for break your build.
>
>Yes. I'll double test the code before pushing it next time.
>
>-Bob
>
>-----Original Message-----
>From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>Sent: Monday, January 14, 2019 3:11 PM
>To: Gao, Liming <liming.gao@intel.com>
>Cc: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org; Carsey,
>Jaben <jaben.carsey@intel.com>
>Subject: Re: [edk2] [Patch] BaseTools: Enable component override
>functionality
>
>On Mon, 14 Jan 2019 at 08:01, Gao, Liming <liming.gao@intel.com> wrote:
>>
>> Ard:
>>   Sorry for this break, Bob just sent hot fix for this issue in
>https://lists.01.org/pipermail/edk2-devel/2019-January/034947.html. Could
>you verify it?
>>   In fact, it is caused by previous patch to remove EDK support in BaseTools
>Python code.
>>
>
>OK, that patch fixes it for me.
>
>But may I kindly suggest that Bob tests his code before pushing it? Thanks.
>
>
>> >-----Original Message-----
>> >From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>> >Sent: Monday, January 14, 2019 2:58 PM
>> >To: Gao, Liming <liming.gao@intel.com>
>> >Cc: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org;
>> >Carsey, Jaben <jaben.carsey@intel.com>
>> >Subject: Re: [edk2] [Patch] BaseTools: Enable component override
>> >functionality
>> >
>> >On Mon, 14 Jan 2019 at 04:00, Gao, Liming <liming.gao@intel.com> wrote:
>> >>
>> >> Reviewed-by: Liming Gao <liming.gao@intel.com>
>> >>
>> >
>> >This patch breaks the build on AArch64/GGC:
>> >
>> >(Python 2.7.13 on linux2) Traceback (most recent call last):
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Pytho
>n/
>> >b
>> >uild/build.py",>
>> >line 2403, in Main
>> >    MyBuild.Launch()
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Pytho
>n/
>> >b
>> >uild/build.py",>
>> >line 2137, in Launch
>> >    self._MultiThreadBuildPlatform()
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Pytho
>n/
>> >b
>> >uild/build.py",>
>> >line 1917, in _MultiThreadBuildPlatform
>> >    self.Progress
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py",>
>> >line 239, in __init__
>> >    self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,
>> >*args, **kwargs)
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py",>
>> >line 403, in _InitWorker
>> >    PlatformPcds = Platform.Pcds
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p
>y",>
>> >line 1194, in Pcds
>> >    self._Pcds = self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST,
>> >self._Pcds)
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p
>y",>
>> >line 1486, in UpdateStructuredPcds
>> >    for Pcd in self.DecPcds:
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p
>y",>
>> >line 3183, in DecPcds
>> >    self._DecPcds, self._GuidDict = GetDeclaredPcd(self, self._Bdb,
>> >self._Arch, self._Target, self._Toolchain, PkgSet)
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/WorkspaceCo
>mm
>> >on.py",>
>> >line 62, in GetDeclaredPcd
>> >    PkgList = GetPackageList(Platform, BuildDatabase, Arch, Target,
>> >Toolchain)
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/WorkspaceCo
>mm
>> >on.py",>
>> >line 44, in GetPackageList
>> >    for ModuleFile in Platform.Modules:
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p
>y",>
>> >line 728, in Modules
>> >    self.OverrideDuplicateModule()
>> >  File "<https://ci.linaro.org/job/leg-virt-tianocore-edk2-
>> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p
>y",>
>> >line 712, in OverrideDuplicateModule
>> >    Macros["EDK_SOURCE"] = GlobalData.gEcpSource
>> >AttributeError: 'module' object has no attribute 'gEcpSource'
>> >
>> >
>> >
>> >> >-----Original Message-----
>> >> >From: Feng, Bob C
>> >> >Sent: Friday, January 11, 2019 10:39 AM
>> >> >To: edk2-devel@lists.01.org
>> >> >Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
>> >> ><liming.gao@intel.com>; Carsey, Jaben <jaben.carsey@intel.com>
>> >> >Subject: [Patch] BaseTools: Enable component override
>> >> >functionality
>> >> >
>> >> >https://bugzilla.tianocore.org/show_bug.cgi?id=1449
>> >> >This patch enable build tools to recognize that when two given
>> >> >files have the same GUID, file path and ARCH in Dsc, The later
>> >> >one's definition will be used.
>> >> >
>> >> >Contributed-under: TianoCore Contribution Agreement 1.1
>> >> >Signed-off-by: Bob Feng <bob.c.feng@intel.com>
>> >> >Cc: Liming Gao <liming.gao@intel.com>
>> >> >Cc: Carsey Jaben <jaben.carsey@intel.com>
>> >> >---
>> >> > .../Source/Python/Workspace/DscBuildData.py   | 24 ++++++++++++---
>---
>> >-
>> >> > .../Source/Python/Workspace/MetaFileParser.py |  5 ++++
>> >> > .../Source/Python/Workspace/MetaFileTable.py  |  7 ++++--
>> >> > 3 files changed, 25 insertions(+), 11 deletions(-)
>> >> >
>> >> >diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py
>> >> >b/BaseTools/Source/Python/Workspace/DscBuildData.py
>> >> >index 7e82e8e934..f9805f58f5 100644
>> >> >--- a/BaseTools/Source/Python/Workspace/DscBuildData.py
>> >> >+++ b/BaseTools/Source/Python/Workspace/DscBuildData.py
>> >> >@@ -704,36 +704,44 @@ class DscBuildData(PlatformBuildClassObject):
>> >> >             if TAB_DEFAULT_STORES_DEFAULT not in self.DefaultStores:
>> >> >                 self.DefaultStores[TAB_DEFAULT_STORES_DEFAULT] =
>> >> >(0,
>> >> >TAB_DEFAULT_STORES_DEFAULT)
>> >> >             GlobalData.gDefaultStores = sorted(self.DefaultStores.keys())
>> >> >         return self.DefaultStores
>> >> >
>> >> >+    def OverrideDuplicateModule(self):
>> >> >+        RecordList = self._RawData[MODEL_META_DATA_COMPONENT,
>> >> >self._Arch]
>> >> >+        Macros = self._Macros
>> >> >+        Macros["EDK_SOURCE"] = GlobalData.gEcpSource
>> >> >+        Components = {}
>> >> >+        for Record in RecordList:
>> >> >+            ModuleId = Record[6]
>> >> >+            file_guid = self._RawData[MODEL_META_DATA_HEADER,
>> >self._Arch,
>> >> >None, ModuleId]
>> >> >+            file_guid_str = file_guid[0][2] if file_guid else "NULL"
>> >> >+            ModuleFile = PathClass(NormPath(Record[0], Macros),
>> >> >GlobalData.gWorkspace, Arch=self._Arch)
>> >> >+            if self._Arch != TAB_ARCH_COMMON and
>> >> >(file_guid_str,str(ModuleFile)) in Components:
>> >> >+
>> >> >self._RawData.DisableOverrideComponent(Components[(file_guid_str,
>s
>> >> >tr
>> >(M
>> >> >oduleFile))])
>> >> >+            Components[(file_guid_str,str(ModuleFile))] = ModuleId
>> >> >+        self._RawData._PostProcessed = False
>> >> >     ## Retrieve [Components] section information
>> >> >     @property
>> >> >     def Modules(self):
>> >> >         if self._Modules is not None:
>> >> >             return self._Modules
>> >> >-
>> >> >+        self.OverrideDuplicateModule()
>> >> >         self._Modules = OrderedDict()
>> >> >         RecordList = self._RawData[MODEL_META_DATA_COMPONENT,
>> >> >self._Arch]
>> >> >         Macros = self._Macros
>> >> >         Macros["EDK_SOURCE"] = GlobalData.gEcpSource
>> >> >         for Record in RecordList:
>> >> >-            DuplicatedFile = False
>> >> >-
>> >> >             ModuleFile = PathClass(NormPath(Record[0], Macros),
>> >> >GlobalData.gWorkspace, Arch=self._Arch)
>> >> >             ModuleId = Record[6]
>> >> >             LineNo = Record[7]
>> >> >
>> >> >             # check the file validation
>> >> >             ErrorCode, ErrorInfo = ModuleFile.Validate('.inf')
>> >> >             if ErrorCode != 0:
>> >> >                 EdkLogger.error('build', ErrorCode,
>> >> > File=self.MetaFile,
>> >Line=LineNo,
>> >> >                                 ExtraData=ErrorInfo)
>> >> >-            # Check duplication
>> >> >-            # If arch is COMMON, no duplicate module is checked since all
>> >modules
>> >> >in all component sections are selected
>> >> >-            if self._Arch != TAB_ARCH_COMMON and ModuleFile in
>> >self._Modules:
>> >> >-                DuplicatedFile = True
>> >> >
>> >> >             Module = ModuleBuildClassObject()
>> >> >             Module.MetaFile = ModuleFile
>> >> >
>> >> >             # get module private library instance @@ -792,12
>> >> >+800,10 @@ class DscBuildData(PlatformBuildClassObject):
>> >> >                 else:
>> >> >                     OptionString = Module.BuildOptions[ToolChainFamily,
>ToolChain]
>> >> >                     Module.BuildOptions[ToolChainFamily,
>> >> >ToolChain] =
>> >OptionString +
>> >> >" " + Option
>> >> >
>> >> >             RecordList = self._RawData[MODEL_META_DATA_HEADER,
>> >self._Arch,
>> >> >None, ModuleId]
>> >> >-            if DuplicatedFile and not RecordList:
>> >> >-                EdkLogger.error('build', FILE_DUPLICATED, File=self.MetaFile,
>> >> >ExtraData=str(ModuleFile), Line=LineNo)
>> >> >             if RecordList:
>> >> >                 if len(RecordList) != 1:
>> >> >                     EdkLogger.error('build', OPTION_UNKNOWN,
>> >> >'Only FILE_GUID
>> >can
>> >> >be listed in <Defines> section.',
>> >> >                                     File=self.MetaFile,
>> >> >ExtraData=str(ModuleFile),
>> >Line=LineNo)
>> >> >                 ModuleFile = ProcessDuplicatedInf(ModuleFile,
>> >> >RecordList[0][2],
>> >> >GlobalData.gWorkspace)
>> >> >diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py
>> >> >b/BaseTools/Source/Python/Workspace/MetaFileParser.py
>> >> >index 032220813b..a52e9229df 100644
>> >> >--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
>> >> >+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
>> >> >@@ -1711,10 +1711,15 @@ class DscParser(MetaFileParser):
>> >> >
>> >> >     def __ProcessBuildOption(self):
>> >> >         self._ValueList = [ReplaceMacro(Value, self._Macros,
>RaiseError=False)
>> >> >                            for Value in self._ValueList]
>> >> >
>> >> >+    def DisableOverrideComponent(self,module_id):
>> >> >+        for ori_id in self._IdMapping:
>> >> >+            if self._IdMapping[ori_id] == module_id:
>> >> >+                self._RawTable.DisableComponent(ori_id)
>> >> >+
>> >> >     _SectionParser = {
>> >> >         MODEL_META_DATA_HEADER                          :   _DefineParser,
>> >> >         MODEL_EFI_SKU_ID                                :   _SkuIdParser,
>> >> >         MODEL_EFI_DEFAULT_STORES                        :   _DefaultStoresParser,
>> >> >         MODEL_EFI_LIBRARY_INSTANCE                      :
>_LibraryInstanceParser,
>> >> >diff --git a/BaseTools/Source/Python/Workspace/MetaFileTable.py
>> >> >b/BaseTools/Source/Python/Workspace/MetaFileTable.py
>> >> >index 004e9494c3..823a87e057 100644
>> >> >--- a/BaseTools/Source/Python/Workspace/MetaFileTable.py
>> >> >+++ b/BaseTools/Source/Python/Workspace/MetaFileTable.py
>> >> >@@ -74,11 +74,11 @@ class MetaFileTable():
>> >> >
>> >> >     def SetEndFlag(self):
>> >> >         self.CurrentContent.append(self._DUMMY_)
>> >> >
>> >> >     def GetAll(self):
>> >> >-        return [item for item in self.CurrentContent if item[0] >= 0 ]
>> >> >+        return [item for item in self.CurrentContent if item[0]
>> >> >+ >= 0 and item[-
>> >> >1]>=0]
>> >> >
>> >> > ## Python class representation of table storing module data
>> >> >class ModuleTable(MetaFileTable):
>> >> >     _COLUMN_ = '''
>> >> >         ID REAL PRIMARY KEY,
>> >> >@@ -371,11 +371,10 @@ class PlatformTable(MetaFileTable):
>> >> >     #
>> >> >     def Query(self, Model, Scope1=None, Scope2=None,
>> >> >BelongsToItem=None, FromItem=None):
>> >> >
>> >> >         QueryTab = self.CurrentContent
>> >> >         result = [item for item in QueryTab if item[1] == Model
>> >> >and item[-1]>0 ]
>> >> >-
>> >> >         if Scope1 is not None and Scope1 != TAB_ARCH_COMMON:
>> >> >             Sc1 = set(['COMMON'])
>> >> >             Sc1.add(Scope1)
>> >> >             result = [item for item in result if item[5] in Sc1]
>> >> >         Sc2 = set( ['COMMON','DEFAULT']) @@ -395,10 +394,14 @@
>> >> >class PlatformTable(MetaFileTable):
>> >> >             result = [item for item in result if item[9] ==
>> >> >FromItem]
>> >> >
>> >> >         result = [ [r[2],r[3],r[4],r[5],r[6],r[7],r[0],r[9]] for r in result ]
>> >> >         return result
>> >> >
>> >> >+    def DisableComponent(self,comp_id):
>> >> >+        for item in self.CurrentContent:
>> >> >+            if item[0] == comp_id or item[8] == comp_id:
>> >> >+                item[-1] = -1
>> >> >
>> >> > ## Factory class to produce different storage for different type
>> >> > of meta-
>> >file
>> >> > class MetaFileStorage(object):
>> >> >     _FILE_TABLE_ = {
>> >> >         MODEL_FILE_INF      :   ModuleTable,
>> >> >--
>> >> >2.19.1.windows.1
>> >>
>> >> _______________________________________________
>> >> edk2-devel mailing list
>> >> edk2-devel@lists.01.org
>> >> https://lists.01.org/mailman/listinfo/edk2-devel