From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=2607:f8b0:4864:20::132; helo=mail-it1-x132.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=edk2-devel@lists.01.org Received: from mail-it1-x132.google.com (mail-it1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 16BD2211B5071 for ; Sun, 13 Jan 2019 23:27:18 -0800 (PST) Received: by mail-it1-x132.google.com with SMTP id p197so10973263itp.0 for ; Sun, 13 Jan 2019 23:27:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3SC0ESyp1u+/hvbtvEq3NcqD7WFTJp4btnuirQQMsGw=; b=RA6QGlKhRF4hQ5+JaCCK3+zmoRCSUPK+g1q/LzWVe3UPs2fLtM64iVxiOz84IH7Xmn rtt2TBDpd51K1ehosnEcbaLpXMZyit5Q+3weo3j5W0qFn+WcHE7o0VMWh/rBu3HFeNuk jbrR22kXEN4TGPd9WLtZ8Ixc5yC0Eyo8+tY+4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3SC0ESyp1u+/hvbtvEq3NcqD7WFTJp4btnuirQQMsGw=; b=sm/cM85EeQucukrDxZ0Am7FrVlUWaLOhGV3YDb9A+pOjcuqhFRX/gHvSDaUL+7nz9J U+sEMB03yLHAC+e7XG7snodf4UR5xyE1HL97yHao1DlwVI+iiMGjpHiwJtWiU8w+8TZQ 1ajzo/sfdDg5Ua07vLL62Ri5GzJjxzGQARkRXupI0pxFWmqU14knSR0vPhpyS+Yr/sln Y1LAwLJrax4nfPuQtsrTngeQGOsZFM1Bxkf+YdIExLH7BKTJDVVVbRY24k7VAKWXU8PE sfcUb+zeSEBh2DvBcIBxyp54gZyFyEoQ2OrJCYMWErgazH78W5ox0L4pYvZMoUjLry83 Voew== X-Gm-Message-State: AJcUukfwZoOro2cr8rT5NpC9DTJ6oO+J4Jfofz0vxtkwyvDl8T+cDCRj 5yfDzLlEpwa9cjh3Iu0XidTE0w0KdRmHGs+klJUKhg== X-Google-Smtp-Source: ALg8bN5R8A7yblNYUz7enVkn9dYYR7tIO5N/Z2v/1AikHTSAQ1gvw5mnv8lHXUeH+ON8qAErRbkiUsUMuFBY6MuFtoQ= X-Received: by 2002:a05:660c:4b:: with SMTP id p11mr7566834itk.71.1547450837422; Sun, 13 Jan 2019 23:27:17 -0800 (PST) MIME-Version: 1.0 References: <20190111023851.9172-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E3B0FD1@SHSMSX152.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E3B11BA@SHSMSX152.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D16005839F@SHSMSX101.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E3B1219@SHSMSX152.ccr.corp.intel.com> In-Reply-To: <4A89E2EF3DFEDB4C8BFDE51014F606A14E3B1219@SHSMSX152.ccr.corp.intel.com> From: Ard Biesheuvel Date: Mon, 14 Jan 2019 08:27:05 +0100 Message-ID: To: "Gao, Liming" Cc: "Feng, Bob C" , "edk2-devel@lists.01.org" , "Carsey, Jaben" Subject: Re: [Patch] BaseTools: Enable component override functionality X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jan 2019 07:27:19 -0000 Content-Type: text/plain; charset="UTF-8" On Mon, 14 Jan 2019 at 08:22, Gao, Liming wrote: > > Ard: > Could you let me know which platform do you use? We will verify it for the base tools change. > I use a CI job here https://ci.linaro.org/view/leg-ci/job/leg-virt-tianocore-edk2-upstream/ that builds {ArmPkg,ArmPlatformPkg,EmbeddedPkg,ArmVirtQemu,ArmVirtXen,ArmVirtQemuKernel} x {DEBUG,RELEASE,NOOPT} x {GCC48,GCC49,GCC5,CLANG35,CLANG38} and {OvmfIa32,OvmfIa32X64,OvmfX64} x {DEBUG,RELEASE,NOOPT} with GCC5 only. If anyone wants to be added to cc for the failure reports, I can easily add them. Thanks, Ard. > Thanks > Liming > >-----Original Message----- > >From: Feng, Bob C > >Sent: Monday, January 14, 2019 3:19 PM > >To: Ard Biesheuvel ; Gao, Liming > > > >Cc: edk2-devel@lists.01.org; Carsey, Jaben > >Subject: RE: [edk2] [Patch] BaseTools: Enable component override > >functionality > > > >Sorry for break your build. > > > >Yes. I'll double test the code before pushing it next time. > > > >-Bob > > > >-----Original Message----- > >From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org] > >Sent: Monday, January 14, 2019 3:11 PM > >To: Gao, Liming > >Cc: Feng, Bob C ; edk2-devel@lists.01.org; Carsey, > >Jaben > >Subject: Re: [edk2] [Patch] BaseTools: Enable component override > >functionality > > > >On Mon, 14 Jan 2019 at 08:01, Gao, Liming wrote: > >> > >> Ard: > >> Sorry for this break, Bob just sent hot fix for this issue in > >https://lists.01.org/pipermail/edk2-devel/2019-January/034947.html. Could > >you verify it? > >> In fact, it is caused by previous patch to remove EDK support in BaseTools > >Python code. > >> > > > >OK, that patch fixes it for me. > > > >But may I kindly suggest that Bob tests his code before pushing it? Thanks. > > > > > >> >-----Original Message----- > >> >From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org] > >> >Sent: Monday, January 14, 2019 2:58 PM > >> >To: Gao, Liming > >> >Cc: Feng, Bob C ; edk2-devel@lists.01.org; > >> >Carsey, Jaben > >> >Subject: Re: [edk2] [Patch] BaseTools: Enable component override > >> >functionality > >> > > >> >On Mon, 14 Jan 2019 at 04:00, Gao, Liming wrote: > >> >> > >> >> Reviewed-by: Liming Gao > >> >> > >> > > >> >This patch breaks the build on AArch64/GGC: > >> > > >> >(Python 2.7.13 on linux2) Traceback (most recent call last): > >> > File " >> >upstream/ws/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Pytho > >n/ > >> >b > >> >uild/build.py",> > >> >line 2403, in Main > >> > MyBuild.Launch() > >> > File " >> >upstream/ws/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Pytho > >n/ > >> >b > >> >uild/build.py",> > >> >line 2137, in Launch > >> > self._MultiThreadBuildPlatform() > >> > File " >> >upstream/ws/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Pytho > >n/ > >> >b > >> >uild/build.py",> > >> >line 1917, in _MultiThreadBuildPlatform > >> > self.Progress > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py",> > >> >line 239, in __init__ > >> > self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, > >> >*args, **kwargs) > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py",> > >> >line 403, in _InitWorker > >> > PlatformPcds = Platform.Pcds > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p > >y",> > >> >line 1194, in Pcds > >> > self._Pcds = self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST, > >> >self._Pcds) > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p > >y",> > >> >line 1486, in UpdateStructuredPcds > >> > for Pcd in self.DecPcds: > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p > >y",> > >> >line 3183, in DecPcds > >> > self._DecPcds, self._GuidDict = GetDeclaredPcd(self, self._Bdb, > >> >self._Arch, self._Target, self._Toolchain, PkgSet) > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/WorkspaceCo > >mm > >> >on.py",> > >> >line 62, in GetDeclaredPcd > >> > PkgList = GetPackageList(Platform, BuildDatabase, Arch, Target, > >> >Toolchain) > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/WorkspaceCo > >mm > >> >on.py",> > >> >line 44, in GetPackageList > >> > for ModuleFile in Platform.Modules: > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p > >y",> > >> >line 728, in Modules > >> > self.OverrideDuplicateModule() > >> > File " >> >upstream/ws/edk2/BaseTools/Source/Python/Workspace/DscBuildData.p > >y",> > >> >line 712, in OverrideDuplicateModule > >> > Macros["EDK_SOURCE"] = GlobalData.gEcpSource > >> >AttributeError: 'module' object has no attribute 'gEcpSource' > >> > > >> > > >> > > >> >> >-----Original Message----- > >> >> >From: Feng, Bob C > >> >> >Sent: Friday, January 11, 2019 10:39 AM > >> >> >To: edk2-devel@lists.01.org > >> >> >Cc: Feng, Bob C ; Gao, Liming > >> >> >; Carsey, Jaben > >> >> >Subject: [Patch] BaseTools: Enable component override > >> >> >functionality > >> >> > > >> >> >https://bugzilla.tianocore.org/show_bug.cgi?id=1449 > >> >> >This patch enable build tools to recognize that when two given > >> >> >files have the same GUID, file path and ARCH in Dsc, The later > >> >> >one's definition will be used. > >> >> > > >> >> >Contributed-under: TianoCore Contribution Agreement 1.1 > >> >> >Signed-off-by: Bob Feng > >> >> >Cc: Liming Gao > >> >> >Cc: Carsey Jaben > >> >> >--- > >> >> > .../Source/Python/Workspace/DscBuildData.py | 24 ++++++++++++--- > >--- > >> >- > >> >> > .../Source/Python/Workspace/MetaFileParser.py | 5 ++++ > >> >> > .../Source/Python/Workspace/MetaFileTable.py | 7 ++++-- > >> >> > 3 files changed, 25 insertions(+), 11 deletions(-) > >> >> > > >> >> >diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py > >> >> >b/BaseTools/Source/Python/Workspace/DscBuildData.py > >> >> >index 7e82e8e934..f9805f58f5 100644 > >> >> >--- a/BaseTools/Source/Python/Workspace/DscBuildData.py > >> >> >+++ b/BaseTools/Source/Python/Workspace/DscBuildData.py > >> >> >@@ -704,36 +704,44 @@ class DscBuildData(PlatformBuildClassObject): > >> >> > if TAB_DEFAULT_STORES_DEFAULT not in self.DefaultStores: > >> >> > self.DefaultStores[TAB_DEFAULT_STORES_DEFAULT] = > >> >> >(0, > >> >> >TAB_DEFAULT_STORES_DEFAULT) > >> >> > GlobalData.gDefaultStores = sorted(self.DefaultStores.keys()) > >> >> > return self.DefaultStores > >> >> > > >> >> >+ def OverrideDuplicateModule(self): > >> >> >+ RecordList = self._RawData[MODEL_META_DATA_COMPONENT, > >> >> >self._Arch] > >> >> >+ Macros = self._Macros > >> >> >+ Macros["EDK_SOURCE"] = GlobalData.gEcpSource > >> >> >+ Components = {} > >> >> >+ for Record in RecordList: > >> >> >+ ModuleId = Record[6] > >> >> >+ file_guid = self._RawData[MODEL_META_DATA_HEADER, > >> >self._Arch, > >> >> >None, ModuleId] > >> >> >+ file_guid_str = file_guid[0][2] if file_guid else "NULL" > >> >> >+ ModuleFile = PathClass(NormPath(Record[0], Macros), > >> >> >GlobalData.gWorkspace, Arch=self._Arch) > >> >> >+ if self._Arch != TAB_ARCH_COMMON and > >> >> >(file_guid_str,str(ModuleFile)) in Components: > >> >> >+ > >> >> >self._RawData.DisableOverrideComponent(Components[(file_guid_str, > >s > >> >> >tr > >> >(M > >> >> >oduleFile))]) > >> >> >+ Components[(file_guid_str,str(ModuleFile))] = ModuleId > >> >> >+ self._RawData._PostProcessed = False > >> >> > ## Retrieve [Components] section information > >> >> > @property > >> >> > def Modules(self): > >> >> > if self._Modules is not None: > >> >> > return self._Modules > >> >> >- > >> >> >+ self.OverrideDuplicateModule() > >> >> > self._Modules = OrderedDict() > >> >> > RecordList = self._RawData[MODEL_META_DATA_COMPONENT, > >> >> >self._Arch] > >> >> > Macros = self._Macros > >> >> > Macros["EDK_SOURCE"] = GlobalData.gEcpSource > >> >> > for Record in RecordList: > >> >> >- DuplicatedFile = False > >> >> >- > >> >> > ModuleFile = PathClass(NormPath(Record[0], Macros), > >> >> >GlobalData.gWorkspace, Arch=self._Arch) > >> >> > ModuleId = Record[6] > >> >> > LineNo = Record[7] > >> >> > > >> >> > # check the file validation > >> >> > ErrorCode, ErrorInfo = ModuleFile.Validate('.inf') > >> >> > if ErrorCode != 0: > >> >> > EdkLogger.error('build', ErrorCode, > >> >> > File=self.MetaFile, > >> >Line=LineNo, > >> >> > ExtraData=ErrorInfo) > >> >> >- # Check duplication > >> >> >- # If arch is COMMON, no duplicate module is checked since all > >> >modules > >> >> >in all component sections are selected > >> >> >- if self._Arch != TAB_ARCH_COMMON and ModuleFile in > >> >self._Modules: > >> >> >- DuplicatedFile = True > >> >> > > >> >> > Module = ModuleBuildClassObject() > >> >> > Module.MetaFile = ModuleFile > >> >> > > >> >> > # get module private library instance @@ -792,12 > >> >> >+800,10 @@ class DscBuildData(PlatformBuildClassObject): > >> >> > else: > >> >> > OptionString = Module.BuildOptions[ToolChainFamily, > >ToolChain] > >> >> > Module.BuildOptions[ToolChainFamily, > >> >> >ToolChain] = > >> >OptionString + > >> >> >" " + Option > >> >> > > >> >> > RecordList = self._RawData[MODEL_META_DATA_HEADER, > >> >self._Arch, > >> >> >None, ModuleId] > >> >> >- if DuplicatedFile and not RecordList: > >> >> >- EdkLogger.error('build', FILE_DUPLICATED, File=self.MetaFile, > >> >> >ExtraData=str(ModuleFile), Line=LineNo) > >> >> > if RecordList: > >> >> > if len(RecordList) != 1: > >> >> > EdkLogger.error('build', OPTION_UNKNOWN, > >> >> >'Only FILE_GUID > >> >can > >> >> >be listed in section.', > >> >> > File=self.MetaFile, > >> >> >ExtraData=str(ModuleFile), > >> >Line=LineNo) > >> >> > ModuleFile = ProcessDuplicatedInf(ModuleFile, > >> >> >RecordList[0][2], > >> >> >GlobalData.gWorkspace) > >> >> >diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py > >> >> >b/BaseTools/Source/Python/Workspace/MetaFileParser.py > >> >> >index 032220813b..a52e9229df 100644 > >> >> >--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py > >> >> >+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py > >> >> >@@ -1711,10 +1711,15 @@ class DscParser(MetaFileParser): > >> >> > > >> >> > def __ProcessBuildOption(self): > >> >> > self._ValueList = [ReplaceMacro(Value, self._Macros, > >RaiseError=False) > >> >> > for Value in self._ValueList] > >> >> > > >> >> >+ def DisableOverrideComponent(self,module_id): > >> >> >+ for ori_id in self._IdMapping: > >> >> >+ if self._IdMapping[ori_id] == module_id: > >> >> >+ self._RawTable.DisableComponent(ori_id) > >> >> >+ > >> >> > _SectionParser = { > >> >> > MODEL_META_DATA_HEADER : _DefineParser, > >> >> > MODEL_EFI_SKU_ID : _SkuIdParser, > >> >> > MODEL_EFI_DEFAULT_STORES : _DefaultStoresParser, > >> >> > MODEL_EFI_LIBRARY_INSTANCE : > >_LibraryInstanceParser, > >> >> >diff --git a/BaseTools/Source/Python/Workspace/MetaFileTable.py > >> >> >b/BaseTools/Source/Python/Workspace/MetaFileTable.py > >> >> >index 004e9494c3..823a87e057 100644 > >> >> >--- a/BaseTools/Source/Python/Workspace/MetaFileTable.py > >> >> >+++ b/BaseTools/Source/Python/Workspace/MetaFileTable.py > >> >> >@@ -74,11 +74,11 @@ class MetaFileTable(): > >> >> > > >> >> > def SetEndFlag(self): > >> >> > self.CurrentContent.append(self._DUMMY_) > >> >> > > >> >> > def GetAll(self): > >> >> >- return [item for item in self.CurrentContent if item[0] >= 0 ] > >> >> >+ return [item for item in self.CurrentContent if item[0] > >> >> >+ >= 0 and item[- > >> >> >1]>=0] > >> >> > > >> >> > ## Python class representation of table storing module data > >> >> >class ModuleTable(MetaFileTable): > >> >> > _COLUMN_ = ''' > >> >> > ID REAL PRIMARY KEY, > >> >> >@@ -371,11 +371,10 @@ class PlatformTable(MetaFileTable): > >> >> > # > >> >> > def Query(self, Model, Scope1=None, Scope2=None, > >> >> >BelongsToItem=None, FromItem=None): > >> >> > > >> >> > QueryTab = self.CurrentContent > >> >> > result = [item for item in QueryTab if item[1] == Model > >> >> >and item[-1]>0 ] > >> >> >- > >> >> > if Scope1 is not None and Scope1 != TAB_ARCH_COMMON: > >> >> > Sc1 = set(['COMMON']) > >> >> > Sc1.add(Scope1) > >> >> > result = [item for item in result if item[5] in Sc1] > >> >> > Sc2 = set( ['COMMON','DEFAULT']) @@ -395,10 +394,14 @@ > >> >> >class PlatformTable(MetaFileTable): > >> >> > result = [item for item in result if item[9] == > >> >> >FromItem] > >> >> > > >> >> > result = [ [r[2],r[3],r[4],r[5],r[6],r[7],r[0],r[9]] for r in result ] > >> >> > return result > >> >> > > >> >> >+ def DisableComponent(self,comp_id): > >> >> >+ for item in self.CurrentContent: > >> >> >+ if item[0] == comp_id or item[8] == comp_id: > >> >> >+ item[-1] = -1 > >> >> > > >> >> > ## Factory class to produce different storage for different type > >> >> > of meta- > >> >file > >> >> > class MetaFileStorage(object): > >> >> > _FILE_TABLE_ = { > >> >> > MODEL_FILE_INF : ModuleTable, > >> >> >-- > >> >> >2.19.1.windows.1 > >> >> > >> >> _______________________________________________ > >> >> edk2-devel mailing list > >> >> edk2-devel@lists.01.org > >> >> https://lists.01.org/mailman/listinfo/edk2-devel