From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 0CA79D801B0 for ; Fri, 2 Feb 2024 10:58:18 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=yTsxO3pthTuB8uAxgOPPUvEqm0Vd6/67rjXLNULYoFQ=; c=relaxed/simple; d=groups.io; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1706871497; v=1; b=MMRCzPMgRe8WBHtIg6BuVzzocvhnt1GQc5qN+Nn4rEKBSXTYP/p2tVghshKRQwqR97JvovcA vt+F3iutyRkVlkZsQPtKTWfb4Rt/SrfxlmpwIAZY6v5JgwE6Li3sAMEUFGCpyMGzmjbd9NXinUa V3kMyP6vDzL9lPGfF59SjXFo= X-Received: by 127.0.0.2 with SMTP id BGn5YY7687511xxDiayjfJ2q; Fri, 02 Feb 2024 02:58:17 -0800 X-Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by mx.groups.io with SMTP id smtpd.web10.20217.1706871496327838689 for ; Fri, 02 Feb 2024 02:58:17 -0800 X-IronPort-AV: E=McAfee;i="6600,9927,10971"; a="11498496" X-IronPort-AV: E=Sophos;i="6.05,237,1701158400"; d="scan'208";a="11498496" X-Received: from orviesa002.jf.intel.com ([10.64.159.142]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Feb 2024 02:58:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,237,1701158400"; d="scan'208";a="30862524" X-Received: from ashrafal-mobl4.gar.corp.intel.com ([10.215.118.139]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Feb 2024 02:58:10 -0800 From: "Ashraf Ali S" To: devel@edk2.groups.io Cc: Ashraf Ali S , Yuwei Chen , Rebecca Cran , Liming Gao , Bob Feng , Amy Chan , Sai Chaganty , Digant H Solanki Subject: [edk2-devel] [PATCH v2] BaseTools: Optimize GenerateByteArrayValue and CollectPlatformGuids APIs Date: Fri, 2 Feb 2024 16:27:46 +0530 Message-Id: <86248b9bd1c4dcb5e8f6fadb0420f3111fa18541.1706871442.git.ashraf.ali.s@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,ashraf.ali.s@intel.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: dTEi3aRFFkyql47leIev7u0kx7686176AA= Content-Transfer-Encoding: 8bit X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=MMRCzPMg; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=intel.com (policy=none) During the Incremental build GenerateByteArrayValue used to generate the ByteArrayValue even when there is no change in the PCD/VPDs. which is time consuming API based on the number of PCD/VPDs and SKU IDs. The optimization is that GenerateByteArrayValue is used to store the StructuredPcdsData in a JSON file for each of the arch. and during the Incremental build this API will check, if there is any change in the Structured PCD/VPDs then rest of the flow remains the same. if there is no change then it will return the provious build data. Flow: during the 1st build StructuredPcdsData.json is not exists, StructuredPcdsData will be dumped to json file. and it will copy the output.txt as well. Note: as the output.txt are different for different Arch, so it will be stored in the Arch folder. During the Incremental build check if there is any change in Structured PCD/VPD. if there is a change in Structured VPD/PCD then recreate the StructuredPcdsData.json, and rest of the flow remains same. if there is no change in VPD/PCD read the output.txt and return the data Unit Test: Test1: Modified the Structured Pcds default from DEC file. current flow is executing. Test2: Override the default value of the PCD from DEC file. current flow is executing. Test3: Modified/Override the PCD from DSC file. current flow executing Test4: Modified/Override the FDF from DSC file. current flow executing Test5: update the default value from Command Line.current flow executing Test6: Build without change in PCD in DSC, FDF, DEC and Command Line the proposed changes will be executing, and the return data remains the same with and without the changes. With these changes it's helping to save around ~2.5min to ~3.5min of Incremental build time in my build environment. Sample PR: https://github.com/tianocore/edk2-basetools/pull/113 Cc: Yuwei Chen Cc: Rebecca Cran Cc: Liming Gao Cc: Bob Feng Cc: Amy Chan Cc: Sai Chaganty Cc: Digant H Solanki Signed-off-by: Ashraf Ali S --- .../Source/Python/AutoGen/WorkspaceAutoGen.py | 16 ++-- .../Source/Python/Workspace/DscBuildData.py | 84 ++++++++++++++++--- 2 files changed, 78 insertions(+), 22 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py index 160e3a3cd3..eec9280c8e 100644 --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py @@ -160,22 +160,18 @@ class WorkspaceAutoGen(AutoGen): def CollectPlatformGuids(self): oriInfList = [] - oriPkgSet = set() - PlatformPkg = set() + pkgSet = set() for Arch in self.ArchList: Platform = self.BuildDatabase[self.MetaFile, Arch, self.BuildTarget, self.ToolChain] oriInfList = Platform.Modules for ModuleFile in oriInfList: ModuleData = self.BuildDatabase[ModuleFile, Platform._Arch, Platform._Target, Platform._Toolchain] - oriPkgSet.update(ModuleData.Packages) - for Pkg in oriPkgSet: - Guids = Pkg.Guids - GlobalData.gGuidDict.update(Guids) + pkgSet.update(ModuleData.Packages) if Platform.Packages: - PlatformPkg.update(Platform.Packages) - for Pkg in PlatformPkg: - Guids = Pkg.Guids - GlobalData.gGuidDict.update(Guids) + pkgSet.update(Platform.Packages) + for Pkg in pkgSet: + Guids = Pkg.Guids + GlobalData.gGuidDict.update(Guids) @cached_property def FdfProfile(self): diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/Source/Python/Workspace/DscBuildData.py index 4768099343..3b4a75d86a 100644 --- a/BaseTools/Source/Python/Workspace/DscBuildData.py +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py @@ -37,6 +37,8 @@ from functools import reduce from Common.Misc import SaveFileOnChange from Workspace.BuildClassObject import PlatformBuildClassObject, StructurePcd, PcdClassObject, ModuleBuildClassObject from collections import OrderedDict, defaultdict +import json +import shutil def _IsFieldValueAnArray (Value): Value = Value.strip() @@ -56,6 +58,7 @@ def _IsFieldValueAnArray (Value): PcdValueInitName = 'PcdValueInit' PcdValueCommonName = 'PcdValueCommon' +StructuredPcdsDataName = 'StructuredPcdsData.json' PcdMainCHeader = ''' /** @@ -2750,6 +2753,23 @@ class DscBuildData(PlatformBuildClassObject): ccflags.add(item) i +=1 return ccflags + + def GetStructurePcdSet (self, OutputValueFile): + if not os.path.isfile(OutputValueFile): + EdkLogger.error("GetStructurePcdSet", FILE_NOT_FOUND, "Output.txt doesn't exist", ExtraData=OutputValueFile) + return [] + File = open (OutputValueFile, 'r') + FileBuffer = File.readlines() + File.close() + + #start update structure pcd final value + StructurePcdSet = [] + for Pcd in FileBuffer: + PcdValue = Pcd.split ('|') + PcdInfo = PcdValue[0].split ('.') + StructurePcdSet.append((PcdInfo[0], PcdInfo[1], PcdInfo[2], PcdInfo[3], PcdValue[2].strip())) + return StructurePcdSet + def GenerateByteArrayValue (self, StructuredPcds): # # Generate/Compile/Run C application to determine if there are any flexible array members @@ -2757,6 +2777,54 @@ class DscBuildData(PlatformBuildClassObject): if not StructuredPcds: return + StructuredPcdsData = {} + + for PcdName in StructuredPcds: + Pcd = StructuredPcds[PcdName] + TokenSpaceGuidCName = Pcd.TokenSpaceGuidCName + TokenCName = Pcd.TokenCName + + # Create a key using TokenSpaceGuidCName and TokenCName + StructuredPcdsData[f"{TokenSpaceGuidCName}_{TokenCName}"] = { + "DefaultValueFromDec": Pcd.DefaultValueFromDec, + "DefaultValues": Pcd.DefaultValues, + "PcdFieldValueFromComm": Pcd.PcdFieldValueFromComm, + "PcdFieldValueFromFdf": Pcd.PcdFieldValueFromFdf, + "DefaultFromDSC": Pcd.DefaultFromDSC + } + + # + # If the output path doesn't exists then create it + # + if not os.path.exists(self.OutputPath): + os.makedirs(self.OutputPath) + + StructuredPcdsDataPath = os.path.join(self.OutputPath, self._Arch, StructuredPcdsDataName) + PcdRecordOutputValueFile = os.path.join(self.OutputPath, self._Arch, 'Output.txt') + + if not os.path.exists(os.path.dirname(StructuredPcdsDataPath)): + os.makedirs(os.path.dirname(StructuredPcdsDataPath)) + # + # Check if the StructuredPcdsData.json exists or not + # if exits then it might be a incremental build then check if the StructuredPcdsData has been changed or not. + # if changed then proceed further, if not changed then return the stored data from earlier build + # + if os.path.isfile(StructuredPcdsDataPath): + with open(StructuredPcdsDataPath, 'r') as file: + StoredStructuredPcdsData = json.load(file) + if StructuredPcdsData == StoredStructuredPcdsData: + return self.GetStructurePcdSet(PcdRecordOutputValueFile) + else: + # update the record as PCD Input has been changed in incremental build + with open(StructuredPcdsDataPath, 'w') as file: + json.dump(StructuredPcdsData, file, indent=2) + else: + # + # 1st build, create the StructuredPcdsData.json + # + with open(StructuredPcdsDataPath, 'w') as file: + json.dump(StructuredPcdsData, file, indent=2) + InitByteValue = "" CApp = PcdMainCHeader @@ -2832,8 +2900,6 @@ class DscBuildData(PlatformBuildClassObject): CApp = CApp + PcdMainCEntry + '\n' - if not os.path.exists(self.OutputPath): - os.makedirs(self.OutputPath) CAppBaseFileName = os.path.join(self.OutputPath, PcdValueInitName) SaveFileOnChange(CAppBaseFileName + '.c', CApp, False) @@ -3042,17 +3108,11 @@ class DscBuildData(PlatformBuildClassObject): if returncode != 0: EdkLogger.warn('Build', COMMAND_FAILURE, 'Can not collect output from command: %s\n%s\n%s\n' % (Command, StdOut, StdErr)) - #start update structure pcd final value - File = open (OutputValueFile, 'r') - FileBuffer = File.readlines() - File.close() + # Copy update output file for each Arch + shutil.copyfile(OutputValueFile, PcdRecordOutputValueFile) - StructurePcdSet = [] - for Pcd in FileBuffer: - PcdValue = Pcd.split ('|') - PcdInfo = PcdValue[0].split ('.') - StructurePcdSet.append((PcdInfo[0], PcdInfo[1], PcdInfo[2], PcdInfo[3], PcdValue[2].strip())) - return StructurePcdSet + #start update structure pcd final value + return self.GetStructurePcdSet(OutputValueFile) @staticmethod def NeedUpdateOutput(OutputFile, ValueCFile, StructureInput): -- 2.39.1.windows.1 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#115043): https://edk2.groups.io/g/devel/message/115043 Mute This Topic: https://groups.io/mt/104117182/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-