From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 16362D811A6 for ; Wed, 17 Jan 2024 07:55:16 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=g5k8QM5CmqwZ9wQylU/obYTbFLQORIkSQrM6cdRu7vE=; c=relaxed/simple; d=groups.io; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1705478115; v=1; b=m8TMZs1omxIx/7lfwJ84O6Z4fA6Wh3A9wPk/+F/ZLhZN/jlnYsMQxFhtP9ln83EMYT65rxPj klxlsKFcZyFBZbW1CzGgkZzydYAiIvqjADRu5Cvob5Ft5jCQGKOGIHTTAuKwXfx5eUm6LZQbaH9 tr4xIz9nXOmtVNWplHf1dGio= X-Received: by 127.0.0.2 with SMTP id YMhNYY7687511xVYIDKtHbym; Tue, 16 Jan 2024 23:55:15 -0800 X-Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by mx.groups.io with SMTP id smtpd.web11.4736.1705478114818279987 for ; Tue, 16 Jan 2024 23:55:15 -0800 X-IronPort-AV: E=McAfee;i="6600,9927,10955"; a="390551783" X-IronPort-AV: E=Sophos;i="6.05,200,1701158400"; d="scan'208";a="390551783" X-Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jan 2024 23:55:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,200,1701158400"; d="scan'208";a="32747665" X-Received: from ashrafal-mobl4.gar.corp.intel.com ([10.213.110.62]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jan 2024 23:55:10 -0800 From: "Ashraf Ali S" To: devel@edk2.groups.io Cc: Ashraf Ali S , Yuwei Chen , Rebecca Cran , Liming Gao , Bob Feng , Amy Chan , Sai Chaganty , Digant H Solanki Subject: [edk2-devel] [PATCH] BaseTools: Optimize GenerateByteArrayValue and CollectPlatformGuids APIs Date: Wed, 17 Jan 2024 13:24:59 +0530 Message-Id: MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,ashraf.ali.s@intel.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: KZGjQhV7K9u9kXkl7OOCCgrhx7686176AA= Content-Transfer-Encoding: 8bit X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=m8TMZs1o; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=intel.com (policy=none) During the Incremental build GenerateByteArrayValue used to generate the ByteArrayValue even when there is no change in the PCD/VPDs. which is time consuming API based on the number of PCD/VPDs and SKU IDs. The optimization is that GenerateByteArrayValue is used to store the PcdRecordList in a JSON file for each of the arch. and during the Incremental build this API will check if there is any change in the PCD /VPDs then rest of the flow remains the same. if there is no change then it will return the provious build data. Flow: during the 1st build PcdRecordList.json is not exists, PcdRecordList will be dumped to json file. and it will copy the output.txt as well. Note: as the output.txt are different for different Arch, so it will be stored in the Arch folder. During the Incremental build check if there is any change in PCD/VPD. if there is a change in VPD/PCD then recreate the PcdRecordList.json. and rest of the flow remains same. if there is no change in VPD/PCD read the output.txt and return the data Cc: Yuwei Chen Cc: Rebecca Cran Cc: Liming Gao Cc: Bob Feng Cc: Amy Chan Cc: Sai Chaganty Cc: Digant H Solanki Signed-off-by: Ashraf Ali S --- .../Source/Python/AutoGen/WorkspaceAutoGen.py | 16 ++--- .../Source/Python/Workspace/DscBuildData.py | 72 +++++++++++++++---- 2 files changed, 64 insertions(+), 24 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py index 160e3a3cd3..eec9280c8e 100644 --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py @@ -160,22 +160,18 @@ class WorkspaceAutoGen(AutoGen): def CollectPlatformGuids(self): oriInfList = [] - oriPkgSet = set() - PlatformPkg = set() + pkgSet = set() for Arch in self.ArchList: Platform = self.BuildDatabase[self.MetaFile, Arch, self.BuildTarget, self.ToolChain] oriInfList = Platform.Modules for ModuleFile in oriInfList: ModuleData = self.BuildDatabase[ModuleFile, Platform._Arch, Platform._Target, Platform._Toolchain] - oriPkgSet.update(ModuleData.Packages) - for Pkg in oriPkgSet: - Guids = Pkg.Guids - GlobalData.gGuidDict.update(Guids) + pkgSet.update(ModuleData.Packages) if Platform.Packages: - PlatformPkg.update(Platform.Packages) - for Pkg in PlatformPkg: - Guids = Pkg.Guids - GlobalData.gGuidDict.update(Guids) + pkgSet.update(Platform.Packages) + for Pkg in pkgSet: + Guids = Pkg.Guids + GlobalData.gGuidDict.update(Guids) @cached_property def FdfProfile(self): diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/Source/Python/Workspace/DscBuildData.py index 4768099343..740b8e22be 100644 --- a/BaseTools/Source/Python/Workspace/DscBuildData.py +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py @@ -37,6 +37,8 @@ from functools import reduce from Common.Misc import SaveFileOnChange from Workspace.BuildClassObject import PlatformBuildClassObject, StructurePcd, PcdClassObject, ModuleBuildClassObject from collections import OrderedDict, defaultdict +import json +import shutil def _IsFieldValueAnArray (Value): Value = Value.strip() @@ -56,6 +58,7 @@ def _IsFieldValueAnArray (Value): PcdValueInitName = 'PcdValueInit' PcdValueCommonName = 'PcdValueCommon' +PcdRecordListName = 'PcdRecordList.json' PcdMainCHeader = ''' /** @@ -1599,7 +1602,7 @@ class DscBuildData(PlatformBuildClassObject): S_pcd_set = DscBuildData.OverrideByComm(S_pcd_set) # Create a tool to caculate structure pcd value - Str_Pcd_Values = self.GenerateByteArrayValue(S_pcd_set) + Str_Pcd_Values = self.GenerateByteArrayValue(S_pcd_set, RecordList) if Str_Pcd_Values: for (skuname, StoreName, PcdGuid, PcdName, PcdValue) in Str_Pcd_Values: @@ -2750,12 +2753,61 @@ class DscBuildData(PlatformBuildClassObject): ccflags.add(item) i +=1 return ccflags - def GenerateByteArrayValue (self, StructuredPcds): + + def GetStructurePcdSet (self, OutputValueFile): + if not os.path.isfile(OutputValueFile): + EdkLogger.error("GetStructurePcdSet", FILE_NOT_FOUND, "Output.txt doesn't exist", ExtraData=OutputValueFile) + return [] + File = open (OutputValueFile, 'r') + FileBuffer = File.readlines() + File.close() + + #start update structure pcd final value + StructurePcdSet = [] + for Pcd in FileBuffer: + PcdValue = Pcd.split ('|') + PcdInfo = PcdValue[0].split ('.') + StructurePcdSet.append((PcdInfo[0], PcdInfo[1], PcdInfo[2], PcdInfo[3], PcdValue[2].strip())) + return StructurePcdSet + + def GenerateByteArrayValue (self, StructuredPcds, PcdRecordList): # # Generate/Compile/Run C application to determine if there are any flexible array members # if not StructuredPcds: return + # + # If the output path doesn't exists then create it + # + if not os.path.exists(self.OutputPath): + os.makedirs(self.OutputPath) + + PcdRecordListPath = os.path.join(self.OutputPath, self._Arch, PcdRecordListName) + PcdRecordOutputValueFile = os.path.join(self.OutputPath, self._Arch, 'Output.txt') + + if not os.path.exists(os.path.dirname(PcdRecordListPath)): + os.makedirs(os.path.dirname(PcdRecordListPath)) + # + # Check if the PcdRecordList.json exists or not + # if exits then it might be a incremental build then check if the PcdRecord list has been changed or not. + # if changed then proceed further, if not changed then return the stored data from earlier build + # + if os.path.isfile(PcdRecordListPath): + with open(PcdRecordListPath, 'r') as file: + file_content_str = file.read() + if file_content_str: + # Use json.loads to convert the string back to a list + file_content = json.loads(file_content_str) + # Check if all PcdRecordList in record_set are present in file_content + # and if there are no extra PcdRecordList in file_content + if set(map(tuple, PcdRecordList)) == set(map(tuple, file_content)): + return self.GetStructurePcdSet(PcdRecordOutputValueFile) + # + # 1st build, create the PcdRecordList.json + # update the record as PCD Input has been changed in incremental build + # + with open(PcdRecordListPath, 'w') as file: + json.dump(PcdRecordList, file) InitByteValue = "" CApp = PcdMainCHeader @@ -2832,8 +2884,6 @@ class DscBuildData(PlatformBuildClassObject): CApp = CApp + PcdMainCEntry + '\n' - if not os.path.exists(self.OutputPath): - os.makedirs(self.OutputPath) CAppBaseFileName = os.path.join(self.OutputPath, PcdValueInitName) SaveFileOnChange(CAppBaseFileName + '.c', CApp, False) @@ -3042,17 +3092,11 @@ class DscBuildData(PlatformBuildClassObject): if returncode != 0: EdkLogger.warn('Build', COMMAND_FAILURE, 'Can not collect output from command: %s\n%s\n%s\n' % (Command, StdOut, StdErr)) - #start update structure pcd final value - File = open (OutputValueFile, 'r') - FileBuffer = File.readlines() - File.close() + # Copy output file for each Arch + shutil.copyfile(OutputValueFile, PcdRecordOutputValueFile) - StructurePcdSet = [] - for Pcd in FileBuffer: - PcdValue = Pcd.split ('|') - PcdInfo = PcdValue[0].split ('.') - StructurePcdSet.append((PcdInfo[0], PcdInfo[1], PcdInfo[2], PcdInfo[3], PcdValue[2].strip())) - return StructurePcdSet + # return structure pcd final value + return self.GetStructurePcdSet(OutputValueFile) @staticmethod def NeedUpdateOutput(OutputFile, ValueCFile, StructureInput): -- 2.39.1.windows.1 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#113934): https://edk2.groups.io/g/devel/message/113934 Mute This Topic: https://groups.io/mt/103781798/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-