From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web08.23810.1604987058741473266 for ; Mon, 09 Nov 2020 21:44:18 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=ackEIunz; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: mingyuex.liang@intel.com) IronPort-SDR: KiDa4YABsHqx8mL7dMvHTAoj0b+VkrQHUFZXgeljelWjAGAjyJU7DNm/S5k0svC4LG9Qfgh0/8 fRIKJfQIjgjA== X-IronPort-AV: E=McAfee;i="6000,8403,9800"; a="149200644" X-IronPort-AV: E=Sophos;i="5.77,465,1596524400"; d="scan'208";a="149200644" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2020 21:44:17 -0800 IronPort-SDR: 2bXPvhDIZh4MofIhb4p23tX7sTP+S07ZycYzHW8fqbVQdDe9d1EMbwrr3o4ETYg75VsJkucsbI n73WtJ1mDq0Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,465,1596524400"; d="scan'208";a="428246760" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga001.fm.intel.com with ESMTP; 09 Nov 2020 21:44:17 -0800 Received: from fmsmsx609.amr.corp.intel.com (10.18.126.89) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 9 Nov 2020 21:44:17 -0800 Received: from fmsmsx605.amr.corp.intel.com (10.18.126.85) by fmsmsx609.amr.corp.intel.com (10.18.126.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 9 Nov 2020 21:44:16 -0800 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 9 Nov 2020 21:44:16 -0800 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.109) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Mon, 9 Nov 2020 21:44:16 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=i+bRyJnOmOF2C663O4nOg/Amdg81IOm2B421SCLDTLZJZ1fPDlCFOXOb1okExzQtrXGW4ocgSPCZm3urSMojdf2JEO2VCRzDGQ8BpDzBTrRmRPzZYJBKhe6oFQFXJa9iLLC/6FUfE0HijTJWDktXhIPG1QZ/nlj5+zCZMbhR45nO0Qy6Ar+EDEYZo37yVhaQjhptJ0AS97khWX3iE76puHEzMecwNqrF5RvdD/x1nMwVM16RiAT8q475TNF7//OxkKP74VQTb+NmA7kTgzXqdyydJCtF0grGeocawVDBELO57q1c1yEYXnMXt7j7dpjXADP4XCoAgw38HkeyVUUDIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MAqpNn6hHt3fVQrnavTeL/zaKTU9znKXiyUKurTk8KI=; b=I1FtCy9RdO7x8uHqoejWk9zQIgnMuJvrYHm/FO0Yb4vrI8Cdha/5Xy5cqPcJfhW7oM754r9bo74h6SzKAumgk/KCODS+bKdehQ2qNJU8qii8xk2LJpWOOxYcwngPG7MBEjc3TgjQhU0lyLxbJlnei2lRgOW+vnIKaf2eVMk3s5UFWgu7k86lOvd+X4l4FUnhahLMBe/q3Pt+06+qVVCKYeGDBTJgFq0m6uZFls+BeCdkn/EhdumcRI9nIy3v/fHi1oflFfT1dzbAMRCIH2Rc+iakJI34jgarAzQpDJaftK/h/lBOGfAMXnwO2FgSfGM8Z8GPowQluPhJ+IpFFzN61Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MAqpNn6hHt3fVQrnavTeL/zaKTU9znKXiyUKurTk8KI=; b=ackEIunz1e6TNB9PzmGffs6/JhR0JrhUx6twDxY3JvhvsFg+o/beoa1MCJ3R8rV8RIY3HH9LBn7fQ3QfpJoE7/Im0IRqHRLTTjdpgXnJe+8VVsH06DTRXAKh4twA3uQsE/NjFReBHrDTPq+Ez5lkX2V0xaO8ppcOdlSf0ywe0Us= Received: from DM6PR11MB3369.namprd11.prod.outlook.com (2603:10b6:5:b::16) by DM6PR11MB2842.namprd11.prod.outlook.com (2603:10b6:5:c5::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.21; Tue, 10 Nov 2020 05:44:15 +0000 Received: from DM6PR11MB3369.namprd11.prod.outlook.com ([fe80::e911:9d94:79ae:cba0]) by DM6PR11MB3369.namprd11.prod.outlook.com ([fe80::e911:9d94:79ae:cba0%4]) with mapi id 15.20.3541.025; Tue, 10 Nov 2020 05:44:15 +0000 From: "Mingyue Liang" To: "Feng, Bob C" , "devel@edk2.groups.io" CC: Liming Gao , "Chen, Christine" , "Vaish, Atul" Subject: Re: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. Thread-Topic: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. Thread-Index: AQHWsxK/NuXVYUWkSEi8CLw1a7fuLqm41EFggABAsICABkWtIIABbaxggAAaSBA= Date: Tue, 10 Nov 2020 05:44:15 +0000 Message-ID: References: <20201105012638.662-1-mingyuex.liang@intel.com> <1645C2A4018343CE.24110@groups.io> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.102.204.37] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f421a360-268d-4199-113b-08d8853ba8db x-ms-traffictypediagnostic: DM6PR11MB2842: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:120; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: AIDGhbNZlRP805f8gcpI8YnXEtlXABm1ICnarPsNngjFIAM61eYyngbrQB0drw70+S/cnQKmGRy8pRN5tbpitTYg1pUZWC3DNCyCPmjXkzQXpMVqOGpw2bJLo1K0ZZNylbQIAkECI4iOAIYBFw+dHLXP1zxSnGmIyzPulZ9S1H4CO8GaiErkTucCVWTt77hp8w1t4Ubm5Ov/EWK3LbbYguywnSOnynxiEF4C02IWLcXT1RJD6kS0ftgL4jnoE/4C7V20KSD87GymHVEBeW5BRXxxxrUFr7gLZ7vh+s+ivwekM4e5FJ8ny7FMlv+VzRn2d+dwPrScUWOgeltROhEGrYCGUoWLGRc5//JQ6mB7JbvHygdGSjeD1CVR8hoz53SUzv06m/H9k58MvG2kNUiYHg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR11MB3369.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(136003)(39860400002)(346002)(396003)(376002)(366004)(76116006)(64756008)(66556008)(316002)(86362001)(54906003)(66946007)(110136005)(107886003)(4326008)(66476007)(9686003)(5660300002)(55016002)(66446008)(83380400001)(52536014)(7696005)(966005)(33656002)(8936002)(53546011)(186003)(478600001)(26005)(8676002)(6506007)(71200400001)(2906002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: GcCVoUVqk1Kqmn8iJEE5nEo2D3uZq3G6hzyIRDQK1vCYrgOxFeCq9CZ1XEbXMP56yObap2KzlAIqo+05/Oc080omyjL5fqDepveih2msDS+c2e1CRfJ4l70qB5G1RDca5/fyo+kMhmtLpBpPQik4Zj27mukG2jOnhRgI17ihK0/H8QySu8tNdCnlE7yShLdDp22Ty36KaaaGon634cUgJpWQVfsxJe6dE1vHbwyHzLOWE8B9EGTGqQBrqldoz7d7b9xJHdN5wsvBDNxT8C83iQjH0tH1ZaJM1n71ey/WYUUmgCpVNIbzcu498YX4k7byGOHB5HqRqsQOtAyWN+WIZAoWJaS4fqMwiZLxg9PDT3iw6EYQxPZ5oFKTEaFzP9L/HnLcKtOxYeT89QfkdoNGSa8LaRCKFeGkI+Ii+RP09xINWnD29z0z82tS15gyoUFuXwewdvza/dzziBSGtQNmJs0O8QEbiOI2x7qhyNYWbVm4eOUiTQbjUlP/lBaQGhuqsqYyR/Lz/0mvGOkMF9djeXzopz5aTT658AZgJuwVcepxOLMedDWQONUBxPE3yIUwv10wvUsICYOYY/gIy99w+bHwXIUc3jQl1CD5t50ERGkilhZE8FEjSF9YE0rgoOtZgbRBatokhHBYzBy9TVhhtQ== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3369.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f421a360-268d-4199-113b-08d8853ba8db X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2020 05:44:15.4959 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wllm3PKygJU2paK+1m/GMxxSu43blJOWjrZEbyuCJ4Z8fZohjj+9fBiJD9DmbaKFqAWT/E24s054lLMKsD0Nbw7oi+nr2n6wiu4eWAU8gCo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB2842 Return-Path: mingyuex.liang@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Updated to V3. -----Original Message----- From: Feng, Bob C =20 Sent: Tuesday, November 10, 2020 12:10 PM To: devel@edk2.groups.io; Feng, Bob C ; Liang, Mingy= ueX Cc: Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. Mingyue, Please remove the statements of "V2: ..." from the commit message.=20 Thanks, Bob -----Original Message----- From: devel@edk2.groups.io On Behalf Of Bob Feng Sent: Monday, November 9, 2020 2:21 PM To: Liang, MingyueX ; devel@edk2.groups.io Cc: Liming Gao ; Chen, Christine ; Vaish, Atul Subject: Re: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. OK.=20 Reviewed-by: Bob Feng -----Original Message----- From: Liang, MingyueX Sent: Thursday, November 5, 2020 2:33 PM To: Feng, Bob C ; devel@edk2.groups.io Cc: Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Bob, This method is not recommended and will result in build errors. Thanks, Mingyue -----Original Message----- From: Feng, Bob C Sent: Thursday, November 5, 2020 10:43 AM To: Liang, MingyueX ; devel@edk2.groups.io Cc: Liang, MingyueX ; Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Mingyue, Would you update the cc_resp_* file content? One c file one line. Thanks, Bob -----Original Message----- From: Mingyue Liang Sent: Thursday, November 5, 2020 9:27 AM To: devel@edk2.groups.io Cc: Liang, MingyueX ; Feng, Bob C ; Liming Gao ; Chen, Christine Subject: [PATCH.V2] BaseTools: Limit command line length. From: mliang2x REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2528 Currently, CL command contains multiple C files will be compiled, and that= caused command line too long, which may trigger build error. In order to solve this issue, the following rules is used in this scene: If the number of C files is greater than one, a txt file will be used to r= ecord these C files, and replaces the corresponding content in command line= with the file name. Else (only one C file listed in the command line), the length of the whole= CL command line will determine whether use a file to record. If the length= exceeds the limited max length, use the recording file; else C file name d= irectly listed in the command line V2: 1,Updated patch description 2,Changing the storage mode of C file is easy to obtain Signed-off-by: Mingyue Liang Cc: Bob Feng Cc: Liming Gao Cc: Yuwei Chen --- BaseTools/Source/Python/AutoGen/GenMake.py | 41 +++++++++++++++---- .../Source/Python/AutoGen/IncludesAutoGen.py | 12 +++++- 2 files changed, 44 insertions(+), 9 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source= /Python/AutoGen/GenMake.py index b04d3f54368e..1cfac1cd82ca 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -576,7 +576,7 @@ cleanlib: EdkLogger.error("build", AUTOGEN_ERROR, "Nothing to build", ExtraData=3D"[%s]" % str(MyAgo)) =20 - self.ProcessBuildTargetList() + self.ProcessBuildTargetList(MyAgo.OutputDir, ToolsDef) self.ParserGenerateFfsCmd() =20 # Generate macros used to represent input files @@ -903,7 +903,7 = @@ cleanlib: BuildTargets[Target].Commands[i] =3D = SingleCommand.replace('$(INC)', '').replace(FlagDict[Flag]['Macro'], RespMa= cro) return RespDict =20 - def ProcessBuildTargetList(self): + def ProcessBuildTargetList(self, RespFile, ToolsDef): # # Search dependency file list for each source file # @@ -1004,6 +1004,7 @@ cleanlib: self.ObjTargetDict[T.Target.SubDir] =3D set() self.ObjTargetDict[T.Target.SubDir].add(NewFile) for Type in self._AutoGenObject.Targets: + resp_file_number =3D 0 for T in self._AutoGenObject.Targets[Type]: # Generate related macros if needed if T.GenFileListMacro and T.FileListMacro not in self.Fil= eListMacros: @@ -1045,7 +1046,10 @@ cleanlib: Deps.append("$(%s)" % T.ListFileMacro) =20 if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMPILE= R_MSFT and Type =3D=3D TAB_C_CODE_FILE: - T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Pars= erCCodeFile(T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict) + T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Pars= erCCodeFile(T, Type, CmdSumDict, CmdTargetDict, + = CmdCppDict, DependencyDict, RespFile, + = ToolsDef, resp_file_number) + resp_file_number +=3D 1 TargetDict =3D {"target": self.PlaceMacro(T.Target.Pa= th, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": CCodeDeps} CmdLine =3D self._BUILD_TARGET_TEMPLATE.Replace(Targe= tDict).rstrip().replace('\t$(OBJLIST', '$(OBJLIST') if T.Commands: @@ -1062,7 +1066,9 @@ cleanlib: AnnexeTargetDict =3D {"target": self.PlaceMacro(i= .Path, self.Macros), "cmd": "", "deps": self.PlaceMacro(T.Target.Path, self= .Macros)} self.BuildTargetList.append(self._BUILD_TARGET_TE= MPLATE.Replace(AnnexeTargetDict)) =20 - def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppD= ict, DependencyDict): + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppD= ict, DependencyDict, RespFile, ToolsDef, + resp_file_number): + SaveFilePath =3D os.path.join(RespFile, "cc_resp_%s.txt" % + resp_file_number) if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: CmdSumDict[item.Target.SubDir] =3D item.Target.BaseName @= @ -1089,17 +1095,36 @@ cleanlib: if Temp.startswith('/Fo'): CmdSign =3D '%s%s' % (Temp.rsplit(TAB_SLASH, = 1)[0], TAB_SLASH) break - else: continue + else: + continue if CmdSign not in list(CmdTargetDict.keys()): - CmdTargetDict[CmdSign] =3D Item.replace(Temp, Cmd= Sign) + cmd =3D Item.replace(Temp, CmdSign) + if SingleCommandList[-1] in cmd: + CmdTargetDict[CmdSign] =3D=20 + [cmd.replace(SingleCommandList[-1], "").rstrip(),=20 + SingleCommandList[-1]] else: - CmdTargetDict[CmdSign] =3D "%s %s" % (CmdTargetDi= ct[CmdSign], SingleCommandList[-1]) + # CmdTargetDict[CmdSign] =3D "%s %s" %=20 + (CmdTargetDict[CmdSign], SingleCommandList[-1]) + + CmdTargetDict[CmdSign].append(SingleCommandList[-1]) Index =3D CommandList.index(Item) CommandList.pop(Index) if SingleCommandList[-1].endswith("%s%s.c" % (TAB_SLA= SH, CmdSumDict[CmdSign[3:].rsplit(TAB_SLASH, 1)[0]])): Cpplist =3D CmdCppDict[T.Target.SubDir] Cpplist.insert(0, '$(OBJLIST_%d): ' % list(self.O= bjTargetDict.keys()).index(T.Target.SubDir)) - T.Commands[Index] =3D '%s\n\t%s' % (' \\\n\t'.joi= n(Cpplist), CmdTargetDict[CmdSign]) + source_files =3D CmdTargetDict[CmdSign][1:] + source_files.insert(0, " ") + if len(source_files)>2: + SaveFileOnChange(SaveFilePath, " ".join(sourc= e_files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)= ' % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[CmdSig= n][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + elif len(source_files)<=3D2 and len(" ".join(CmdT= argetDict[CmdSign][:2]))>GlobalData.gCommandMaxLength: + SaveFileOnChange(SaveFilePath, " ".join(sourc= e_files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)= ' % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[Cm= dSign][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + else: + T.Commands[Index] =3D '%s\n\t%s' % ('=20 + \\\n\t'.join(Cpplist), " ".join(CmdTargetDict[CmdSign])) else: T.Commands.pop(Index) return T, CmdSumDict, CmdTargetDict, CmdCppDict diff --git a/Base= Tools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/Au= toGen/IncludesAutoGen.py index c3e6333217c4..b06ef42573fd 100644 --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -201,7 +201,17 @@ ${END} cc_options =3D line[len(cc_cmd)+2:].split() else: cc_options =3D line[len(cc_cmd):].split() - SourceFileAbsPathMap =3D {os.path.basename(item):= item for item in cc_options if not item.startswith("/") and os.path.exists(= item)} + for item in cc_options: + if not item.startswith("/"): + if item.endswith(".txt") and item.startsw= ith("@"): + with open(item[1:], "r") as file: + source_files =3D file.readlines()= [0].split() + SourceFileAbsPathMap =3D {os.path= .basename(file): file for file in source_files if + os.path.e= xists(file)} + else: + if os.path.exists(item): + SourceFileAbsPathMap.update({os.p= ath.basename(item): item.strip()}) + # SourceFileAbsPathMap =3D=20 + {os.path.basename(item):item for item in cc_options if not + item.startswith("/") and os.path.exists(item)} if line in SourceFileAbsPathMap: current_source =3D line if current_source not in ModuleDepDict: -- 2.28.0.windows.1