From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx.groups.io with SMTP id smtpd.web12.23121.1604981408873085380 for ; Mon, 09 Nov 2020 20:10:09 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=lHr+vh4N; spf=pass (domain: intel.com, ip: 192.55.52.88, mailfrom: bob.c.feng@intel.com) IronPort-SDR: CC0lP2S7zddhwd8/HHwCFpMLKN5vAuPXk1i5M4ZymRb8xJ76SNOgZKRzByNNLnU9wJM3fuDcDF RiLcCVH2PgOA== X-IronPort-AV: E=McAfee;i="6000,8403,9800"; a="187875938" X-IronPort-AV: E=Sophos;i="5.77,465,1596524400"; d="scan'208";a="187875938" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2020 20:10:07 -0800 IronPort-SDR: vO/YEo3DgaUz41+On1V+9D/cMWHtxGDHzDpWAyEwoQLru9hIk3Bc7nh8h4I81og+0WJAc2foer fRjHztsQNOWw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,465,1596524400"; d="scan'208";a="327533358" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga006.jf.intel.com with ESMTP; 09 Nov 2020 20:10:07 -0800 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 9 Nov 2020 20:10:06 -0800 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 9 Nov 2020 20:10:06 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 9 Nov 2020 20:10:06 -0800 Received: from NAM02-CY1-obe.outbound.protection.outlook.com (104.47.37.55) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Mon, 9 Nov 2020 20:10:05 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kZyRT0wwZKX6xnf4ZC/u1zrKEeACFVC6u8KurbyRJ3a9WgJQ9FAkHUDsz1jNyHO4nYtjvL6503MjUvxfPZVTotgUzlQL1racNjZ8hYgU9dptkjOsPasesyAUc+A5wGqoa1uzWoXb7rnsbJBCqPAKV59v5/owszsNjesmmQokp1NAW4iBmeQlON4lv7K1dSVJNzAqX1CsTD+VS+RllB3Job9rvxJ/PBfDARXE1qmA5xInDKZEiSbNQPKiYZ/OOY3scwY369ncwaBTYfKEPcREOYs3nkj4Q4wB0FYFAjA3Ntqkn5aJVNiCWRfA9ybLx+dGmU1wnVXYXk/oogkpLrhaPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GpEpzdIWx5lY5IemcNYj5/Eyz3yjVLhSRwsD1kTEhcE=; b=UKRURo0OpC3rthkj+/O8ln23JsUGVQnIYTEI2/bgC8c9mzr1H9G/dnQZa61/K/x+yx85Hr9XDppAbXxBw9ROaQ5nIV9lEUpRM51Ks3mKIigyh1607wfcdljoV7VyEIavAAYRv66pSGwJWqg3WXAeVEYprC9UKZQEBqda1mvprVAYFpCCYjhh8Nc9ixbFm7pwAv8nvEQnIZaHFVQzf+LDIzNrKcvaoJqXZddHvcNtGzwrl/QVJTiiTPGRiUa92/Ix/Uzah8EQtptQtAzmpQH59PYa74tMc7PycJQKIfmxV5h3m0IkB/cMQmeT5AcouoBBsbObLHrnOP4YYCB+p4ms8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GpEpzdIWx5lY5IemcNYj5/Eyz3yjVLhSRwsD1kTEhcE=; b=lHr+vh4Nl5hviZBhyL3CVGdns2HoYv3Z/op+6D0EhNIsN0u8o8hsKqL3bNohWkBidoD4lNWWZlD009psh3rxKGQxYIDXSbcjg377e/KfkvF0yEXx4IQLSQkcNvfYMZiuJoZp9VcdXnPQygAnAbCISy+o8dnCXodzVoGYxghmkL4= Received: from DM6PR11MB4073.namprd11.prod.outlook.com (2603:10b6:5:19f::22) by DM5PR1101MB2188.namprd11.prod.outlook.com (2603:10b6:4:53::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.24; Tue, 10 Nov 2020 04:10:03 +0000 Received: from DM6PR11MB4073.namprd11.prod.outlook.com ([fe80::c9f:47ea:6321:5ccb]) by DM6PR11MB4073.namprd11.prod.outlook.com ([fe80::c9f:47ea:6321:5ccb%3]) with mapi id 15.20.3541.025; Tue, 10 Nov 2020 04:10:03 +0000 From: "Bob Feng" To: "devel@edk2.groups.io" , "Feng, Bob C" , "Liang, MingyueX" CC: Liming Gao , "Chen, Christine" , "Vaish, Atul" Subject: Re: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. Thread-Topic: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. Thread-Index: AQHWsxK/gedGv+mhpky4pZTiMyYLJKm41EFggABAsICABkWtIIABbaxg Date: Tue, 10 Nov 2020 04:10:03 +0000 Message-ID: References: <20201105012638.662-1-mingyuex.liang@intel.com> <1645C2A4018343CE.24110@groups.io> In-Reply-To: <1645C2A4018343CE.24110@groups.io> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.55.46.39] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c50dfd25-871c-42c6-42f2-08d8852e7fd8 x-ms-traffictypediagnostic: DM5PR1101MB2188: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:120; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lM+iK2sIcyc47K5kMxst2xPnNpoyX/hSJty0e+Ep8Td0PNtAPRDmWRPU2Xey8LFnJ85Wn9BYgZ1FmVIKVvaidu4HkQDBgHIz67rGpyTxnduuJ4bWNYDkoxpvrCerwOb0ZZoQxWzf/QwQpxNe8IfSpZJTyUejIak0eSyQQ8ZrWYMUHL4zfHOwDUicgsjtKsC0amC2nlUNT6ize+9M3UHo+prDPV7AiGZGo/Ut2wYKY2yn9XT5WEC+IzFbl/+h7RgqkRyYvm/ut+gI34+uXhpFbjJfMvxgGFCLScVk0+XLeiR8WXFD4Wd4JLmwS1qztJBfJHBOLAKT5ju7X0kFwVxhAtKKcmmvoJKJ9BZJcoDyvZrsHqEmaVmN4/Kn5A9vIM1iUZxbwuBn42faG/i+LflwRg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR11MB4073.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(396003)(136003)(366004)(376002)(39860400002)(346002)(8676002)(66556008)(66476007)(64756008)(33656002)(71200400001)(66446008)(316002)(55016002)(107886003)(5660300002)(966005)(86362001)(7696005)(186003)(9686003)(478600001)(110136005)(54906003)(6636002)(26005)(66946007)(6506007)(4326008)(83380400001)(52536014)(8936002)(53546011)(2906002)(76116006);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: uhMtZut8JyA1aN0c2wmU5ODpFjevy0+wBvAKANpSkCKh79tYwnSZI2NqPbOv2yaQ1GjuJopQjj+BwT7P853rvaL/nUZwTm7TZJiVFaZs2jVGIsvbSElfkHqCINLQRUn1qqBIhjifxN/fgwRW+vNbkZqKiH9aNyFQZU6gB5P8LSMyi0ES7pCxtDuBiUs4Lt5s6xIWQEfnCRcMG1z+5YMQuG9bKtPyXD852VBcqytZ3/Vhze7wr6D5XYAr+dIKh9USGJxKy3Hx7Mm1pR/7hSgzsL47vvuhV518TzIUqVk/TsQKVCR6Zt0Umv8000BzQxHV6DbPgaedRtybVtk6eH+uIHRwSgXhn4EzndCdgL8TgwyOIIGnTzVPiqVpCuK2NjFdGtCEVNQq5Tp8VNDqa0RcIUSzOQ786tOBITck4A5t77l01oOPnGEZR2phkZdLR+0VcUPl/kFhX6536I/LyP16L9TY/mjiNC5CQZrvcxmEGPi+oPXjXpU1LwALFogmYicyziVXKChOlwO5azfoo0t2J3a2WzXkPGe3XwDqizVZLHpkC9gM9rdOK3gkZQWtgAePmGs2HSp826FSFb7QF19PgWpX7ZawkRbB0DO6dGoP5f6dg4UcT+0UEbNxjIrarUeITB2p1QUwXcCGlOWP0tWeqZswdJcR5HWX2W2UUJp6lab3RuuYGJEJBnkyT0bDQm1AEhUYIESR90+6KWwpPHOk1fVIcow638EZ+qp2/vmJKbg9xcuPO0bpRn9vq7Iieq9goS1zW5eZE5lY5RV+j43/V00Sm60keQjKCCh3TvSS9+oDV7iqMPBkyoCIWrALixSn+yi1Hwd6I8ElsYwCUamZ9xUir1Ioegvv7gkbeDXZh8YhS+FXYi8RnPqIqEP+nQGkN36c0nceWYsxshy95LQxgA== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4073.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c50dfd25-871c-42c6-42f2-08d8852e7fd8 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2020 04:10:03.1694 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 8niNkFZVgHkvzq77ul8uEVXzhK9tuts/3ku4aO01uNGTLFgtL0hvgk8dnbbeTIlKvh2mAZtBzy3+u4KMZEFmFg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1101MB2188 Return-Path: bob.c.feng@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mingyue, Please remove the statements of "V2: ..." from the commit message.=20 Thanks, Bob -----Original Message----- From: devel@edk2.groups.io On Behalf Of Bob Feng Sent: Monday, November 9, 2020 2:21 PM To: Liang, MingyueX ; devel@edk2.groups.io Cc: Liming Gao ; Chen, Christine ; Vaish, Atul Subject: Re: [edk2-devel] [PATCH.V2] BaseTools: Limit command line length. OK.=20 Reviewed-by: Bob Feng -----Original Message----- From: Liang, MingyueX Sent: Thursday, November 5, 2020 2:33 PM To: Feng, Bob C ; devel@edk2.groups.io Cc: Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Bob, This method is not recommended and will result in build errors. Thanks, Mingyue -----Original Message----- From: Feng, Bob C Sent: Thursday, November 5, 2020 10:43 AM To: Liang, MingyueX ; devel@edk2.groups.io Cc: Liang, MingyueX ; Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Mingyue, Would you update the cc_resp_* file content? One c file one line. Thanks, Bob -----Original Message----- From: Mingyue Liang Sent: Thursday, November 5, 2020 9:27 AM To: devel@edk2.groups.io Cc: Liang, MingyueX ; Feng, Bob C ; Liming Gao ; Chen, Christine Subject: [PATCH.V2] BaseTools: Limit command line length. From: mliang2x REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2528 Currently, CL command contains multiple C files will be compiled, and that= caused command line too long, which may trigger build error. In order to solve this issue, the following rules is used in this scene: If the number of C files is greater than one, a txt file will be used to r= ecord these C files, and replaces the corresponding content in command line= with the file name. Else (only one C file listed in the command line), the length of the whole= CL command line will determine whether use a file to record. If the length= exceeds the limited max length, use the recording file; else C file name d= irectly listed in the command line V2: 1,Updated patch description 2,Changing the storage mode of C file is easy to obtain Signed-off-by: Mingyue Liang Cc: Bob Feng Cc: Liming Gao Cc: Yuwei Chen --- BaseTools/Source/Python/AutoGen/GenMake.py | 41 +++++++++++++++---- .../Source/Python/AutoGen/IncludesAutoGen.py | 12 +++++- 2 files changed, 44 insertions(+), 9 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source= /Python/AutoGen/GenMake.py index b04d3f54368e..1cfac1cd82ca 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -576,7 +576,7 @@ cleanlib: EdkLogger.error("build", AUTOGEN_ERROR, "Nothing to build", ExtraData=3D"[%s]" % str(MyAgo)) =20 - self.ProcessBuildTargetList() + self.ProcessBuildTargetList(MyAgo.OutputDir, ToolsDef) self.ParserGenerateFfsCmd() =20 # Generate macros used to represent input files @@ -903,7 +903,7 = @@ cleanlib: BuildTargets[Target].Commands[i] =3D = SingleCommand.replace('$(INC)', '').replace(FlagDict[Flag]['Macro'], RespMa= cro) return RespDict =20 - def ProcessBuildTargetList(self): + def ProcessBuildTargetList(self, RespFile, ToolsDef): # # Search dependency file list for each source file # @@ -1004,6 +1004,7 @@ cleanlib: self.ObjTargetDict[T.Target.SubDir] =3D set() self.ObjTargetDict[T.Target.SubDir].add(NewFile) for Type in self._AutoGenObject.Targets: + resp_file_number =3D 0 for T in self._AutoGenObject.Targets[Type]: # Generate related macros if needed if T.GenFileListMacro and T.FileListMacro not in self.Fil= eListMacros: @@ -1045,7 +1046,10 @@ cleanlib: Deps.append("$(%s)" % T.ListFileMacro) =20 if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMPILE= R_MSFT and Type =3D=3D TAB_C_CODE_FILE: - T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Pars= erCCodeFile(T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict) + T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Pars= erCCodeFile(T, Type, CmdSumDict, CmdTargetDict, + = CmdCppDict, DependencyDict, RespFile, + = ToolsDef, resp_file_number) + resp_file_number +=3D 1 TargetDict =3D {"target": self.PlaceMacro(T.Target.Pa= th, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": CCodeDeps} CmdLine =3D self._BUILD_TARGET_TEMPLATE.Replace(Targe= tDict).rstrip().replace('\t$(OBJLIST', '$(OBJLIST') if T.Commands: @@ -1062,7 +1066,9 @@ cleanlib: AnnexeTargetDict =3D {"target": self.PlaceMacro(i= .Path, self.Macros), "cmd": "", "deps": self.PlaceMacro(T.Target.Path, self= .Macros)} self.BuildTargetList.append(self._BUILD_TARGET_TE= MPLATE.Replace(AnnexeTargetDict)) =20 - def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppD= ict, DependencyDict): + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppD= ict, DependencyDict, RespFile, ToolsDef, + resp_file_number): + SaveFilePath =3D os.path.join(RespFile, "cc_resp_%s.txt" % + resp_file_number) if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: CmdSumDict[item.Target.SubDir] =3D item.Target.BaseName @= @ -1089,17 +1095,36 @@ cleanlib: if Temp.startswith('/Fo'): CmdSign =3D '%s%s' % (Temp.rsplit(TAB_SLASH, = 1)[0], TAB_SLASH) break - else: continue + else: + continue if CmdSign not in list(CmdTargetDict.keys()): - CmdTargetDict[CmdSign] =3D Item.replace(Temp, Cmd= Sign) + cmd =3D Item.replace(Temp, CmdSign) + if SingleCommandList[-1] in cmd: + CmdTargetDict[CmdSign] =3D=20 + [cmd.replace(SingleCommandList[-1], "").rstrip(),=20 + SingleCommandList[-1]] else: - CmdTargetDict[CmdSign] =3D "%s %s" % (CmdTargetDi= ct[CmdSign], SingleCommandList[-1]) + # CmdTargetDict[CmdSign] =3D "%s %s" %=20 + (CmdTargetDict[CmdSign], SingleCommandList[-1]) + + CmdTargetDict[CmdSign].append(SingleCommandList[-1]) Index =3D CommandList.index(Item) CommandList.pop(Index) if SingleCommandList[-1].endswith("%s%s.c" % (TAB_SLA= SH, CmdSumDict[CmdSign[3:].rsplit(TAB_SLASH, 1)[0]])): Cpplist =3D CmdCppDict[T.Target.SubDir] Cpplist.insert(0, '$(OBJLIST_%d): ' % list(self.O= bjTargetDict.keys()).index(T.Target.SubDir)) - T.Commands[Index] =3D '%s\n\t%s' % (' \\\n\t'.joi= n(Cpplist), CmdTargetDict[CmdSign]) + source_files =3D CmdTargetDict[CmdSign][1:] + source_files.insert(0, " ") + if len(source_files)>2: + SaveFileOnChange(SaveFilePath, " ".join(sourc= e_files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)= ' % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[CmdSig= n][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + elif len(source_files)<=3D2 and len(" ".join(CmdT= argetDict[CmdSign][:2]))>GlobalData.gCommandMaxLength: + SaveFileOnChange(SaveFilePath, " ".join(sourc= e_files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)= ' % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[Cm= dSign][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + else: + T.Commands[Index] =3D '%s\n\t%s' % ('=20 + \\\n\t'.join(Cpplist), " ".join(CmdTargetDict[CmdSign])) else: T.Commands.pop(Index) return T, CmdSumDict, CmdTargetDict, CmdCppDict diff --git a/Base= Tools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/Au= toGen/IncludesAutoGen.py index c3e6333217c4..b06ef42573fd 100644 --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -201,7 +201,17 @@ ${END} cc_options =3D line[len(cc_cmd)+2:].split() else: cc_options =3D line[len(cc_cmd):].split() - SourceFileAbsPathMap =3D {os.path.basename(item):= item for item in cc_options if not item.startswith("/") and os.path.exists(= item)} + for item in cc_options: + if not item.startswith("/"): + if item.endswith(".txt") and item.startsw= ith("@"): + with open(item[1:], "r") as file: + source_files =3D file.readlines()= [0].split() + SourceFileAbsPathMap =3D {os.path= .basename(file): file for file in source_files if + os.path.e= xists(file)} + else: + if os.path.exists(item): + SourceFileAbsPathMap.update({os.p= ath.basename(item): item.strip()}) + # SourceFileAbsPathMap =3D=20 + {os.path.basename(item):item for item in cc_options if not + item.startswith("/") and os.path.exists(item)} if line in SourceFileAbsPathMap: current_source =3D line if current_source not in ModuleDepDict: -- 2.28.0.windows.1