From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web10.3986.1604558008471357347 for ; Wed, 04 Nov 2020 22:33:28 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=N6TVdEZs; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: mingyuex.liang@intel.com) IronPort-SDR: 4n/jVyAEbUy0tR3ZUDMMiBMYdCuOMbK7DHlg4NCGTOyH8rag8osjUCngGEeASuvvu8DCO5vxeC 3q+wt1sUtdSg== X-IronPort-AV: E=McAfee;i="6000,8403,9795"; a="148615434" X-IronPort-AV: E=Sophos;i="5.77,452,1596524400"; d="scan'208";a="148615434" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Nov 2020 22:33:26 -0800 IronPort-SDR: vSMS/XY83q9uKXqqvS5MT3xd9UTYJtOl2m9QP/ajUScAHvw9pnpgXahhTNEa/H7wx7Q3icicve qBOJP/43pOMQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,452,1596524400"; d="scan'208";a="352936359" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orsmga008.jf.intel.com with ESMTP; 04 Nov 2020 22:33:26 -0800 Received: from orsmsx609.amr.corp.intel.com (10.22.229.22) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 4 Nov 2020 22:33:26 -0800 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX609.amr.corp.intel.com (10.22.229.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 4 Nov 2020 22:33:25 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Wed, 4 Nov 2020 22:33:25 -0800 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (104.47.73.42) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Wed, 4 Nov 2020 22:33:18 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MoerEjIl7patFjmHWrvF5WFryKCN3RBxwl0u66RGV+v6tNUlU9mcXyem16mnWsRbTdTbZlmbnJS6v86ceOmS/w0NpWllqVb3Z0srbG2Erq0JkiBbYYiJtRvkDj9uuBvD4zUvLKHfd1/Q6/3R+weDCt2fztLZUm2W8KXF3rXtEh6T73gJgyIoHAHwFmqWIIq3tWNH+0aCTf953lRKAKDgBjik1N2i1BPHB7p01y5mD23J1FsL0+mqXimx3lsdcu9d11JaL+FA3CAxh/7S3lvGHT1xKKkDNlvSbo8wvX+Th2/pHpmdLUepirdQT2pj4oPP4drlZFd/hqN/50B9za8JYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VCqUtoEGMwBS1etyHtrG7d/fXivzyj6u78ZBaXEWDZY=; b=kD1R/EiZqD2sfPA6r/aiqmu12Y6V76tZCYc79WVmnwBan2x53RXxFZmn/uB800AfmVJf5C7BgFZAh8Q2p+qysMTJNi96BwR2dJjJZ0Hh4X8zAaWlrZsDKiJau5WrvzQOXMsZEJLQDk+h8e8HzuMVIITypOEENVAeJaYPTUbed3DigAYSu2jsVXI5kcOjDdd218J5BSbGxgRIVR2MOGpzgMYeCZyO9eY98te15q0nW9SmXVwHXPR3om15/b8cTOx/KxaAAkNWXxXWlykWpvuQccVz2dE6H0XF3ssm84DkzNhxNuero85NEkvFt9s/sJnKRp0fv9nldnPb8+ypvyl/MQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VCqUtoEGMwBS1etyHtrG7d/fXivzyj6u78ZBaXEWDZY=; b=N6TVdEZsdIGdEKbGcxd2QJd2wA9GRB+pT7820HcPGGtTj1wvV1nL5bVa+NvVYNUfVYrpk9qXQ1WLwcQL0l2U5pRwe8uzg4tHJHq7buYUzqV8FpD36ZDcJPHnFAhFCYAimoHM9812Y57PVQrualqHa9MFTQ+azAKaAldY1KIvgUY= Received: from DM6PR11MB3369.namprd11.prod.outlook.com (2603:10b6:5:b::16) by DM6PR11MB4172.namprd11.prod.outlook.com (2603:10b6:5:19f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.29; Thu, 5 Nov 2020 06:33:17 +0000 Received: from DM6PR11MB3369.namprd11.prod.outlook.com ([fe80::e911:9d94:79ae:cba0]) by DM6PR11MB3369.namprd11.prod.outlook.com ([fe80::e911:9d94:79ae:cba0%4]) with mapi id 15.20.3499.032; Thu, 5 Nov 2020 06:33:17 +0000 From: "Mingyue Liang" To: "Feng, Bob C" , "devel@edk2.groups.io" CC: Liming Gao , "Chen, Christine" , "Vaish, Atul" Subject: Re: [PATCH.V2] BaseTools: Limit command line length. Thread-Topic: [PATCH.V2] BaseTools: Limit command line length. Thread-Index: AQHWsxK/NuXVYUWkSEi8CLw1a7fuLqm41KgAgABALBA= Date: Thu, 5 Nov 2020 06:33:17 +0000 Message-ID: References: <20201105012638.662-1-mingyuex.liang@intel.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.102.204.38] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 142456ed-2899-4f03-a620-08d88154ae82 x-ms-traffictypediagnostic: DM6PR11MB4172: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:419; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: KXY4YZg3F4eIP4vQhMDXGMhWaMJw59KwKppqpOqbau8r6e3c8aarcWUobIDaK2WRTnhogpqUxsO6EYmvaonwDa2DXBoJmiLjzZzJjfBxlBNktHGpBK4A1IvSRuz7asTWqr6ZhtIOJMf/czGxf3HQNqdIBbQzwuIV2+ynKGpbDubWcbJ+RXJ2W7U7NcDAoBh7PX/+M2vneSYAeD3ip0dwnQVKfu4I7by2KssDJSx41pAw1gynpYG0JcK/PrbbVDEzShhozUnIXGPe+eVua4GcR6OMo5dLOTBWmN1W68J/OuYI0qjr+QXXM81yrpBHvzXJmIgGxNrBSeThEcMzP/c+wfLLM0nZnqhTkuYPJPl28rFV+ALWAkILLU4ZqN+/t88D+XGlhaUe3rtMcwMQ0bwqoQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR11MB3369.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(346002)(396003)(136003)(366004)(2906002)(66946007)(71200400001)(64756008)(33656002)(66446008)(66476007)(66556008)(76116006)(55016002)(52536014)(110136005)(26005)(8936002)(54906003)(4326008)(8676002)(966005)(83380400001)(316002)(186003)(5660300002)(9686003)(6506007)(53546011)(478600001)(7696005)(107886003)(86362001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: nSEtxZZk7oa53n4KCTgCxGwTfNFygCMRKkNLm9GVb+AFwUezZG/RayWu2WzK6d8Hp9QurL+efvIt9FLfXYimV6VAS8A8ma4apCCceWkpO0/yoUwffGDlYkYdJow5Jof12vWCDxeHMug2seouBzexSolOj1WOJcQ4SLbuapKmbyfnsId3cYHfZ7N4Pe/jktSm/ilvV6qpwsFysoLVzrgMx0PkkMgGtBMQkV2fKMcZgjiHG92jEe/mzG3v3EtdGH8XCOJkOcctwOkFX02Hevrtx4UcT7n/RrsRuUQfsKAq7vhv3HXTCJV+J820Fpg348Yz6EHHP6F7zBQbJnaGEtz40n4PzWExId1euFfmrDLZ1sIEVNSGIpgoAeAe/e7dijVaw5fpiORj1xOy8S/KectJA8fphFjFqW2p2MgV3C6mPmufBZnSzuN8ZHrViK6RDlk7sZ9OtHc906UBK/7r2JwsFvYlbo6cNpd/9XcWYhHoP1qzJBTjjd+3LWBMHURh6F/UZvUl/G+cq8gWgIy/VsCXTGXo3MIrQ/DTCq4XWw9lY2S3A2fwdDw5re3lc9Uipnet9n+jAuc3db8x6rykFIqCh0ydr8tE+oXNHMGsdHlaZHTmm7E483SnbzSq7WN4xGzepQ1jYc6fPzKLhrig0xv+tw== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB3369.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 142456ed-2899-4f03-a620-08d88154ae82 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Nov 2020 06:33:17.6729 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: PrXwrztLnK7QAEKsFsOcxqgrPave8vE8CV4N/GSzOGPfrdyh3EvHd4wQIQgYQ8HF8mlNHAwhNAHiZ46urKWSh/bdyJiZyarHT/D27q/00qY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4172 Return-Path: mingyuex.liang@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Bob, This method is not recommended and will result in build errors. Thanks, Mingyue -----Original Message----- From: Feng, Bob C =20 Sent: Thursday, November 5, 2020 10:43 AM To: Liang, MingyueX ; devel@edk2.groups.io Cc: Liang, MingyueX ; Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Mingyue, Would you update the cc_resp_* file content? One c file one line. Thanks, Bob -----Original Message----- From: Mingyue Liang Sent: Thursday, November 5, 2020 9:27 AM To: devel@edk2.groups.io Cc: Liang, MingyueX ; Feng, Bob C ; Liming Gao ; Chen, Christine Subject: [PATCH.V2] BaseTools: Limit command line length. From: mliang2x REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2528 Currently, CL command contains multiple C files will be compiled, and that = caused command line too long, which may trigger build error. In order to solve this issue, the following rules is used in this scene: If the number of C files is greater than one, a txt file will be used to re= cord these C files, and replaces the corresponding content in command line = with the file name. Else (only one C file listed in the command line), the length of the whole = CL command line will determine whether use a file to record. If the length = exceeds the limited max length, use the recording file; else C file name di= rectly listed in the command line V2: 1,Updated patch description 2,Changing the storage mode of C file is easy to obtain Signed-off-by: Mingyue Liang Cc: Bob Feng Cc: Liming Gao Cc: Yuwei Chen --- BaseTools/Source/Python/AutoGen/GenMake.py | 41 +++++++++++++++---- .../Source/Python/AutoGen/IncludesAutoGen.py | 12 +++++- 2 files changed, 44 insertions(+), 9 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index b04d3f54368e..1cfac1cd82ca 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -576,7 +576,7 @@ cleanlib: EdkLogger.error("build", AUTOGEN_ERROR, "Nothing to build", ExtraData=3D"[%s]" % str(MyAgo)) =20 - self.ProcessBuildTargetList() + self.ProcessBuildTargetList(MyAgo.OutputDir, ToolsDef) self.ParserGenerateFfsCmd() =20 # Generate macros used to represent input files @@ -903,7 +903,7 @= @ cleanlib: BuildTargets[Target].Commands[i] =3D S= ingleCommand.replace('$(INC)', '').replace(FlagDict[Flag]['Macro'], RespMac= ro) return RespDict =20 - def ProcessBuildTargetList(self): + def ProcessBuildTargetList(self, RespFile, ToolsDef): # # Search dependency file list for each source file # @@ -1004,6 +1004,7 @@ cleanlib: self.ObjTargetDict[T.Target.SubDir] =3D set() self.ObjTargetDict[T.Target.SubDir].add(NewFile) for Type in self._AutoGenObject.Targets: + resp_file_number =3D 0 for T in self._AutoGenObject.Targets[Type]: # Generate related macros if needed if T.GenFileListMacro and T.FileListMacro not in self.File= ListMacros: @@ -1045,7 +1046,10 @@ cleanlib: Deps.append("$(%s)" % T.ListFileMacro) =20 if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMPILER= _MSFT and Type =3D=3D TAB_C_CODE_FILE: - T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Parse= rCCodeFile(T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict) + T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Parse= rCCodeFile(T, Type, CmdSumDict, CmdTargetDict, + = CmdCppDict, DependencyDict, RespFile, + = ToolsDef, resp_file_number) + resp_file_number +=3D 1 TargetDict =3D {"target": self.PlaceMacro(T.Target.Pat= h, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": CCodeDeps} CmdLine =3D self._BUILD_TARGET_TEMPLATE.Replace(Target= Dict).rstrip().replace('\t$(OBJLIST', '$(OBJLIST') if T.Commands: @@ -1062,7 +1066,9 @@ cleanlib: AnnexeTargetDict =3D {"target": self.PlaceMacro(i.= Path, self.Macros), "cmd": "", "deps": self.PlaceMacro(T.Target.Path, self.= Macros)} self.BuildTargetList.append(self._BUILD_TARGET_TEM= PLATE.Replace(AnnexeTargetDict)) =20 - def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDi= ct, DependencyDict): + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDi= ct, DependencyDict, RespFile, ToolsDef, + resp_file_number): + SaveFilePath =3D os.path.join(RespFile, "cc_resp_%s.txt" % + resp_file_number) if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: CmdSumDict[item.Target.SubDir] =3D item.Target.BaseName @@= -1089,17 +1095,36 @@ cleanlib: if Temp.startswith('/Fo'): CmdSign =3D '%s%s' % (Temp.rsplit(TAB_SLASH, 1= )[0], TAB_SLASH) break - else: continue + else: + continue if CmdSign not in list(CmdTargetDict.keys()): - CmdTargetDict[CmdSign] =3D Item.replace(Temp, CmdS= ign) + cmd =3D Item.replace(Temp, CmdSign) + if SingleCommandList[-1] in cmd: + CmdTargetDict[CmdSign] =3D=20 + [cmd.replace(SingleCommandList[-1], "").rstrip(),=20 + SingleCommandList[-1]] else: - CmdTargetDict[CmdSign] =3D "%s %s" % (CmdTargetDic= t[CmdSign], SingleCommandList[-1]) + # CmdTargetDict[CmdSign] =3D "%s %s" %=20 + (CmdTargetDict[CmdSign], SingleCommandList[-1]) + =20 + CmdTargetDict[CmdSign].append(SingleCommandList[-1]) Index =3D CommandList.index(Item) CommandList.pop(Index) if SingleCommandList[-1].endswith("%s%s.c" % (TAB_SLAS= H, CmdSumDict[CmdSign[3:].rsplit(TAB_SLASH, 1)[0]])): Cpplist =3D CmdCppDict[T.Target.SubDir] Cpplist.insert(0, '$(OBJLIST_%d): ' % list(self.Ob= jTargetDict.keys()).index(T.Target.SubDir)) - T.Commands[Index] =3D '%s\n\t%s' % (' \\\n\t'.join= (Cpplist), CmdTargetDict[CmdSign]) + source_files =3D CmdTargetDict[CmdSign][1:] + source_files.insert(0, " ") + if len(source_files)>2: + SaveFileOnChange(SaveFilePath, " ".join(source= _files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)'= % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[CmdSign= ][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + elif len(source_files)<=3D2 and len(" ".join(CmdTa= rgetDict[CmdSign][:2]))>GlobalData.gCommandMaxLength: + SaveFileOnChange(SaveFilePath, " ".join(source= _files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)'= % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[Cmd= Sign][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + else: + T.Commands[Index] =3D '%s\n\t%s' % ('=20 + \\\n\t'.join(Cpplist), " ".join(CmdTargetDict[CmdSign])) else: T.Commands.pop(Index) return T, CmdSumDict, CmdTargetDict, CmdCppDict diff --git a/BaseT= ools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/Aut= oGen/IncludesAutoGen.py index c3e6333217c4..b06ef42573fd 100644 --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -201,7 +201,17 @@ ${END} cc_options =3D line[len(cc_cmd)+2:].split() else: cc_options =3D line[len(cc_cmd):].split() - SourceFileAbsPathMap =3D {os.path.basename(item):i= tem for item in cc_options if not item.startswith("/") and os.path.exists(i= tem)} + for item in cc_options: + if not item.startswith("/"): + if item.endswith(".txt") and item.startswi= th("@"): + with open(item[1:], "r") as file: + source_files =3D file.readlines()[= 0].split() + SourceFileAbsPathMap =3D {os.path.= basename(file): file for file in source_files if + os.path.ex= ists(file)} + else: + if os.path.exists(item): + SourceFileAbsPathMap.update({os.pa= th.basename(item): item.strip()}) + # SourceFileAbsPathMap =3D=20 + {os.path.basename(item):item for item in cc_options if not + item.startswith("/") and os.path.exists(item)} if line in SourceFileAbsPathMap: current_source =3D line if current_source not in ModuleDepDict: -- 2.28.0.windows.1