From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web11.7896.1604902851617445424 for ; Sun, 08 Nov 2020 22:20:51 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=Ee8jhrqw; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: bob.c.feng@intel.com) IronPort-SDR: EWmTsRgUqXslEZ9hSLVVPHanjRIePAaBBVzZAY+SoiFLi8Pm6XqdGvAdga5bvg3tgjnF/gehIX 8HxDWRjlkzBw== X-IronPort-AV: E=McAfee;i="6000,8403,9799"; a="149034402" X-IronPort-AV: E=Sophos;i="5.77,462,1596524400"; d="scan'208";a="149034402" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Nov 2020 22:20:51 -0800 IronPort-SDR: MLk2xIxac9YQ1sZtB6Xq2nIiGA7XfePNowPcIcncVaxVcDTotTi/GXkeHUZvtDFdRCgtYN2e1J em7VI01H/nXQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,462,1596524400"; d="scan'208";a="355537577" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmsmga004.fm.intel.com with ESMTP; 08 Nov 2020 22:20:51 -0800 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Sun, 8 Nov 2020 22:20:50 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Sun, 8 Nov 2020 22:20:50 -0800 Received: from NAM02-BL2-obe.outbound.protection.outlook.com (104.47.38.52) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Sun, 8 Nov 2020 22:20:50 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m8bxKvxNPpIfXAjAeFsSUWEWA5UChEz2yDAMsdOlyc9N/SVMlAa8+I7+CiVuDkqGMzkKcI6VUEyBbBtTx8ylG+I4ETJ8Q+rBkPCbwRYcsC5WDoaAFW5p+b9GbNF1NvPX+nKFGSvzIaWsKoFVOyUxFolSe07Xfa6f0uTiWTRgF7if5eXpyVjle88DcXvmAJ1hioMk0+kXvQz7y40HMhoatQOZUlpV5ZP/14/OHoEsEveYGNSLMn5l/gVsST5UspNpyPp5hGy1pKmZigEOM8YYYEqMzRI7HtQ3wkqggwSkWWLsxOwq3SniIF6RBb7PeJ0k6blOTacplcNFO1mYIWFlPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5wx45dpeEWZWT84xQLaBRquH0fBtieZyD0vwUQtpYRA=; b=YlLj/3CaCf0YDxwCcEk52h0u+DapG2sdLOIKi78nFrfogHyEukgzuu6frndgX8p8+qj5Qbhi9xf3ZlJMyL+EZbVoHC7tXWJf4osPa3ILkY/IqdM/Ov7uz4xCFh+u8Nuh1VC/pXC/Q7xdAcj4eG6szDA3ccycxfc5zgM/DwfvKXtW81yOz/cnt7VqQ8y/650+03SdkKlYW++kuVwSsg09UG0reg1F4LY+YJgft6BjSJ/U9kNtKRffbxTbQd54tRB/M3CwaT6kbMS3KGmyAgAZ3xYAmdEqlI/Cr/u1eDcGHdBL7E7cPma65JemdkIWKf3Bm9Lx8Cb8pUIiUzfPSHB9eA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5wx45dpeEWZWT84xQLaBRquH0fBtieZyD0vwUQtpYRA=; b=Ee8jhrqw8PTIcMgy/mjtYqwUVRRFNj35pMBtwuHghlkqMUn9IVD+neNfdVQcKgSNo2TBTiiEPC6mnSK4G07w2wha9NM9V8s2Djb9YfY2iHytciHVab3Z/WMvbVfLqCzzVS3M6sooxuo61jxq0lyanigSDmqS+usuxBVXdCK6o00= Received: from DM6PR11MB4073.namprd11.prod.outlook.com (2603:10b6:5:19f::22) by DM6PR11MB2715.namprd11.prod.outlook.com (2603:10b6:5:be::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.21; Mon, 9 Nov 2020 06:20:48 +0000 Received: from DM6PR11MB4073.namprd11.prod.outlook.com ([fe80::c9f:47ea:6321:5ccb]) by DM6PR11MB4073.namprd11.prod.outlook.com ([fe80::c9f:47ea:6321:5ccb%3]) with mapi id 15.20.3541.022; Mon, 9 Nov 2020 06:20:48 +0000 From: "Bob Feng" To: "Liang, MingyueX" , "devel@edk2.groups.io" CC: Liming Gao , "Chen, Christine" , "Vaish, Atul" Subject: Re: [PATCH.V2] BaseTools: Limit command line length. Thread-Topic: [PATCH.V2] BaseTools: Limit command line length. Thread-Index: AQHWsxK/gedGv+mhpky4pZTiMyYLJKm41EFggABAsICABkWtIA== Date: Mon, 9 Nov 2020 06:20:48 +0000 Message-ID: References: <20201105012638.662-1-mingyuex.liang@intel.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.55.46.46] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bcc66077-f0e1-4146-5a48-08d8847799c4 x-ms-traffictypediagnostic: DM6PR11MB2715: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:419; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nAfVqjBzLqtFcjDne/Z/ybJ2h/ieUdoEuwJk4OxIa0BrITXvrgv3AVKUs4Ma4UX8WC8p/iJycE5A5Ed9s/egUZby5CfBVLY5uMMedBnF3/OxsrRCh+2G+fCMY/KRUbznUyQ4Aze0S2l6zDHTDlV7sRii/4qyvQ5d2/71ORKKiWMeg0ZgUAue6gz8hf3P4PDZeMagJHLxm0P0kAR0J6V0STprzySWy2tp8oDUyRTBa7T05dkte5diQzZp90+LIrMmco7y/OY3nwOwhZNb9OQ+qb5OVqONEk9M7X8/ebvvkxFk+dd4zVcwrADS6DgZ/c+8K2sz+d/iHrsMC42OoyvuNvQcSVjgLtKK8WyquR7lWBgd42a+4KOy9zsrhqePD9GL8xWLqk9kWMSC/l12eImqqA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR11MB4073.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(136003)(396003)(366004)(346002)(376002)(66946007)(110136005)(71200400001)(54906003)(186003)(2906002)(52536014)(478600001)(316002)(5660300002)(86362001)(8676002)(64756008)(66446008)(76116006)(66556008)(8936002)(83380400001)(9686003)(966005)(33656002)(7696005)(107886003)(4326008)(55016002)(26005)(53546011)(6506007)(66476007);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: HLQob211b37foLkJn9fZcgUVq0Z9LWw4yF4Lkkm/IyB3Y4aNerCdL+cdU2MXF8pZigaBvI2kucUKq8EOMpXdGw2l/kXkSZ3zer7H48WwTi7gCwIJ+7O/VqqHCW38+3QxTuZgU+J+lNz7mLqGVhuo0ecH4lzDY/fO82rOYlOiwn8Bgogpm8qeYqpn4clLmBPrCTjTA/8pQZiYfBvFvVJpTir1gNeEgKuK7kXOBKmcF1uW+pletP8R++Jy1zkM1xIHZOU9cM+MguJ9niFceoYfc12GMGrucEbnACcthkg3dbV6b1RttRYC0vDG0LIHRyzPDYLRFNJjsWAdcYVIJcU7iZNpZKBKcw2YVesPiHoD2zbYe75uHN5XGL1/OBvI1KBWUklOWpQh/QX5qUt84HkrMnLywmC3lfqrjJJVeYfDOvjeu1Ic8bi3nDY1hb2In8q+G/Fy23twlSQLo1L+Hj0+HE7OcW++76vEk96Mdc4G1zlR9vlxrbfnLblYgowBRWNHCOiG2wBXddqr6q818edQ6ucz6/wwWDY1aP/lg0feYB4kDir1jqGUopz5+1BTORNJsFr/a+bapUpJ72b8+S9oXrNh4hxcfBR+ltHTLrwoIX/xw4scobF3CooUNQ1Pz5m/sQX6TNu56vKyG035tDimcQ== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4073.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: bcc66077-f0e1-4146-5a48-08d8847799c4 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Nov 2020 06:20:48.7922 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /5bOf34QgnnaPALZFhBUCClFq5gm575X8OV2aQ9Sq8sSOt3IwJiKSrFIjc3fCn4iiTxb0bvxl8fYKZfzu4w25Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB2715 Return-Path: bob.c.feng@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable OK.=20 Reviewed-by: Bob Feng -----Original Message----- From: Liang, MingyueX =20 Sent: Thursday, November 5, 2020 2:33 PM To: Feng, Bob C ; devel@edk2.groups.io Cc: Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Bob, This method is not recommended and will result in build errors. Thanks, Mingyue -----Original Message----- From: Feng, Bob C Sent: Thursday, November 5, 2020 10:43 AM To: Liang, MingyueX ; devel@edk2.groups.io Cc: Liang, MingyueX ; Liming Gao ; Chen, Christine ; Vaish, Atul Subject: RE: [PATCH.V2] BaseTools: Limit command line length. Hi Mingyue, Would you update the cc_resp_* file content? One c file one line. Thanks, Bob -----Original Message----- From: Mingyue Liang Sent: Thursday, November 5, 2020 9:27 AM To: devel@edk2.groups.io Cc: Liang, MingyueX ; Feng, Bob C ; Liming Gao ; Chen, Christine Subject: [PATCH.V2] BaseTools: Limit command line length. From: mliang2x REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2528 Currently, CL command contains multiple C files will be compiled, and that = caused command line too long, which may trigger build error. In order to solve this issue, the following rules is used in this scene: If the number of C files is greater than one, a txt file will be used to re= cord these C files, and replaces the corresponding content in command line = with the file name. Else (only one C file listed in the command line), the length of the whole = CL command line will determine whether use a file to record. If the length = exceeds the limited max length, use the recording file; else C file name di= rectly listed in the command line V2: 1,Updated patch description 2,Changing the storage mode of C file is easy to obtain Signed-off-by: Mingyue Liang Cc: Bob Feng Cc: Liming Gao Cc: Yuwei Chen --- BaseTools/Source/Python/AutoGen/GenMake.py | 41 +++++++++++++++---- .../Source/Python/AutoGen/IncludesAutoGen.py | 12 +++++- 2 files changed, 44 insertions(+), 9 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index b04d3f54368e..1cfac1cd82ca 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -576,7 +576,7 @@ cleanlib: EdkLogger.error("build", AUTOGEN_ERROR, "Nothing to build", ExtraData=3D"[%s]" % str(MyAgo)) =20 - self.ProcessBuildTargetList() + self.ProcessBuildTargetList(MyAgo.OutputDir, ToolsDef) self.ParserGenerateFfsCmd() =20 # Generate macros used to represent input files @@ -903,7 +903,7 @= @ cleanlib: BuildTargets[Target].Commands[i] =3D S= ingleCommand.replace('$(INC)', '').replace(FlagDict[Flag]['Macro'], RespMac= ro) return RespDict =20 - def ProcessBuildTargetList(self): + def ProcessBuildTargetList(self, RespFile, ToolsDef): # # Search dependency file list for each source file # @@ -1004,6 +1004,7 @@ cleanlib: self.ObjTargetDict[T.Target.SubDir] =3D set() self.ObjTargetDict[T.Target.SubDir].add(NewFile) for Type in self._AutoGenObject.Targets: + resp_file_number =3D 0 for T in self._AutoGenObject.Targets[Type]: # Generate related macros if needed if T.GenFileListMacro and T.FileListMacro not in self.File= ListMacros: @@ -1045,7 +1046,10 @@ cleanlib: Deps.append("$(%s)" % T.ListFileMacro) =20 if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMPILER= _MSFT and Type =3D=3D TAB_C_CODE_FILE: - T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Parse= rCCodeFile(T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict) + T, CmdTarget, CmdTargetDict, CmdCppDict =3D self.Parse= rCCodeFile(T, Type, CmdSumDict, CmdTargetDict, + = CmdCppDict, DependencyDict, RespFile, + = ToolsDef, resp_file_number) + resp_file_number +=3D 1 TargetDict =3D {"target": self.PlaceMacro(T.Target.Pat= h, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": CCodeDeps} CmdLine =3D self._BUILD_TARGET_TEMPLATE.Replace(Target= Dict).rstrip().replace('\t$(OBJLIST', '$(OBJLIST') if T.Commands: @@ -1062,7 +1066,9 @@ cleanlib: AnnexeTargetDict =3D {"target": self.PlaceMacro(i.= Path, self.Macros), "cmd": "", "deps": self.PlaceMacro(T.Target.Path, self.= Macros)} self.BuildTargetList.append(self._BUILD_TARGET_TEM= PLATE.Replace(AnnexeTargetDict)) =20 - def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDi= ct, DependencyDict): + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDi= ct, DependencyDict, RespFile, ToolsDef, + resp_file_number): + SaveFilePath =3D os.path.join(RespFile, "cc_resp_%s.txt" % + resp_file_number) if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: CmdSumDict[item.Target.SubDir] =3D item.Target.BaseName @@= -1089,17 +1095,36 @@ cleanlib: if Temp.startswith('/Fo'): CmdSign =3D '%s%s' % (Temp.rsplit(TAB_SLASH, 1= )[0], TAB_SLASH) break - else: continue + else: + continue if CmdSign not in list(CmdTargetDict.keys()): - CmdTargetDict[CmdSign] =3D Item.replace(Temp, CmdS= ign) + cmd =3D Item.replace(Temp, CmdSign) + if SingleCommandList[-1] in cmd: + CmdTargetDict[CmdSign] =3D=20 + [cmd.replace(SingleCommandList[-1], "").rstrip(),=20 + SingleCommandList[-1]] else: - CmdTargetDict[CmdSign] =3D "%s %s" % (CmdTargetDic= t[CmdSign], SingleCommandList[-1]) + # CmdTargetDict[CmdSign] =3D "%s %s" %=20 + (CmdTargetDict[CmdSign], SingleCommandList[-1]) + =20 + CmdTargetDict[CmdSign].append(SingleCommandList[-1]) Index =3D CommandList.index(Item) CommandList.pop(Index) if SingleCommandList[-1].endswith("%s%s.c" % (TAB_SLAS= H, CmdSumDict[CmdSign[3:].rsplit(TAB_SLASH, 1)[0]])): Cpplist =3D CmdCppDict[T.Target.SubDir] Cpplist.insert(0, '$(OBJLIST_%d): ' % list(self.Ob= jTargetDict.keys()).index(T.Target.SubDir)) - T.Commands[Index] =3D '%s\n\t%s' % (' \\\n\t'.join= (Cpplist), CmdTargetDict[CmdSign]) + source_files =3D CmdTargetDict[CmdSign][1:] + source_files.insert(0, " ") + if len(source_files)>2: + SaveFileOnChange(SaveFilePath, " ".join(source= _files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)'= % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[CmdSign= ][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + elif len(source_files)<=3D2 and len(" ".join(CmdTa= rgetDict[CmdSign][:2]))>GlobalData.gCommandMaxLength: + SaveFileOnChange(SaveFilePath, " ".join(source= _files), False) + T.Commands[Index] =3D '%s\n\t%s $(cc_resp_%s)'= % ( + ' \\\n\t'.join(Cpplist), CmdTargetDict[Cmd= Sign][0], resp_file_number) + ToolsDef.append("cc_resp_%s =3D @%s" %=20 + (resp_file_number, SaveFilePath)) + + else: + T.Commands[Index] =3D '%s\n\t%s' % ('=20 + \\\n\t'.join(Cpplist), " ".join(CmdTargetDict[CmdSign])) else: T.Commands.pop(Index) return T, CmdSumDict, CmdTargetDict, CmdCppDict diff --git a/BaseT= ools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/Aut= oGen/IncludesAutoGen.py index c3e6333217c4..b06ef42573fd 100644 --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -201,7 +201,17 @@ ${END} cc_options =3D line[len(cc_cmd)+2:].split() else: cc_options =3D line[len(cc_cmd):].split() - SourceFileAbsPathMap =3D {os.path.basename(item):i= tem for item in cc_options if not item.startswith("/") and os.path.exists(i= tem)} + for item in cc_options: + if not item.startswith("/"): + if item.endswith(".txt") and item.startswi= th("@"): + with open(item[1:], "r") as file: + source_files =3D file.readlines()[= 0].split() + SourceFileAbsPathMap =3D {os.path.= basename(file): file for file in source_files if + os.path.ex= ists(file)} + else: + if os.path.exists(item): + SourceFileAbsPathMap.update({os.pa= th.basename(item): item.strip()}) + # SourceFileAbsPathMap =3D=20 + {os.path.basename(item):item for item in cc_options if not + item.startswith("/") and os.path.exists(item)} if line in SourceFileAbsPathMap: current_source =3D line if current_source not in ModuleDepDict: -- 2.28.0.windows.1