From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mx.groups.io with SMTP id smtpd.web11.6541.1592297337584271446 for ; Tue, 16 Jun 2020 01:48:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=OzVITJs0; spf=pass (domain: intel.com, ip: 134.134.136.100, mailfrom: bob.c.feng@intel.com) IronPort-SDR: sOZjWbsfzwcuptAidVQBQuoOpKdJQiSVqsrf4WIjR4FtfVI644SUoz07azSrNR+ici+gK3PX+m P2/nbxg4y0XQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2020 01:48:56 -0700 IronPort-SDR: fzBYiZ81rr4fwTat9ptveQePKff+gx1q/d5rjWz+ftnTXVqLNMCV53vQBkEHckxMHffw4JCR7k LNmHTdqY9+1A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,518,1583222400"; d="scan'208";a="309038181" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga002.fm.intel.com with ESMTP; 16 Jun 2020 01:48:56 -0700 Received: from fmsmsx605.amr.corp.intel.com (10.18.126.85) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 16 Jun 2020 01:48:56 -0700 Received: from fmsmsx605.amr.corp.intel.com (10.18.126.85) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 16 Jun 2020 01:48:55 -0700 Received: from FMSEDG001.ED.cps.intel.com (10.1.192.133) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Tue, 16 Jun 2020 01:48:55 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.105) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 16 Jun 2020 01:48:53 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bX+vlhQBuJ5rYx3fQZtdJml7ngqRTBqLKDafV43Wz9mNk3fqtv/oSY/gwTJujM+6pZnP3zfW31bTBCbeBkoMmbZqNtIqgQG4RnhXmXbIZHu+STaiX70W1mrZiG3nBf0UFl/XmEcs9c2vBg3AfjqEpFfrPiO3jpnD+fO+nznbWIaYNGIDQbDSt5OmC6YCZm+Q2gh/eUXb4gtaGCECj8H5KKl9nsBzA9Sgxkil3T8eEwbwaPTcnkIpvIzmE6PvByAXbah8edDCmr3NVRoHAkAp7vIrbPbemp3iaXky/70KZ7JcaFMnBD/1wwt1kEXR8CAAMzSXIAZdHjoar9qKRkExbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hie18m/qV6gWGG76z8014j4rPDNldibM+MNcRPgTwZI=; b=TUVYD11dBAtNugsp+11aPbNpSYNgllkQBGn8hUWE+3qUg4j6Npt3/KuiLyXML2UdKDYKZrmoxWHayFVE65UYtCXiy1MBXm5kx2aKkO3fh80HwDAszv5dmf7gNx1OM/4ILwSUv9Gp8eVq+Df70fJn7BUDacqwUhKnbPfU7BRGxZYlc3fkbyNu3G+3LPR2XZrXxA3W9LFrbmc+zQ/BuGZKfUl0DMS1K+9o3m3qqg8Um7qKdPyfY4/2eyz5RhoVEfwxIwBmTwGovO+XdqdkjFa2rvQ3VraW6MMXQJOpDIQr5r36x6GAW9Oj6M7KJicJ1lmO69rJfhWgfuf+fyTwM5LPdg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hie18m/qV6gWGG76z8014j4rPDNldibM+MNcRPgTwZI=; b=OzVITJs0hfK5eZAq6lV9pI5Zyw/HgBRBFAq6TLX1/LwDCkwN45y+Mg9qgyxcBTT9mxfRxeLqX55NG3m4wuPkrUHzMigfbtUuCsHfK4K06VksfHL3r5UE+QODqMIqrSsm4lJ2F7jEuJ0P4r4hjO4xCtURd59pISZwYfDZ8FU2v9s= Received: from BN6PR11MB0068.namprd11.prod.outlook.com (2603:10b6:405:69::17) by BN6PR11MB1556.namprd11.prod.outlook.com (2603:10b6:405:d::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.21; Tue, 16 Jun 2020 08:48:52 +0000 Received: from BN6PR11MB0068.namprd11.prod.outlook.com ([fe80::6870:82d3:5bde:4153]) by BN6PR11MB0068.namprd11.prod.outlook.com ([fe80::6870:82d3:5bde:4153%6]) with mapi id 15.20.3088.028; Tue, 16 Jun 2020 08:48:51 +0000 From: "Bob Feng" To: Pierre Gondois , "devel@edk2.groups.io" CC: "Gao, Liming" , Sami Mujawar , Tomas Pilar , nd Subject: Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Thread-Topic: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Thread-Index: AQHWLR5Z/G9BS65hW0CiAqbPj8x3x6jFQTvAgAAhEICACX3vgIACcE3AgAiwlQCAARf9EA== Date: Tue, 16 Jun 2020 08:48:51 +0000 Message-ID: References: <20200518141120.66132-1-pierre.gondois@arm.com> <20200518141120.66132-2-pierre.gondois@arm.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.55.46.36] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d2243e21-5fcc-47c0-4298-08d811d2182b x-ms-traffictypediagnostic: BN6PR11MB1556: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 04362AC73B x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nQwGvghWcmhkY4Ro7ggJmoC8JvI6iVyfxALQN7QmysLCgZ1S/tIvhSeOUne0dQmNtkhu2MRS+EYYtOkvhK816UikRR1PKoknc5wyjtl90IZCvsI890MrzCd2eL0uBHZfmXgvaoTjZ2iL7S9RCQW8tjLYPdQ/Slbe/m/F3BLIkaajDD9161nVGpCY3/FQKsbU523+j1+GPxUKck77hFcwbSxBiqxwqA/QPCsVXJV1gH2nQ9jidWqdrYxurmsIfNqJEfwLmr15+ZEgm2+gj/Si+SgvK0OnARFSSgzTaUjkY3BmVcc8oZnrwwg85pXEnaDbHVKQkkWT97vLJJoQpKcIhbClmSPoB9vulESF4IZXm7jIEcTZLVRwTpbC0Wk1+eb7o04n/RGiZAX61p5ahJVdPg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BN6PR11MB0068.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(136003)(396003)(366004)(376002)(39860400002)(346002)(2906002)(83380400001)(33656002)(966005)(4326008)(9686003)(52536014)(5660300002)(30864003)(55016002)(6506007)(53546011)(76116006)(66446008)(64756008)(186003)(316002)(8936002)(26005)(66556008)(66946007)(66476007)(71200400001)(86362001)(478600001)(8676002)(110136005)(54906003)(7696005);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: GCAQvMNlMzzgv2cRxgFYIevLAOvkT4X3CJZkZ7F3SIwED+zIXIYQmebr6mX3kEsLc0w9wLxcpOspc2VKji+roZB43Li9+aiS7er/TSoOEVo3yCWQkwdGYatSyQt9p+YrZsbGkJdx7C/QE06o/74lhcF47T/m2mluul4o+//nz+QCUZ3fbmBi8c4GXu0+stKUwnPbU7ASOa2pA/oM7x78qRd279Dz0WyoxDhj8Cw9Eyp+xwHhyGl3QcIJPo4Y+Be2Cip7O86L6SKEtjGuDvOE9ivq7S+CayebRlZe46YDAPEXaW/R0rciqbYJBlY4WjUmTBtTkeTmQ5dMBwuNxx0lTsxsKCjkqCO4FA7YIYbTPk3o0cOz7nhkmqNCSbIFNP2ZzISYssws5klRCTobbf5V2I7jt4UR9xFjfiLcSJ7p5LHZlAeSHZTasGsemdQESdbgUMV4clQi3BJFwOjc17t0idATyo3faRAT80S+G/YZCeQ= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: d2243e21-5fcc-47c0-4298-08d811d2182b X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jun 2020 08:48:51.7218 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: tQzgMrTSr66oqJSrlfeRsZlAoUZskJMUBhyFUx8Jkwh7Psjz8SMWqvm0PXJvH7Js57Cgq1xVj/VOmUopku/vjg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1556 Return-Path: bob.c.feng@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Pierre, Yes. It's fine for me.=20 Thanks, Bob -----Original Message----- From: Pierre Gondois =20 Sent: Monday, June 15, 2020 11:58 PM To: Feng, Bob C ; devel@edk2.groups.io Cc: Gao, Liming ; Sami Mujawar ; Tomas Pilar ; nd Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rule= s when multiple output files Hello Bob, I have locally corrected the patch. On non-arm architecture, I had previou= sly tested it for the following configurations, but this wasn't enough appa= rently. I am currently testing it on more platforms and I will send you the= list of the tested platforms along with the V2. Plarform=3D"Platform/Intel/MinPlatformPkg/MinPlatformPkg.dsc"|"Platform/In= tel/BoardModulePkg/BoardModulePkg.dsc" Host=3DWindows|Linux BuildType=3DDEBUG|RELEASE Toolchain=3DVS2017|GCC5 Arch=3DX64|IA32 Before submitting a V2, I am planning to create a pull request on the edk2= github repository, allowing me to check whether it passes the CI tests. Wo= uld it be fine for you? About your comment about changing the logic of ApplyBuildRule, the logic o= f _ApplyBuildRule currently breaks out of the loop whenever a final target = is found. The new rule on ".amli" files needs to continue looking for rules= to apply, otherwise the processing on the initial ASL file stops when the = AML file is generated (thus, not continuing the path ".amli"->".c"->".obj")= . I agree this is hard to check, but I believe the edk2 CI tests should all= ow to put more trust in the patch serie. Regards, Pierre -----Original Message----- From: Feng, Bob C Sent: 10 June 2020 04:58 To: Pierre Gondois ; devel@edk2.groups.io Cc: Gao, Liming ; Sami Mujawar ; Tomas Pilar ; nd Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rule= s when multiple output files Hi Pierre, I found there is an incorrect target generated in the OvmfPkg/AcpiTables M= akefile when I tried to build Ovmf. That incorrect target causes ovmf to bu= ild failed. $(DEBUG_DIR)\PlatformAcpiTables : $(MAKE_FILE) $(DEBUG_DIR)\PlatformAcpiTa= bles : $(STATIC_LIBRARY_FILES) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_L= IBRARY_FILES_LIST) "$(DLINK)" $(DLINK_FLAGS) $(DLINK_SPATH) @$(STATIC_LIBRARY_FILES_LIST) Regarding this patch, I think it changes the logic of the _ApplyBuildRule,= replacing "break" with "continue" and removing some if and elseif blocks, = so it would be hard for me to make sure your current logic can cover all th= e original cases. Would you show me how many testing you have done? Thanks, Bob=20 -----Original Message----- From: Pierre Gondois Sent: Monday, June 8, 2020 10:01 PM To: Feng, Bob C ; devel@edk2.groups.io Cc: Gao, Liming ; Sami Mujawar ; Tomas Pilar ; nd Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rule= s when multiple output files Hello Bob, Should I modify the patch ? Regards, Pierre -----Original Message----- From: Pierre Gondois Sent: Tuesday, June 2, 2020 2:04 PM To: Feng, Bob C ; devel@edk2.groups.io Cc: Gao, Liming ; Sami Mujawar ; Tomas Pilar ; nd Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rule= s when multiple output files Hello Bob, Thank you for your answer, I put my comments as [Pierre], Regards, Pierre -----Original Message----- From: Feng, Bob C Sent: 02 June 2020 12:16 To: devel@edk2.groups.io; Pierre Gondois Cc: Gao, Liming ; Sami Mujawar ; Tomas Pilar ; nd Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rule= s when multiple output files My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io On Behalf Of PierreGondo= is Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois ; Feng, Bob C ; Gao, Liming ; Sami.Mujawar@arm.com; Tomas.Pila= r@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules wh= en multiple output files From: Pierre Gondois BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2425 This patch modifies the Makefile generation not to stop adding Makfile rul= es when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Ru= le1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final targ= et. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source = file list, and in the later loop, that output file will be handled by anoth= er rule. Doesn't that algorithm handle your case above? [Pierre] The rule's output file was effectively added to the list of source files, = and a rule was searched for this output file. However, the loop stopped wh= en the first final target was found. By final target I mean "a file that is= n't the input of a rule". For the asl/aml/amli case, this meant that: -(first loop iteration: treating the ASL file) The rule for ASL files was= found and applied to the input ASL file. The AML and ".amli" files were ad= ded to the list of source files to look for a rule to apply to them. The lo= op continues. -(second loop iteration: treating the AML file) There is no rule for AML = files. This means that the AML file is a final target. The loop ends withou= t having applied the rule on the ".amli" file. This is why all the "break" instructions of the loop have been replaced wi= th "continue". This prevents the loop from exiting without having treated a= ll the files in the "SourceList". This patch also adds a dependency between the first ouput file of a rule a= nd the other output files. For instance, with the same example as above, File1.Y and File1.Z are gene= rated by the following rule: File1.Y: File1.X and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the M= akefile. Indeed, .Y and .Z files are generated by the execution of a common= set of commands, and without this rule, there is no explicit dependency re= lation between them. [Bob] If there are 3 output files, for example:=20 A B C Will the dependency relationship be B: A C: B [Pierre] Currently, the dependency relation that would be created is: B: A C: A This can be changed to the dependency you described. Signed-off-by: Pierre Gondois --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Com= pile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------= - 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source= /Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c= 46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict =3D {"target": self.PlaceMacro(T.Target.Pa= th, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLA= TE.Replace(TargetDict)) =20 + # Add a Makefile rule for targets generating multiple= files. + # The main output is a prerequisite for the other out= put files. + for i in T.Outputs[1:]: + AnnexeTargetDict =3D {"target":=20 + self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps":=20 + self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppD= ict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/= Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f44= 14e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory =3D os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget =3D None + TargetList =3D set() + FinalTargetName =3D set() RuleChain =3D set() SourceList =3D [File] Index =3D 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source =3D SourceList[Index] Index =3D Index + 1 - + FileType =3D TAB_UNKNOWN_FILE if Source !=3D File: CreateDirectory(Source.Dir) =20 @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject =3D self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject =3D self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject =3D self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType =3D RuleObject.SourceFileType self._FileTypes[FileType].add(Source) =20 # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType =3D=3D TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue =20 Target =3D RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue =20 + TargetList.add(Target) self._BuildTargets[FileType].add(Target) =20 if not Source.IsBinary and Source =3D=3D File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): =20 # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic=20 + dependency detected while generating rule for %s" % str(Source)) =20 RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget =3D Target - FileType =3D TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName =3D=3D Target.Target: + self._FinalBuildTargetList.add(Target) =20 @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)'