From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from nwk-aaemail-lapp03.apple.com (nwk-aaemail-lapp03.apple.com [17.151.62.68]) by mx.groups.io with SMTP id smtpd.web12.7762.1572322305702594866 for ; Mon, 28 Oct 2019 21:11:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=o09QvMgF; spf=pass (domain: apple.com, ip: 17.151.62.68, mailfrom: afish@apple.com) Received: from pps.filterd (nwk-aaemail-lapp03.apple.com [127.0.0.1]) by nwk-aaemail-lapp03.apple.com (8.16.0.27/8.16.0.27) with SMTP id x9T47bvi028480; Mon, 28 Oct 2019 21:11:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=sender : content-type : content-transfer-encoding : from : mime-version : subject : date : message-id : references : cc : in-reply-to : to; s=20180706; bh=lis3BLMnodxsMCGis3SVvpudxaGeu8PmY7BWCfntOY0=; b=o09QvMgF95/baaiXypj2OFqMXSggJrShLEY318JiHYSAHtrdU0s6rOufvQuyP/ckHQIJ 8l4eYHtv5CUb/UUvhpCgvomeIIUTvyRmkmyuUgOGiRmn5+EVw3IdZlJmO/2RJ8d9JiCA +QOt5XtaYdvrzXVIXIedZVphoUJMu/q7X/DzOcZYYf/QL5GweOcvVxSFp9XHBl0Bk9Na hhAL5fwPowJUVOAqit45SxBUUYIHTBWIW6S4Pq5VG113bQ5cE4e5IMPHkSiPmYO7ddC7 1eOB/j+H7sEiWuC7804O3l7lefyoiqjolLfVULqxMeMYuRrmZgqFkrsVaHczLkDKCGLz bw== Received: from ma1-mtap-s01.corp.apple.com (ma1-mtap-s01.corp.apple.com [17.40.76.5]) by nwk-aaemail-lapp03.apple.com with ESMTP id 2vw6cm6vbp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 28 Oct 2019 21:11:43 -0700 Received: from nwk-mmpp-sz11.apple.com (nwk-mmpp-sz11.apple.com [17.128.115.155]) by ma1-mtap-s01.corp.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPS id <0Q0400JJYBNI1310@ma1-mtap-s01.corp.apple.com>; Mon, 28 Oct 2019 21:11:43 -0700 (PDT) Received: from process_milters-daemon.nwk-mmpp-sz11.apple.com by nwk-mmpp-sz11.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) id <0Q0400H00BMJ9C00@nwk-mmpp-sz11.apple.com>; Mon, 28 Oct 2019 21:11:42 -0700 (PDT) X-Va-A: X-Va-T-CD: be7616426468224f7c429d708b2b3a08 X-Va-E-CD: 7c049b726eb4bd94d2bc056bc12a4189 X-Va-R-CD: bdc755f033a2e724f8c3b8bf8392ed19 X-Va-CD: 0 X-Va-ID: cbfe5e35-994c-4b55-afe2-965c424e9b73 X-V-A: X-V-T-CD: be7616426468224f7c429d708b2b3a08 X-V-E-CD: 7c049b726eb4bd94d2bc056bc12a4189 X-V-R-CD: bdc755f033a2e724f8c3b8bf8392ed19 X-V-CD: 0 X-V-ID: 40807e62-3961-4fcb-809e-ad8b1eb31117 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-10-29_02:,, signatures=0 Received: from [17.235.43.132] (unknown [17.235.43.132]) by nwk-mmpp-sz11.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPSA id <0Q040060ZBNF2K20@nwk-mmpp-sz11.apple.com>; Mon, 28 Oct 2019 21:11:39 -0700 (PDT) Sender: afish@apple.com From: "Andrew Fish" MIME-version: 1.0 (1.0) Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to generate source code dependency files. Date: Mon, 28 Oct 2019 21:11:37 -0700 Message-id: References: <08650203BA1BD64D8AD9B6D5D74A85D161558EE4@SHSMSX104.ccr.corp.intel.com> Cc: "devel@edk2.groups.io" , "Yao, Jiewen" , Ryszard Knop In-reply-to: <08650203BA1BD64D8AD9B6D5D74A85D161558EE4@SHSMSX104.ccr.corp.intel.com> To: "Feng, Bob C" X-Mailer: iPhone Mail (17A878) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-10-29_02:,, signatures=0 Content-type: multipart/alternative; boundary=Apple-Mail-0E268126-5B6C-47BD-B78B-30C8F5B7D69B Content-transfer-encoding: 7bit --Apple-Mail-0E268126-5B6C-47BD-B78B-30C8F5B7D69B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Bob, I was trying to point out the .d file extension also implies a C dependenc= y file in Gcc world. So to me I would have picked .d. But other choices ar= e OK too. > On Oct 28, 2019, at 6:12 PM, Feng, Bob C wrote: >=20 > =EF=BB=BF > Yes. For Gcc and Clang, we can use =E2=80=93MMD =E2=80=93MF to generate custom name dependency file for each source file. For examp= le, gcc main.c =E2=80=93MMD =E2=80=93MF main.deps.=20 > > For MSVC and Intel compiler, there is a build option /showIncludes which= makes compiler print the dependency files on stdout but not a file. The bu= ild tool will capture the message from stdout and generate .deps files on f= ile system. The format will be the same as GCC generate. > > Thanks, > Bob > > From: afish@apple.com [mailto:afish@apple.com]=20 > Sent: Tuesday, October 29, 2019 1:19 AM > To: devel@edk2.groups.io; Yao, Jiewen > Cc: Feng, Bob C ; Ryszard Knop > Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to gener= ate source code dependency files. > > The .d is the default file name extension used by GCC for dependency fil= es. Given the dependency files are in the build output and the makefiles re= ference them explicitly I'm not sure there is going to be lots of confusion= .=20 > > But I think it is likely the default dependency file name is mostly just= a convenience in constructing the makefiles, since if you use a custom nam= e each invocation has to do string operations to create a file with a custo= m extension. Given our makefiles are generated by a tool this is probably n= ot really an issue for us.=20 > > Does Visual Studio have the concept of a dependency file? If yes what su= ffix does that use, as I guess we could use that? I guess the dependency ha= ndling could just be some XML data in some larger Visual Studio meta data..= .. > > GCC/clang flags around dependencies.=20 > > -MD > -MD is equivalent to -M -MF file, except that -E is not implied. The dri= ver determines file based on whether an -o option is given. If it is, the d= river uses its argument but with a suffix of .d, otherwise it takes the nam= e of the input file, removes any directory components and suffix, and appli= es a .d suffix. > If -MD is used in conjunction with -E, any -o switch is understood to sp= ecify the dependency output file (see -MF), but if used without -E, each -o= is understood to specify a target object file. > Since -E is not implied, -MD can be used to generate a dependency output= file as a side effect of the compilation process > > > -MF file > When used with -M or -MM, specifies a file to write the dependencies to.= If no -MF switch is given the preprocessor sends the rules to the same pla= ce it would send preprocessed output. > When used with the driver options -MD or -MMD, -MF overrides the default= dependency output file. > If file is -, then the dependencies are written to stdout. > > Thanks, > > Andrew Fish >=20 >=20 > On Oct 28, 2019, at 8:03 AM, Yao, Jiewen wrote: > > I think we need avoid confusing for future. > I don=E2=80=99t believe .d is good choice, since it is a known conflict. >=20 > Thank you > Yao Jiewen >=20 >=20 >=20 > -----Original Message----- > From: devel@edk2.groups.io On Behalf Of Bob Feng > Sent: Monday, October 28, 2019 10:57 PM > To: Ryszard Knop ; devel@edk2.groups.io > Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to gener= ate > source code dependency files. >=20 > Thanks for your comment. I think .d file should be fine since edk2 does = not > support D language. >=20 > Thanks, > Bob >=20 > -----Original Message----- > From: Ryszard Knop > Sent: Monday, October 28, 2019 8:24 PM > To: devel@edk2.groups.io; Feng, Bob C > Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to gener= ate > source code dependency files. >=20 > Just a quick note: .d files are used by the D language. You might want t= o use an > extension like .deps instead. >=20 > On 2019-10-28 11:47, Bob Feng wrote: >=20 > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2311 >=20 > To support incremental build, build tool generates the dependent > header file for each of source file. This procedure is done in AutoGen p= hase. > The build tool goes through all the source file and header file and > use regular expression to find out all the dependent files for a > source file. This procedure is much time-consuming. And this method > can't handle the MACRO in #include, for example #include PATH(xxx.h). >=20 > This patch is going to use compiler to generate dependent files. This > method will be faster and more accurate. >=20 > The basic idea is: > 1. In AutoGen phase, build tool add "!Include deps.txt" into Makefile > instead of defining COMMON_DEPS list. > 2. During the Make phase, the compilers, Trim and C preprocessor > generate dependent files, .d file, for each source file. > 3. After Make, The build tool combines the .d files and generate a > file deps.txt which list all the included files for a module. > 4. Each source file will depends on the Module's includes files. The > difference with orignial behavior is that if the user change the > source file, build tool will only build that source file in > incremental build; while if the user change a module's header file, > build tool will build the whole module in incremental build. >=20 > In this way, the time of AutoGen phase will be reduced much. And since > we will use c preprocessor to handle #include, the MACRO will be > handled well and the final dependent files will be more accurate. >=20 > Feng, Bob C (1): > BaseTools: Using compiler to generate source code dependency files. >=20 > BaseTools/Conf/build_rule.template | 89 ++++++----- > BaseTools/Conf/tools_def.template | 138 +++++++++--------- > BaseTools/Source/Python/AutoGen/GenMake.py | 73 +++------ > .../Source/Python/AutoGen/IncludesAutoGen.py | 99 +++++++++++++ > BaseTools/Source/Python/Trim/Trim.py | 113 +++++++++++--- > BaseTools/Source/Python/build/build.py | 58 ++++++-- > 6 files changed, 378 insertions(+), 192 deletions(-) > create mode 100644 > BaseTools/Source/Python/AutoGen/IncludesAutoGen.py >=20 > >=20 >=20 >=20 >=20 > --Apple-Mail-0E268126-5B6C-47BD-B78B-30C8F5B7D69B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Bob,

I was trying t= o point out the .d file extension also implies a C dependency file in Gcc w= orld.  So to me I would have picked .d. But other choices are OK too.<= br>

On Oct 28, 2019, at 6:12 PM, Feng, Bob C <bob.c.feng@intel.com> wro= te:

= =EF=BB=BF

Yes. For Gcc and Clang, we can use = =E2=80=93MMD =E2=80=93MF <filename.deps> to generate custom name dep= endency file for each source file. For example, gcc main.c  =E2=80=93M= MD =E2=80=93MF main.deps. 

 

For MSVC and Intel compiler, there i= s a build option /showIncludes which makes compiler print the dependency fi= les on stdout but not a file. The build tool will capture the message from stdout and generate .deps files on file system. = The format will be the same as GCC generate.

 

Thanks,

Bob

 

From: afish@apple.com [mailto:afish= @apple.com]
Sent: Tuesday, October 29, 2019 1:19 AM
To: devel@edk2.groups.io; Yao, Jiewen <jiewen.yao@intel.com><= br> Cc: Feng, Bob C <bob.c.feng@intel.com>; Ryszard Knop <rysz= ard.knop@linux.intel.com>
Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to = generate source code dependency files.

 

The .d is the default file name extension used by G= CC for dependency files. Given the dependency files are in the build output= and the makefiles reference them explicitly I'm not sure there is going to= be lots of confusion. 

 

But I think it is likely the default dependency fil= e name is mostly just a convenience in constructing the makefiles, since if= you use a custom name each invocation has to do string operations to creat= e a file with a custom extension. Given our makefiles are generated by a tool this is probably not really an issu= e for us. 

 

Does Visual Studio have the concept of a dependency= file? If yes what suffix does that use, as I guess we could use that? I gu= ess the dependency handling could just be some XML data in some larger Visu= al Studio meta data....

 

GCC/clang flags around dependencies. 

 

-MD

-MD is equivalent to -M -= MF file, except that -E is not implied. The driver determines file based on = whether an -o op= tion is given. If it is, the driver uses its argument but with a suffix of&= nbsp;.d, otherwise it takes the name of the input file, removes any directory comp= onents and suffix, and applies a .d suffix.

If -MD is= used in conjunction with -E<= /span>, any -o s= witch is understood to specify the dependency output file (see <= span style=3D"color:#003399;text-decoration:none">-MF), but if used without -E, each -o is understood to specify a target object file.

Since -E = is not implied, -MD can be used to generate a dependency output file as a side effect of the comp= ilation process

 =

 

= -MF file

When used with -M or -MM, specifies a file to write the dependencies to. If no -MF switch is given the prepro= cessor sends the rules to the same place it would send preprocessed output.

When used with the driver options -MD or -MMD, -MF overrides the default dependency output file.

If file is -= , then the dependencies are written to stdout.

 

Thanks,

 

Andrew Fish



On Oct 28, 2019, at 8:03 AM, Yao, Jiewen <jiewen.yao@intel.com> wrote:<= /o:p>

 

I think we need avoid confusing for future.
I don=E2=80=99t believe .d is good choice, since it is a known conflict.
Thank you
Yao Jiewen



-----Original Message-----
From: devel@edk2.groups.io <devel@edk= 2.groups.io> On Behalf Of Bob Feng
Sent: Monday, October 28, 2019 10:57 PM
To: Ryszard Knop <rysza= rd.knop@linux.intel.com>; = ;devel@edk2.groups.io Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to generat= e
source code dependency files.

Thanks for your comment. I think .d file should be fine since edk2 does no= t
support D language.

Thanks,
Bob

-----Original Message-----
From: Ryszard Knop <rys= zard.knop@linux.intel.com>
Sent: Monday, October 28, 2019 8:24 PM
To: devel@edk2.groups.io; Feng= , Bob C <bob.c.feng@intel.com>
Subject: Re: [edk2-devel] [Patch 0/1] BaseTools: Using compiler to generat= e
source code dependency files.

Just a quick note: .d files are used by the D language. You might want to = use an
extension like .deps instead.

On 2019-10-28 11:47, Bob Feng wrote:

BZ: https://= bugzilla.tianocore.org/show_bug.cgi?id=3D2311

To support incremental build, build tool generates the dependent
header file for each of source file. This procedure is done in AutoGen pha= se.
The build tool goes through all the source file and header file and
use regular expression to find out all the dependent files for a
source file. This procedure is much time-consuming. And this method
can't handle the MACRO in #include, for example #include PATH(xxx.h).

This patch is going to use compiler to generate dependent files. This
method will be faster and more accurate.

The basic idea is:
1. In AutoGen phase, build tool add "!Include deps.txt" into Makefile
instead of defining COMMON_DEPS list.
2. During the Make phase, the compilers, Trim and C preprocessor
generate dependent files, .d file, for each source file.
3. After Make, The build tool combines the .d files and generate a
file deps.txt which list all the included files for a module.
4. Each source file will depends on the Module's includes files. The
difference with orignial behavior is that if the user change the
source file, build tool will only build that source file in
incremental build; while if the user change a module's header file,
build tool will build the whole module in incremental build.

In this way, the time of AutoGen phase will be reduced much. And since
we will use c preprocessor to handle #include, the MACRO will be
handled well and the final dependent files will be more accurate.

Feng, Bob C (1):
  BaseTools: Using compiler to generate source code dependency f= iles.

 BaseTools/Conf/build_rule.template      &nb= sp;     |  89 ++++++-----
 BaseTools/Conf/tools_def.template      &nbs= p;      | 138 +++++++++---------
 BaseTools/Source/Python/AutoGen/GenMake.py    |  = ;73 +++------
 .../Source/Python/AutoGen/IncludesAutoGen.py  |  99 ++++++= +++++++
 BaseTools/Source/Python/Trim/Trim.py      &= nbsp;   | 113 +++++++++++---
 BaseTools/Source/Python/build/build.py      = ;  |  58 ++++++--
 6 files changed, 378 insertions(+), 192 deletions(-)
 create mode 100644
BaseTools/Source/Python/AutoGen/IncludesAutoGen.py

 <= /span>



 

--Apple-Mail-0E268126-5B6C-47BD-B78B-30C8F5B7D69B--