From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.20, mailfrom: liming.gao@intel.com) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by groups.io with SMTP; Mon, 20 May 2019 19:18:26 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 May 2019 19:18:25 -0700 X-ExtLoop1: 1 Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga008.fm.intel.com with ESMTP; 20 May 2019 19:18:25 -0700 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 20 May 2019 19:18:24 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.33]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.216]) with mapi id 14.03.0415.000; Tue, 21 May 2019 10:18:22 +0800 From: "Liming Gao" To: "afish@apple.com" , "devel@edk2.groups.io" CC: "Justen, Jordan L" , Ard Biesheuvel Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8 Thread-Topic: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8 Thread-Index: AQHU/E3ClCQnniT0z0eoB13gdfWnb6ZQvUVAgCHDBYCAAao9MIAAFDcAgAC8NbA= Date: Tue, 21 May 2019 02:18:22 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26D@SHSMSX104.ccr.corp.intel.com> References: <20190426144242.19024-1-liming.gao@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E431937@SHSMSX104.ccr.corp.intel.com> <155829688766.24668.13457231493488240669@jljusten-skl> <4A89E2EF3DFEDB4C8BFDE51014F606A14E44DCA3@SHSMSX104.ccr.corp.intel.com> <7C375416-55C4-40B9-957B-BE6376A55676@apple.com> In-Reply-To: <7C375416-55C4-40B9-957B-BE6376A55676@apple.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMTNiMzVlN2YtMTBmNi00ODYyLTljNjctZTdhYTVmY2I5YjYxIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiZUVTcVJSOGtjZlNqTUpYVlc0MWZQR21mZWlcLzRGQzViTFwvaGlrN1owbWNpcFlsZGFJMlo1b01wdHl5VUpUYXhoIn0= dlp-product: dlpe-windows dlp-version: 11.0.600.7 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: liming.gao@intel.com Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26DSHSMSX104ccrcor_" --_000_4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26DSHSMSX104ccrcor_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Andrew: Thanks for your sharing. I would like the official LLVM tool chain inste= ad of the customized version. I will investigate it more and go back. Now, I push this patch set into = staging https://github.com/tianocore/edk2-staging/tree/llvm8 for the evalua= tion. Thanks Liming From: afish@apple.com [mailto:afish@apple.com] Sent: Tuesday, May 21, 2019 6:53 AM To: devel@edk2.groups.io; Gao, Liming Cc: Justen, Jordan L ; Ard Biesheuvel Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new= LLVM/CLANG8 On May 20, 2019, at 6:47 AM, Liming Gao > wrote: Jordan: -----Original Message----- From: Justen, Jordan L Sent: Monday, May 20, 2019 4:15 AM To: Ard Biesheuvel >; Gao, Liming >; ed= k2-devel-groups-io > Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new= LLVM/CLANG8 On 2019-04-27 17:55:02, Liming Gao wrote: -----Original Message----- From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org] Sent: Saturday, April 27, 2019 12:33 AM This series confuses me. The existing CLANGxx toolchains already use GenFw and ELF to PE/COFF conversion, so the name CLANG8ELF is misleading. LLVM/CLANG8.0 compiler supports to generate PE image or ELF image. This tool chain is to generate ELF image and be converted to PE image. I am investigating another tool chain with CLANG8.0 to directly generate PE image. To differentiate them, I use the tool chain name CLANG8ELF and CLANG8PE for them. Assuming CLANG8ELF and CLANG8PE were functional, would both be needed? It kind of sounds like this a half-finished investigation. One point is that they will generate the different debug format symbols (W= inDBG or GDB). I have not done the full investigation to generate the different debug for= mat symbols in single tool chain. Liming, I don't know the current answer but we moved over to using clang a while b= ack on Mac so here are some general thoughts incase they help your investig= ation. We had to take a different approach for i386 vs x86_64.: 1) i386 For i386 we used -arch i386 and compiler flags to produce the correct ABI.= The thing about using -arch is the target triple defaults to your current = machine, thus -arch i386 on a Mac is going to make a Mach-O object. if you = use it one Linux it is going to make an ELF, on Windows a PE/COFF. To make = this work we add mtoc to CCTOOLS this is a tool that converts Mach-O to PE/= COFF. The PE/COFF contains a UUID for the Mach-O and the Mach-O dSYM (kind = of like the PDB file) and that is what is used for debugging. You can even = load the Mach-O directly into the debugger and do offline investigation of = the image at its linked address. 2) x86_64 Since the Mac uses the System V ABI we needed a cross compiler. Clang impl= ements cross compilers via the Triple (this is the -target argument you pas= s to clang). We actually end up open sourcing a triple that did what we nee= ded. For building on a Mac the triple is: -target x86_64-pc-win32-macho --- < x86_64 ><>-< pc >-< win32 >-< macho > Examples: arch =3D x86_64, i386, arm, thumb, mips, etc. sub =3D for ex. on ARM: v5, v6m, v7a, v7m, etc. vendor =3D pc, apple, nvidia, ibm, etc. sys =3D none, linux, win32, darwin, cuda, etc. abi =3D eabi, gnu, android, macho, elf, etc. OK so they expanded the Triple to have 4 fields, so that is kind of naming= fail, but they still seem to call it a triple. When we did the port "-DEFI= API=3D__attribute__((ms_abi))" was not an option and that is why we ended u= p having to male our own triple. As far as I can tell __attribute__((ms_abi= )) works correctly in the Xcode compiler. To make the debugging work we ended up defining a new CodeView signature [= 1] to the PE/COFF Debug Directory Entry. At this point the native lldb (llv= m debugger) does not know how to load Mach-O DWARF from PE/COFF. We could t= each it, but we have not really found a need as we ended up writing debugge= r scripts to load symbols and lldb can load the Mach-O image (our build sys= tem leaves them around) for offline debugging. In terms of PDB vs DWARF generally the compiler emits the DWARF (or PDB) d= ata at compile time and the linker just aggregates it together. I see a blo= g from last April [2] talking about LLVM PDB support and it ends with "Do I= hear cross-compilation?". So you might have to compile from the start for = PDB or DWARF. This seems to imply you are going to need a triple to invoke = a cross compiler? Also you are going to need that triple present in the cla= ng binary, so you might be able to make a custom version of clang that supp= orts the triples you need, but that may not be in the official binary for c= lang. Converting ELF to PE/COFF is much more straight forward than converting th= e debugging information so that makes sense that it could just be the backe= nd of the linker. I look forward to getting the results of your investigation. [1] https://github.com/tianocore/edk2/blob/master/MdePkg/Include/IndustryS= tandard/PeImage.h#L647 [2] http://blog.llvm.org/2017/08/llvm-on-windows-now-supports-pdb-debug.ht= ml Thanks, Andrew Fish I'm guessing that if CLANG8PE produces equal or better code, then it would be preferred. Therefore, shouldn't we just finish the investigation, and add a single CLANG8 toolchain at the end? Or, maybe to reflect that it mostly uses the LLVM tool stack we could name it LLVM8. I also prefer single CLANG8 tool chain. I will collect more information an= d see whether it is possible. Now, I would like to add this tool chain for the developer evaluation. Bec= ause I can't address all comments now, I will remove this feature from Q2 stable tag. I will add it= into edk2-staging first. -Jordan --_000_4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26DSHSMSX104ccrcor_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Andrew:

  Thanks for you= r sharing. I would like the official LLVM tool chain instead of the customi= zed version.

 

  I will investi= gate it more and go back. Now, I push this patch set into staging https://github.com/tianocore/edk2-staging/tree/llvm8 for the evaluation.=

 

Thanks

Liming

From: afish@apple.com [mailto:afish@apple= .com]
Sent: Tuesday, May 21, 2019 6:53 AM
To: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com><= br> Cc: Justen, Jordan L <jordan.l.justen@intel.com>; Ard Biesheu= vel <ard.biesheuvel@linaro.org>
Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain = for new LLVM/CLANG8

 

 



On May 20, 2019, at 6:47 AM, L= iming Gao <liming.gao@intel.com<= /a>> wrote:

 

Jordan:


-----Original Message-----
From: Justen, Jordan L
Sent: Monday, May 20, 2019 4:15 AM
To: Ard Biesheuvel <ard.bi= esheuvel@linaro.org>; Gao, Liming <liming.gao@intel.com>; edk2-devel-groups-io <devel@edk2.groups.io>
Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new= LLVM/CLANG8

On 2019-04-27 17:55:02, Liming Gao wrote:

---= --Original Message-----
From: Ard Biesheuvel [mailto:= ard.biesheuvel@linaro.org]
Sent: Saturday, April 27, 2019 12:33 AM


This series confuses me. The existing CLANGxx toolchains already use
GenFw and ELF to PE/COFF conversion, so the name CLANG8ELF is
misleading.

LLVM/CLANG8.0 compiler supports to= generate PE image or ELF image.
This tool chain is to generate ELF image and be converted to PE
image. I am investigating another tool chain with CLANG8.0 to
directly generate PE image. To differentiate them, I use the tool
chain name CLANG8ELF and CLANG8PE for them.

Assuming CLANG8ELF and CLANG8PE were functional, would both be needed?
It kind of sounds like this a half-finished investigation.

One point is that they will genera= te the different debug format symbols (WinDBG or GDB).
I have not done the full investigation to generate the different debug for= mat symbols in single tool chain.

 

 

Liming,

 

I don't know the current answe= r but we moved over to using clang a while back on Mac so here are some gen= eral thoughts incase they help your investigation. <= /p>

 

We had to take a different app= roach for i386 vs x86_64.:

1) i386

For i386 we used -arch i386 an= d compiler flags to produce the correct ABI. The thing about using -arch is= the target triple defaults to your current machine, thus -arch i386 on a M= ac is going to make a Mach-O object. if you use it one Linux it is going to make an ELF, on Windows a PE/COFF.= To make this work we add mtoc to CCTOOLS this is a tool that converts Mach= -O to PE/COFF. The PE/COFF contains a UUID for the Mach-O and the Mach-O dS= YM (kind of like the PDB file) and that is what is used for debugging. You can even load the Mach-O directly= into the debugger and do offline investigation of the image at its linked = address. 

 

2) x86_64

Since the Mac uses the System = V ABI we needed a cross compiler. Clang implements cross compilers via the = Triple (this is the -target argument you pass to clang). We actually end up= open sourcing a triple that did what we needed. 

 

For building on a Mac the trip= le is: -target x86_64-pc-win32-macho

 

 <arch><sub>-= <vendor>-<sys>-<abi>

 < x86_64 &g= t;<>-< pc     >-< win32 >-<&n= bsp;macho >

 

Examples:

arch =3D x86_64, i386, arm, th= umb, mips, etc.

sub =3D for ex. on ARM: v5, v6= m, v7a, v7m, etc.

vendor =3D pc, apple, nvidia, = ibm, etc.

sys =3D none, linux, win32, da= rwin, cuda, etc.

abi =3D eabi, gnu, android, ma= cho, elf, etc.

 

OK so they expanded the Triple= to have 4 fields, so that is kind of naming fail, but they still seem to c= all it a triple. When we did the port "-DEFIAPI=3D__attribute__((ms_ab= i))" was not an option and that is why we ended up having to male our own triple. As far as I can tell __attribute__((ms_= abi)) works correctly in the Xcode compiler. 

 

To make the debugging work we = ended up defining a new CodeView signature [1] to the PE/COFF Debug Directo= ry Entry. At this point the native lldb (llvm debugger) does not know how t= o load Mach-O DWARF from PE/COFF. We could teach it, but we have not really found a need as we ended up writin= g debugger scripts to load symbols and lldb can load the Mach-O image (our = build system leaves them around) for offline debugging.

 

In terms of PDB vs DWARF gener= ally the compiler emits the DWARF (or PDB) data at compile time and the lin= ker just aggregates it together. I see a blog from last April [2] talking a= bout LLVM PDB support and it ends with "Do I hear cross-compilation?". So you might have to compile fr= om the start for PDB or DWARF. This seems to imply you are going to need a = triple to invoke a cross compiler? Also you are going to need that triple p= resent in the clang binary, so you might be able to make a custom version of clang that supports the triples you need= , but that may not be in the official binary for clang. 

 

Converting ELF to PE/COFF is m= uch more straight forward than converting the debugging information so that= makes sense that it could just be the backend of the linker. 

 

I look forward to getting the = results of your investigation.

 

 

Thanks,

 

Andrew Fish<= /p>

 

I'm guessing that if CLANG8PE prod= uces equal or better code, then it
would be preferred.

Therefore, shouldn't we just finish the investigation, and add a
single CLANG8 toolchain at the end? Or, maybe to reflect that it
mostly uses the LLVM tool stack we could name it LLVM8.<= /p>

I also prefer single CLANG8 tool c= hain. I will collect more information and see whether it is possible. 
Now, I would like to add this tool chain for the developer evaluation. Bec= ause I can't address all =
comments now, I will remove this feature from Q2 stable tag. I will add it= into edk2-staging first. 


-Jordan


 

--_000_4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26DSHSMSX104ccrcor_--