From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=oETiv9Yy; spf=pass (domain: apple.com, ip: 17.151.62.66, mailfrom: afish@apple.com) Received: from nwk-aaemail-lapp01.apple.com (nwk-aaemail-lapp01.apple.com [17.151.62.66]) by groups.io with SMTP; Mon, 20 May 2019 20:37:11 -0700 Received: from pps.filterd (nwk-aaemail-lapp01.apple.com [127.0.0.1]) by nwk-aaemail-lapp01.apple.com (8.16.0.27/8.16.0.27) with SMTP id x4L3Vv3n060505; Mon, 20 May 2019 20:37:11 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=mime-version : content-type : sender : from : message-id : subject : date : in-reply-to : cc : to : references; s=20180706; bh=CzgrI8FyhfRsCymtXkJJ6IYiixNkPGu+tHJQefcNpUw=; b=oETiv9YyKn8UnGYfDJd018I9CjXVkVwmhHocztDuFt2wIyyDZDKKn82k+3JeQiZz930z gutF48ckfDqkdkPWh3yTCtBnzSdP4iphr8BKfEOpA3CDCSt5cdAy7AlneWlJdY5Db5n9 AGz1ApMcQjxp/tVI2fdk65195NyH/xN29wBkyHlpjk7wL76ljZHMcvT1JZRCOaxiWVZI rpnJnzqRWiMi49sHTPA4nTIIt6K0PsnE9eSeZ70Jb3pdnN6akLh4vrdV/Ckmn+b/ozbC KWj9RDLEsS1YlpXu8z0JChUmlgWUGRYPeIpdzaAzjMHwTuQTt8Hou3fFCUUbTim/jyxT Mg== Received: from ma1-mtap-s02.corp.apple.com (ma1-mtap-s02.corp.apple.com [17.40.76.6]) by nwk-aaemail-lapp01.apple.com with ESMTP id 2sjh00nxdj-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 20 May 2019 20:37:11 -0700 MIME-version: 1.0 Received: from nwk-mmpp-sz11.apple.com (nwk-mmpp-sz11.apple.com [17.128.115.155]) by ma1-mtap-s02.corp.apple.com (Oracle Communications Messaging Server 8.0.2.3.20181024 64bit (built Oct 24 2018)) with ESMTPS id <0PRU007V44PXVH20@ma1-mtap-s02.corp.apple.com>; Mon, 20 May 2019 20:37:10 -0700 (PDT) Received: from process_milters-daemon.nwk-mmpp-sz11.apple.com by nwk-mmpp-sz11.apple.com (Oracle Communications Messaging Server 8.0.2.3.20181024 64bit (built Oct 24 2018)) id <0PRU00D0049Z8100@nwk-mmpp-sz11.apple.com>; Mon, 20 May 2019 20:37:09 -0700 (PDT) X-Va-A: X-Va-T-CD: 98e262f3d41ed2218ef8f9253e109c8a X-Va-E-CD: e498e3d4c78d649fbccdc891e3510fc8 X-Va-R-CD: 54df24a98728e145d87e6f6196c940db X-Va-CD: 0 X-Va-ID: 18d39ef5-e394-448f-8b35-b9c608a7a3a6 X-V-A: X-V-T-CD: 98e262f3d41ed2218ef8f9253e109c8a X-V-E-CD: e498e3d4c78d649fbccdc891e3510fc8 X-V-R-CD: 54df24a98728e145d87e6f6196c940db X-V-CD: 0 X-V-ID: f1181dc4-1010-4d6c-959c-3502d3d3c059 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-20_09:,, signatures=0 Received: from [17.235.14.21] (unknown [17.235.14.21]) by nwk-mmpp-sz11.apple.com (Oracle Communications Messaging Server 8.0.2.3.20181024 64bit (built Oct 24 2018)) with ESMTPSA id <0PRU000154PTT930@nwk-mmpp-sz11.apple.com>; Mon, 20 May 2019 20:37:06 -0700 (PDT) Sender: afish@apple.com From: "Andrew Fish" Message-id: Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8 Date: Mon, 20 May 2019 20:36:46 -0700 In-reply-to: <4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26D@SHSMSX104.ccr.corp.intel.com> Cc: Jordan Justen , Ard Biesheuvel To: devel@edk2.groups.io, Liming Gao References: <20190426144242.19024-1-liming.gao@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E431937@SHSMSX104.ccr.corp.intel.com> <155829688766.24668.13457231493488240669@jljusten-skl> <4A89E2EF3DFEDB4C8BFDE51014F606A14E44DCA3@SHSMSX104.ccr.corp.intel.com> <7C375416-55C4-40B9-957B-BE6376A55676@apple.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E44E26D@SHSMSX104.ccr.corp.intel.com> X-Mailer: Apple Mail (2.3445.6.18) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-20_09:,, signatures=0 Content-type: multipart/alternative; boundary="Boundary_(ID_a6Jr23nkqzyIhe3IaTLjpQ)" --Boundary_(ID_a6Jr23nkqzyIhe3IaTLjpQ) Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT > On May 20, 2019, at 7:18 PM, Liming Gao wrote: > > Andrew: > Thanks for your sharing. I would like the official LLVM tool chain instead of the customized version. > Liming, Hopefully all the bits are in place so you can cross compile for Linux on Windows, and Windows on Linux so you will not need a customer binary. If not that seems like something reasonable to ask for, bur remember some one is going to ask for tests. On thing to try is to get the default triple on Linux and try it Window, and get the default triple on Windows and try it on Linux. if you `clang -v` that should print out the default triple. From this writeup [1] some of the 4 fields in the triple can get skipped. You can then try to build the Windows triple on Linux etc. One of the good things about the edk2 is there is not a dependency on the include files or runtime so you just need to make the compiler/linker do the right thing. [1] https://clang.llvm.org/docs/CrossCompilation.html Thanks, Andrew Fish > I will investigate it more and go back. Now, I push this patch set into staging https://github.com/tianocore/edk2-staging/tree/llvm8 for the evaluation. > > Thanks > Liming <> > <>From: afish@apple.com [mailto:afish@apple.com ] > Sent: Tuesday, May 21, 2019 6:53 AM > To: devel@edk2.groups.io ; Gao, Liming > > Cc: Justen, Jordan L >; Ard Biesheuvel > > Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8 > > > > > On May 20, 2019, at 6:47 AM, Liming Gao > wrote: > > Jordan: > > > -----Original Message----- > From: Justen, Jordan L > Sent: Monday, May 20, 2019 4:15 AM > To: Ard Biesheuvel >; Gao, Liming >; edk2-devel-groups-io > > Subject: Re: [edk2-devel] [Patch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8 > > On 2019-04-27 17:55:02, Liming Gao wrote: > > -----Original Message----- > From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org ] > Sent: Saturday, April 27, 2019 12:33 AM > > > This series confuses me. The existing CLANGxx toolchains already use > GenFw and ELF to PE/COFF conversion, so the name CLANG8ELF is > misleading. > > LLVM/CLANG8.0 compiler supports to generate PE image or ELF image. > This tool chain is to generate ELF image and be converted to PE > image. I am investigating another tool chain with CLANG8.0 to > directly generate PE image. To differentiate them, I use the tool > chain name CLANG8ELF and CLANG8PE for them. > > Assuming CLANG8ELF and CLANG8PE were functional, would both be needed? > It kind of sounds like this a half-finished investigation. > > One point is that they will generate the different debug format symbols (WinDBG or GDB). > I have not done the full investigation to generate the different debug format symbols in single tool chain. > > > > Liming, > > I don't know the current answer but we moved over to using clang a while back on Mac so here are some general thoughts incase they help your investigation. > > We had to take a different approach for i386 vs x86_64.: > 1) i386 > For i386 we used -arch i386 and compiler flags to produce the correct ABI. The thing about using -arch is the target triple defaults to your current machine, thus -arch i386 on a Mac is going to make a Mach-O object. if you use it one Linux it is going to make an ELF, on Windows a PE/COFF. To make this work we add mtoc to CCTOOLS this is a tool that converts Mach-O to PE/COFF. The PE/COFF contains a UUID for the Mach-O and the Mach-O dSYM (kind of like the PDB file) and that is what is used for debugging. You can even load the Mach-O directly into the debugger and do offline investigation of the image at its linked address. > > 2) x86_64 > Since the Mac uses the System V ABI we needed a cross compiler. Clang implements cross compilers via the Triple (this is the -target argument you pass to clang). We actually end up open sourcing a triple that did what we needed. > > For building on a Mac the triple is: -target x86_64-pc-win32-macho > > --- > < x86_64 ><>-< pc >-< win32 >-< macho > > > Examples: > arch = x86_64, i386, arm, thumb, mips, etc. > sub = for ex. on ARM: v5, v6m, v7a, v7m, etc. > vendor = pc, apple, nvidia, ibm, etc. > sys = none, linux, win32, darwin, cuda, etc. > abi = eabi, gnu, android, macho, elf, etc. > > OK so they expanded the Triple to have 4 fields, so that is kind of naming fail, but they still seem to call it a triple. When we did the port "-DEFIAPI=__attribute__((ms_abi))" was not an option and that is why we ended up having to male our own triple. As far as I can tell __attribute__((ms_abi)) works correctly in the Xcode compiler. > > To make the debugging work we ended up defining a new CodeView signature [1] to the PE/COFF Debug Directory Entry. At this point the native lldb (llvm debugger) does not know how to load Mach-O DWARF from PE/COFF. We could teach it, but we have not really found a need as we ended up writing debugger scripts to load symbols and lldb can load the Mach-O image (our build system leaves them around) for offline debugging. > > In terms of PDB vs DWARF generally the compiler emits the DWARF (or PDB) data at compile time and the linker just aggregates it together. I see a blog from last April [2] talking about LLVM PDB support and it ends with "Do I hear cross-compilation?". So you might have to compile from the start for PDB or DWARF. This seems to imply you are going to need a triple to invoke a cross compiler? Also you are going to need that triple present in the clang binary, so you might be able to make a custom version of clang that supports the triples you need, but that may not be in the official binary for clang. > > Converting ELF to PE/COFF is much more straight forward than converting the debugging information so that makes sense that it could just be the backend of the linker. > > I look forward to getting the results of your investigation. > > [1] https://github.com/tianocore/edk2/blob/master/MdePkg/Include/IndustryStandard/PeImage.h#L647 > [2] http://blog.llvm.org/2017/08/llvm-on-windows-now-supports-pdb-debug.html > > Thanks, > > Andrew Fish > > I'm guessing that if CLANG8PE produces equal or better code, then it > would be preferred. > > Therefore, shouldn't we just finish the investigation, and add a > single CLANG8 toolchain at the end? Or, maybe to reflect that it > mostly uses the LLVM tool stack we could name it LLVM8. > I also prefer single CLANG8 tool chain. I will collect more information and see whether it is possible. > Now, I would like to add this tool chain for the developer evaluation. Because I can't address all > comments now, I will remove this feature from Q2 stable tag. I will add it into edk2-staging first. > > > -Jordan > > > --Boundary_(ID_a6Jr23nkqzyIhe3IaTLjpQ) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: quoted-printable
On May 20= , 2019, at 7:18 PM, Liming Gao <liming.gao@intel.com> wrote:

Andrew:
  Thanks for your sharing. I would like the off= icial LLVM tool chain instead of the customized version.
 

Liming,

Hopefully all the bits are in place so you can cross co= mpile for Linux on Windows, and Windows on Linux so you will not need a cus= tomer binary. If not that seems like something reasonable to ask for, bur r= emember some one is going to ask for tests. 

=
On thing to try is to get the default triple on Linux and try it= Window, and get the default triple on Windows and try it on Linux. if you = `clang -v` that should print out the default triple. From this writeup [1] = some of the 4 fields in the triple can get skipped. You can then try to bui= ld the Windows triple on Linux etc. One of the good things about the edk2 i= s there is not a dependency on the include files or runtime so you just nee= d to make the compiler/linker do the right thing. 



Andrew Fish

  I will investigate it more and go back.= Now, I push this patch set into staging https://github.com/tianocore/edk2-staging= /tree/llvm8 for the evaluation.
 
Thanks
Liming=
From: afish@apple.com [mailto:afish@apple.com] 
Sent: Tuesday, May 21, 2019 6:53 AMTo:&nb= sp;devel@edk2.groups.io; Gao, Limin= g <liming.gao@intel.com>
Cc: = Justen, Jordan L <jordan.l.justen@inte= l.com>; Ard Biesheuvel <ard.bie= sheuvel@linaro.org>
Subject: Re: [edk2-devel] [Patch 0/7] Add= new CLANG8ELF tool chain for new LLVM/CLANG8<= /div>
 
&= nbsp;


On May 20, 2019, at 6:47 AM, Limin= g Gao <liming.gao@intel.com> wrote:<= o:p class=3D"">
 =
Jordan:


-----Original Message-----
From: J= usten, Jordan L
Sent: Monday, May 20, 2019 4:15 AM
To: Ard Biesheuvel <ard.biesheu= vel@linaro.org>; Gao, Liming <liming= .gao@intel.com>; edk2-devel-groups-io <devel@edk2.groups.io>
Subject: Re: [edk2-devel] [Pa= tch 0/7] Add new CLANG8ELF tool chain for new LLVM/CLANG8
On 2019-04-27 17:55:02, Liming Gao wrote:

-----Original Message-----
From: Ard B= iesheuvel [mailto:ard.biesheuvel@linaro.o= rg]
Sent: Saturday, April 27, 2019 12:33 AM


This series confuses me. The existing CLANGx= x toolchains already use
GenFw and ELF to PE/COFF conversion,= so the name CLANG8ELF is
misleading.

LLVM/CLANG8.0 compiler supports to generate PE image or ELF image= .
This tool chain is to generate ELF image and be converted t= o PE
image. I am investigating another tool chain with CLANG8= .0 to
directly generate PE image. To differentiate them, I us= e the tool
chain name CLANG8ELF and CLANG8PE for them.


Assuming CLANG8ELF and CLANG8= PE were functional, would both be needed?
It kind of sounds l= ike this a half-finished investigation.

One= point is that they will generate the different debug format symbols (WinDB= G or GDB).
I have not done the full investigation to generate= the different debug format symbols in single tool chain.

 
 
Liming,
 
=
= I don't know the current answer but we moved over to using clang a while ba= ck on Mac so here are some general thoughts incase they help your investiga= tion. 
 
We had to take a di= fferent approach for i386 vs x86_64.:
1) i386
For i386 we used -arch i386 and compiler flags to produce the corre= ct ABI. The thing about using -arch is the target triple defaults to your c= urrent machine, thus -arch i386 on a Mac is going to make a Mach-O object. = if you use it one Linux it is going to make an ELF, on Windows a PE/COFF. T= o make this work we add mtoc to CCTOOLS this is a tool that converts Mach-O= to PE/COFF. The PE/COFF contains a UUID for the Mach-O and the Mach-O dSYM= (kind of like the PDB file) and that is what is used for debugging. You ca= n even load the Mach-O directly into the debugger and do offline investigat= ion of the image at its linked address. <= /div>
 
<= div class=3D"">
2) x86_64
Since the Mac uses the System V ABI we needed a cross compiler. Clan= g implements cross compilers via the Triple (this is the -target argument y= ou pass to clang). We actually end up open sourcing a triple that did what = we needed. 
 
For building o= n a Mac the triple is: -target x86_64-pc-win32-macho
 <= /div>
 <arch><sub>-&l= t;vendor>-<sys>-<abi>
 < x86_64 ><>-< pc &nb= sp;   >-< win32 >-< macho >
 
Examples:
arch =3D x86_64, i386, = arm, thumb, mips, etc.
sub =3D for ex. on ARM: v5, v6m, v7a, v7m, etc.
vendor =3D pc, apple, nvidia, ibm, e= tc.
sys =3D none, l= inux, win32, darwin, cuda, etc.
abi =3D eabi, gnu, android, macho, elf, etc.
 <= /div>
OK so they expanded the Triple to have 4 fields, so that i= s kind of naming fail, but they still seem to call it a triple. When we did= the port "-DEFIAPI=3D__attribute__((ms_abi))" was not an option and that i= s why we ended up having to male our own triple. As far as I can tell __att= ribute__((ms_abi)) works correctly in the Xcode compiler. 
 
To make the debugging work we ended = up defining a new CodeView signature [1] to the PE/COFF Debug Directory Ent= ry. At this point the native lldb (llvm debugger) does not know how to load= Mach-O DWARF from PE/COFF. We could teach it, but we have not really found= a need as we ended up writing debugger scripts to load symbols and lldb ca= n load the Mach-O image (our build system leaves them around) for offline d= ebugging.
 
In terms of PDB vs DW= ARF generally the compiler emits the DWARF (or PDB) data at compile time an= d the linker just aggregates it together. I see a blog from last April [2] = talking about LLVM PDB support and it ends with "Do I hear cross-compilatio= n?". So you might have to compile from the start for PDB or DWARF. This see= ms to imply you are going to need a triple to invoke a cross compiler? Also= you are going to need that triple present in the clang binary, so you migh= t be able to make a custom version of clang that supports the triples you n= eed, but that may not be in the official binary for clang. 
 
Converting ELF to PE/COFF is much mo= re straight forward than converting the debugging information so that makes= sense that it could just be the backend of the linker. 
 
<= span lang=3D"EN-US" class=3D"">I look forward to getting the results of you= r investigation.
 
[1] https://github.= com/tianocore/edk2/blob/master/MdePkg/Include/IndustryStandard/PeImage.h#L6= 47
[2] = h= ttp://blog.llvm.org/2017/08/llvm-on-windows-now-supports-pdb-debug.html=
&n= bsp;
Thanks,<= /span>
 
=
Andrew Fish
=
 
<= blockquote style=3D"margin-top: 5pt; margin-bottom: 5pt; font-variant-caps:= normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0= px;" class=3D"">
I'm guessing that if CLANG8PE produces equal or better code, then it
would be preferred.

Therefore, shou= ldn't we just finish the investigation, and add a
single CLAN= G8 toolchain at the end? Or, maybe to reflect that it
mostly = uses the LLVM tool stack we could name it LLVM8.
I also prefer single CLANG8 tool chain. I will collect more infor= mation and see whether it is possible. 
Now, I would like to add this tool chain for t= he developer evaluation. Because I can't address all 
comments now, I will remove this= feature from Q2 stable tag. I will add it into edk2-staging first. 


-Jordan
<= /blockquote>
<= br class=3D"">=
 <= /div>

--Boundary_(ID_a6Jr23nkqzyIhe3IaTLjpQ)--