From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from rn-mailsvcp-ppex-lapp14.apple.com (rn-mailsvcp-ppex-lapp14.apple.com [17.179.253.33]) by mx.groups.io with SMTP id smtpd.web12.3372.1627677945198862263 for ; Fri, 30 Jul 2021 13:45:45 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=SlHON9Z/; spf=pass (domain: apple.com, ip: 17.179.253.33, mailfrom: afish@apple.com) Received: from pps.filterd (rn-mailsvcp-ppex-lapp14.rno.apple.com [127.0.0.1]) by rn-mailsvcp-ppex-lapp14.rno.apple.com (8.16.1.2/8.16.1.2) with SMTP id 16UKgp3r005384; Fri, 30 Jul 2021 13:45:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=VNJgq5PbUZRbaI/AfeMtj4PVderBtJWZtkXTt/N6rJQ=; b=SlHON9Z/eAPaTa4e7j0VTnThutitTXFHFWPcwZ23O25p1459bwyr+PDUnvGpetmNohC3 oF/lDXBGmWLTdYuz9xFLz/UFtWxbNihq3GlHe/QveszuNsSchBwUmnm9BLQ8p02zuxoH YORd4Ng1eiMSD5PvDaw+5IbNeVUdP92BFPBfLBv0OnIYguQetpCCBIpbo0XXp3FoHxCv a0eK2RpYvGzG7ODZQP9oMe3gEHFsMnXn27eeCZxL9xmHpc6hP5d/uP9xKEbnng2Q2UlJ rOqkJi57WOWK16CuPlB6uG7O4AY7N9CaA0R33PrQu/sN4TYBV0LyxizoHNW9cY0BRGvm hw== Received: from rn-mailsvcp-mta-lapp03.rno.apple.com (rn-mailsvcp-mta-lapp03.rno.apple.com [10.225.203.151]) by rn-mailsvcp-ppex-lapp14.rno.apple.com with ESMTP id 3a235r0er4-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 30 Jul 2021 13:45:44 -0700 Received: from rn-mailsvcp-mmp-lapp01.rno.apple.com (rn-mailsvcp-mmp-lapp01.rno.apple.com [17.179.253.14]) by rn-mailsvcp-mta-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QX2007JFSC8JTC0@rn-mailsvcp-mta-lapp03.rno.apple.com>; Fri, 30 Jul 2021 13:45:44 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp01.rno.apple.com by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QX200B00S3FMY00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Fri, 30 Jul 2021 13:45:43 -0700 (PDT) X-Va-A: X-Va-T-CD: cc354bcf01ea39de908abab4e73c9ec0 X-Va-E-CD: 78086c89ca9147d7b0a0fbdb6791b8cc X-Va-R-CD: c1d26d53d1da76757d656d65137baef3 X-Va-CD: 0 X-Va-ID: 2d8ad099-687d-4ec4-ade2-8001e4296510 X-V-A: X-V-T-CD: cc354bcf01ea39de908abab4e73c9ec0 X-V-E-CD: 78086c89ca9147d7b0a0fbdb6791b8cc X-V-R-CD: c1d26d53d1da76757d656d65137baef3 X-V-CD: 0 X-V-ID: d466c986-80af-41b4-ab52-9908c4d17bc9 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-07-30_11:2021-07-30,2021-07-30 signatures=0 Received: from [17.235.58.45] (unknown [17.235.58.45]) by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QX200WIJSC6CB00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Fri, 30 Jul 2021 13:45:43 -0700 (PDT) From: "Andrew Fish" Message-id: MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.1\)) Subject: Re: [edk2-devel] EmulatorPkg and the state of DlLoadImage() Date: Fri, 30 Jul 2021 13:45:41 -0700 In-reply-to: Cc: Ray Ni To: devel@edk2.groups.io, mhaeuser@posteo.de References: <3f9e363e-26cd-cce2-21dc-50962043c56c@posteo.de> <02DBBE3E-4C71-4765-8C58-929B01739C33@apple.com> X-Mailer: Apple Mail (2.3654.20.0.2.1) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-07-30_11:2021-07-30,2021-07-30 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_A426FA0E-3B42-475F-A6C1-03EF34F7995D" --Apple-Mail=_A426FA0E-3B42-475F-A6C1-03EF34F7995D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jul 30, 2021, at 1:34 PM, Marvin H=C3=A4user wro= te: >=20 > 30.07.2021 17:58:05 Andrew Fish via groups.io >: >=20 >>=20 >>=20 >>> On Jul 30, 2021, at 3:37 AM, Marvin H=C3=A4user w= rote: >>>=20 >>> Good day everyone, >>>=20 >>> I'm currently refining the port of EmulatorPkg to my new PE/COFF loade= r library instance. >>> In the process, I found the function DlOpenImage() [1], which loads UE= FI Images via the OS loader to utilise its symbol loading capability. Theor= etically, this should e.g. allow arbitrary debuggers using the OS APIs to s= ymbolise the backtrace. >>>=20 >>> macOS: The function seems to be unused entirely. [2] >>>=20 >>> Linux: On my system running Fedora 34, the function neither works out-= of-the-box, nor after significant time of trying to fix it. The first issue= is that it only proceeds if the Image has a PDB path with ".pdb" extension= [3], while the GCC5 toolchain generates Images with ".dll" files for PDB p= aths (see errors below). Once this is resolved, there is an error message i= ndicating insufficient Image section alignment: >>>=20 >>> [...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Universal/EbcDxe/E= bcDxe/DEBUG/EbcDxe.dll: ELF load command alignment not page-aligned >>>=20 >>=20 >> The requiring *.pdb seems like something that rotted out and could be f= ixed.=20 >=20 > OK, yes. >=20 >>> Resolving this yields an error that executable files cannot be loaded = dynamically: >>>=20 >>> [...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/D= EBUG/PeiCore.dll: cannot dynamically load executable >>>=20 >>> With my very limited knowledge about Linux and ELF I tried the naive a= pproach of building the Images as shared (hoping it would be similar to DLL= s, which are built on Windows), but this just silently crashes. >>>=20 >>=20 >> This code is very very old. Notice the comment about gdb predates gdb P= ython support [1]. >=20 > Right, that is why I wondered, whether it is still needed. >> What happens if you comment out the DlLoadImage path? There seems to be= some gdb scripts? >=20 > It already is pretty much no-op on my system. On break, the debug code w= rites the symbol file paths into a file (Host.gdb) and a gdb script loads t= hem. With gdb, this code seems to not be needed at all. >=20 >> The macOS path sets breakpoints on SecGdbScriptBreak() in an lldb scrip= t and loads symbols via that path. That his probably the best path forward = for gdb too?=20 >> It looks like if you `build.sh run` you should launch the emulator unde= r gdb and source the symbol loading file. >> EmulatorPkg/build.sh:221: /usr/bin/gdb $BUILD_ROOT_ARCH/Host -q -cd=3D= $BUILD_ROOT_ARCH -x $WORKSPACE/EmulatorPkg/Unix/*GdbRun.sh* >>=20 >> If you comment out the dlopen() path does it start working? Looks like = breaking in with gdb should get symbols loaded?=20 >=20 > Yes, it works, even without uncommenting. I'm not experienced with gdb e= nough to know how it should work, but I could help with a better solution i= f such is needed later. For now, I'd like to figure out whether dlopen() is= legacy code nobody uses or needs first. >=20 Marvin, Sorry I started looking at the code and forgot to give the history. The dl= open() was a trick copied from the Windows port. Basically an EFI image is = loaded into EFI memory and it is also loaded into the application memory ou= tside of EFI vis the dlopen() and the entry point is returned to be the dlo= pen() one. This fakes EFI out enough to be happy, and since the EFI code wa= s loaded via dlopen() symbols get loaded automatically. So the only purpose= of the dlopen is automatic source level debugging without requiring any de= bugger scripting.=20 The gdb/lldb scripts have magic to tell the debugger the location things E= FI got loaded, so you don=E2=80=99t need the dlopen() to load symbols. Basi= cally not using the dlopen is a more realistic simulation of EFI.=20 >>=20 >>> So my questions are: >>> 1) Does this code currently work for anyone? >>> 2) Does anyone use a debugging setup that is incompatible with Images = loaded by EDK II rather than the OS? >>=20 >> Not a 100% sure what you are asking? >=20 > Sorry if it was unclear, "loaded by EDK II" refers to Images loaded with= out dlopen() and "loaded by the OS" to Images loaded with dlopen(). >=20 Hopefully the above history clears that up. >> In a lot of cases you are debugging what is compatible with the OS? For= example on macOS we build a mach-O and convert that to PE/COFF. We point t= he PDB entry at the mach-O file and that is what the debugger sees. As long= as the PE/COFF lines up with the mach-O it does not really matter, as at t= he end of the day the debugger is just processing the dwarf debug info asso= ciated with addresses in system memory.=20 >>> 3) Are the issues above known and planned to be fixed? >>>=20 >>=20 >> Not likely please file a BZ.=20 >=20 > OK, will do soon. >=20 >>=20 >> Note I=E2=80=99m working on getting a generic gdb debugging script into= the edk2 [2] and that should also work with the Emulator. I think you coul= d replace the ` -x $WORKSPACE/EmulatorPkg/Unix/GdbRun.sh` with `-ex efi_gdb= .py=E2=80=99. There is not a break hook in those scripts so you would have = to run the `efi` command the 1st time you attach to load symbols. The efi_g= db.py script works on stock EFI so it does not depend on any of the hooks i= n the EmulatorPkg to work.=20 >=20 > I'm not good with Python, so will have to check. :) I was forced to learn Python to figure about debugging scripts :) Thanks, Andrew Fish >=20 > Best regards, > Marvin >=20 >>=20 >>> Thank you for your time! >>>=20 >>> Best regards, >>> Marvin >>>=20 >>>=20 >>> [1] >>> https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe76= 2e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1065-L1113 >>>=20 >>> [2] >>> https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe76= 2e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1071-L1073 >>>=20 >>> [3] >>> https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe76= 2e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1084-L1086 >>> https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe76= 2e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1003-L1026 >>>=20 >>=20 >> [1] https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9f= e762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1179 >> [2] https://github.com/ajfish/edk2/blob/BZ3500-gdb/efi_gdb.py >>=20 >> Thanks, >>=20 >> Andrew Fish >>=20 >=20 >=20 >=20 --Apple-Mail=_A426FA0E-3B42-475F-A6C1-03EF34F7995D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Jul 30, 2= 021, at 1:34 PM, Marvin H=C3=A4user <mhaeuser@posteo.de> wrote:

30.07.2021 17:58:05 Andrew Fish via&= nbsp;groups.io <afish=3Dapple.com@groups.io>:



On Jul 30, 2021, at 3:37 AM, Marvin H=C3= = =A4user <mhaeuser@post= eo.de> wrote:

Good day everyone,

I'm currently refining the port of EmulatorPkg to my= new PE/COFF loader library instance.
In the process, I found= the function DlOpenImage() [1], which loads UEFI Images via the OS loader = to utilise its symbol loading capability. Theoretically, this should e.g. a= llow arbitrary debuggers using the OS APIs to symbolise the backtrace.

macOS: The function seems to be unused entirely. [= 2]

Linux: On my system running Fedora 34, the = function neither works out-of-the-box, nor after significant time of trying= to fix it. The first issue is that it only proceeds if the Image has a PDB= path with ".pdb" extension [3], while the GCC5 toolchain generates Images = with ".dll" files for PDB paths (see errors below). Once this is resolved, = there is an error message indicating insufficient Image section alignment:<= br class=3D"">
[...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModu= lePkg/Universal/EbcDxe/EbcDxe/DEBUG/EbcDxe.dll: ELF load command alignment = not page-aligned


T= he requiring *.pdb seems like something that rotted out and could be fixed.=  

OK, yes.

Resolving this yields a= n error that executable files cannot be loaded dynamically:
<= br class=3D"">[...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/= PeiMain/DEBUG/PeiCore.dll: cannot dynamically load executable

With my very limited knowledge about Linux and ELF I tried = the naive approach of building the Images as shared (hoping it would be sim= ilar to DLLs, which are built on Windows), but this just silently crashes.<= br class=3D"">

This code is very = very old. Notice the comment about gdb predates gdb Python support [1].

Rig= ht, that is why I wondered, whether it is still needed.
What happens if you comment o= ut the DlLoadImage path? There seems to be some gdb scripts?
=

It already is pr= etty much no-op on my system. On break, the debug code writes the symbol fi= le paths into a file (Host.gdb) and a gdb script loads them. With gdb, this= code seems to not be needed at all.

The macOS= path sets breakpoints on SecGdbScriptBreak() in an lldb script and lo= ads symbols via that path. That his probably the best path forward for gdb = too? 
It lo= oks like if you `build.sh run` you should launch the emulator under gdb and= source the symbol loading file.
EmulatorPkg/build.sh:221:&nb= sp; /usr/bin/gdb $BUILD_ROOT_ARCH/Host -q -cd=3D$BUILD_ROOT_ARCH -x $WORKSP= ACE/EmulatorPkg/Unix/*GdbRun.sh*

If you commen= t out the dlopen() path does it start working? Looks like breaking in with = gdb should get symbols loaded? <= /span>

Yes, it works, even without uncommenting. I'm not experienced with g= db enough to know how it should work, but I could help with a better soluti= on if such is needed later. For now, I'd like to figure out whether dlopen(= ) is legacy code nobody uses or needs first.


Marvin,

So= rry I started looking at the code and forgot to give the history. The dlope= n() was a trick copied from the Windows port. Basically an EFI image is loa= ded into EFI memory and it is also loaded into the application memory outsi= de of EFI vis the dlopen() and the entry point is returned to be the dlopen= () one. This fakes EFI out enough to be happy, and since the EFI code was l= oaded via dlopen() symbols get loaded automatically. So the only purpose of= the dlopen is automatic source level debugging without requiring any debug= ger scripting. 

The gdb/lldb scrip= ts have magic to tell the debugger the location things EFI got loaded, so y= ou don=E2=80=99t need the dlopen() to load symbols. Basically not using the= dlopen is a more realistic simulation of EFI. 


So my questions are:
1) Does this code currently work for anyone?
2) Does = anyone use a debugging setup that is incompatible with Images loaded by EDK= II rather than the OS?

Not a 100= % sure what you are asking?

Sorry if it was unclear, "loaded by EDK II" refer= s to Images loaded without dlopen() and "loaded by the OS" to Images loaded= with dlopen().


= Hopefully the above history clears that up.

In a lot o= f cases you are debugging what is compatible with the OS? For example on ma= cOS we build a mach-O and convert that to PE/COFF. We point the PDB entry a= t the mach-O file and that is what the debugger sees. As long as the PE/COF= F lines up with the mach-O it does not really matter, as at the end of the = day the debugger is just processing the dwarf debug info associated with ad= dresses in system memory. 
3) Are the issues abov= e known and planned to be fixed?

=
Not likely please file a BZ. 

OK, will do soon.


Note I=E2=80=99m working on getting a generic gdb debugging script i= nto the edk2 [2] and that should also work with the Emulator. I think you c= ould replace the ` -x $WORKSPACE/EmulatorPkg/Unix/GdbRun.sh` with `-ex=  efi_gdb.py=E2=80=99. There is not a break hook in those scripts so yo= u would have to run the `efi` command the 1st time you attach to load symbo= ls. The efi_gdb.py script works on stock EFI so it does not depend on any o= f the hooks in the EmulatorPkg to work. 

I'm not good with Python, so will have to check. :)

=
I was forced to learn Python to figure about debugging scripts := )

Thanks,

Andrew Fish


Best regards= ,
Marvin


Thank yo= u for your time!

Best regards,
M= arvin


[1]
https://gith= ub.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe762e14ca8c9/Emulato= rPkg/Unix/Host/Host.c#L1065-L1113

[2]
https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a= 9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1071-L1073

[3]
https://github.com/tianocore/edk2/blob/be282b14= 938846960cce30825a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1084-L1086https://github.com/tianocore/edk2/blob/be282b14938846960cce308= 25a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1003-L1026

[1] https://github.com/tianocore/edk2/blo= b/be282b14938846960cce30825a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L11= 79
[2] https://github.com/ajfish/edk2/blob/BZ3= 500-gdb/efi_gdb.py

Thanks,
<= br class=3D"">Andrew Fish




--Apple-Mail=_A426FA0E-3B42-475F-A6C1-03EF34F7995D--