From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from nwk-aaemail-lapp02.apple.com (nwk-aaemail-lapp02.apple.com [17.151.62.67]) by mx.groups.io with SMTP id smtpd.web12.12550.1584810790768812476 for ; Sat, 21 Mar 2020 10:13:11 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=EUBAC/CX; spf=pass (domain: apple.com, ip: 17.151.62.67, mailfrom: afish@apple.com) Received: from pps.filterd (nwk-aaemail-lapp02.apple.com [127.0.0.1]) by nwk-aaemail-lapp02.apple.com (8.16.0.27/8.16.0.27) with SMTP id 02LHCJjo056543; Sat, 21 Mar 2020 10:13:09 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=sender : from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=iVo1147xXr8STwig1Vm8T4XpF05QColYlmZp32bXAJQ=; b=EUBAC/CXfZMhp/C5IhNfKxhJVo85gYdU4O+Sb7Oiz8CJsVNjpSB2iEao2JcdjqTQEvGV GSPtLoQFNYpsZYusce3vnL2cbSI5VaKtAcXaE6M4UmOD3fqANmmFjwlsToA0U4PSbo7F YLuFzHnJMa/MlzcfkcvNHdGQ7aehdCgGvn+geZIY4Rqqik8NQkRsCEp9exfl3a5eptp7 jmO1R8yx9W+CFi2PbkPM2ogxsHLZXuA/7JTv5to6dx0XWIKYdPya/dni1uoLswk2h0Cb yGsOP2kQ6rfUv15IdiHBr/YcOD4zgplTZjohtPtnh3ykwxgYihmmCLD3WDJjNlIPqCth PA== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by nwk-aaemail-lapp02.apple.com with ESMTP id 2ywfgfencu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sat, 21 Mar 2020 10:13:09 -0700 Received: from nwk-mmpp-sz11.apple.com (nwk-mmpp-sz11.apple.com [17.128.115.155]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) with ESMTPS id <0Q7J00B7VZTXY190@rn-mailsvcp-mta-lapp01.rno.apple.com>; Sat, 21 Mar 2020 10:13:09 -0700 (PDT) Received: from process_milters-daemon.nwk-mmpp-sz11.apple.com by nwk-mmpp-sz11.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) id <0Q7J00D00XRFQY00@nwk-mmpp-sz11.apple.com>; Sat, 21 Mar 2020 10:13:09 -0700 (PDT) X-Va-A: X-Va-T-CD: 2c323b19c9bf43c24bf8f3f2bb9cbf14 X-Va-E-CD: 36a32cfa42a5ae8aa02473601c03cec7 X-Va-R-CD: d2d46cdd9ed31419d4eb89636f43916d X-Va-CD: 0 X-Va-ID: cdd848ee-97ed-4ba1-8f40-4c58a763f2ee X-V-A: X-V-T-CD: 2c323b19c9bf43c24bf8f3f2bb9cbf14 X-V-E-CD: 36a32cfa42a5ae8aa02473601c03cec7 X-V-R-CD: d2d46cdd9ed31419d4eb89636f43916d X-V-CD: 0 X-V-ID: 30ef8a6c-7612-4e3d-86d8-58e2899ec8b7 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.645 definitions=2020-03-21_07:2020-03-20,2020-03-21 signatures=0 Received: from [17.235.38.16] by nwk-mmpp-sz11.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPSA id <0Q7J0001VZTUN200@nwk-mmpp-sz11.apple.com>; Sat, 21 Mar 2020 10:13:07 -0700 (PDT) Sender: afish@apple.com From: "Andrew Fish" Message-id: <63396616-D8CF-4135-B967-772C1E6136BD@apple.com> MIME-version: 1.0 (Mac OS X Mail 13.0 \(3594.4.17\)) Subject: Re: [edk2-devel] CLANGPDB binary debugging Date: Sat, 21 Mar 2020 10:13:05 -0700 In-reply-to: Cc: "Gao, Liming" , =?utf-8?Q?Marvin_H=C3=A4user?= To: devel@edk2.groups.io, cheptsov@ispras.ru References: <9804C565-0C9E-4778-92A7-06EA6AD8D694@ispras.ru> <7E18AD8F-9A44-45FE-A8C8-CE06A6328930@apple.com> X-Mailer: Apple Mail (2.3594.4.17) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.645 definitions=2020-03-21_07:2020-03-20,2020-03-21 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_6517C3C0-C420-4505-8017-42A6662F6C15" --Apple-Mail=_6517C3C0-C420-4505-8017-42A6662F6C15 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 21, 2020, at 3:28 AM, Vitaly Cheptsov wrote: >=20 > Hello, >=20 > Andrey, thanks for the hint, it was very helpful. I rewrote the GDB scri= pts to work with LLDB[1] and was able to debug OVMF built with CLANGPDB. Wh= ile it is still quite dirty, at the very least it works. >=20 > Unfortunately the experience was close to terrible. I may certainly do s= omething wrong, but it is clear that PDB and LLDB do not support each other= well enough. After spending several hours on playing with the tools my con= clusion is that LLDB is simply not suited for UEFI PDB debugging, and we re= ally want DWARF as there is no other opensource debugger that supports PDB= on macOS and Linux >=20 > In case somebody knows workarounds here are the issues I faced: >=20 > 1. All integer alias typedefs are discarded in favour of underlying type= s. This way EFI_STATUS and EFI_TPL become unsigned long long, CHAR8 becomes= char, and CHAR16 becomes unsigned short. It does not look like LLDB has th= e original types anywhere at all, and it also does not have them registered= . >=20 > frame #0: 0x000000007fe242aa DxeCore.dll`CoreAllocatePoolPagesI(Pool= Type=3DEfiBootServicesData, NoPages=3D1, Granularity=3D4096, NeedGuard=3D'\= 0') at Pool.c:322 > 319 =09 return NULL; > 320 =09 } > 321 =09 > -> 322 =09 Buffer =3D CoreAllocatePoolPages (PoolType, NoPages, Granula= rity, NeedGuard); > 323 =09 CoreReleaseMemoryLock (); > 324 =09 > 325 =09 if (Buffer !=3D NULL) { > (lldb) p Status > (unsigned long long) $3 =3D 0 >=20 > Structures work more or less fine, but for simpler types like strings we= are out of even potential pretty-printing. >=20 Vitaly, You can teach lldb about types. There is some example code here: https://g= ithub.com/tianocore/edk2/blob/master/EmulatorPkg/Unix/lldbefi.py > 2. Global variables are not accessible. I am not sure what happens, but = they either seem to not relocate or conflict with the other names: >=20 > (lldb) p gST > error: Couldn't materialize: couldn't get the value of variable ::gST: r= ead memory from 0x6e18 failed > error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression > (lldb) p &gST > error: Couldn't materialize: couldn't get the value of variable ::gST: r= ead memory from 0x6e18 failed > error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression >=20 That is strange as globals usually work best? The common issue I've seen i= s getting the slide wrong. The EFI modules are linked at a value near zero = and relocated into memory, so the slide represents that adjustment.=20 You can use `image dump sections` and ` image dump symtab` to see lldb's v= iew of symbols. More info here [1].=20 > 3. Quite a number of crashes. >=20 > In most cases autocompletion by tab press causes a crash. E.g. >=20 > b I >=20 > So will do printing of a GUID, e.g. p gEfiGlobalVariableGuid. >=20 > This may have to do with Python compatibility as Xcode 11 LLDB that uses= Python 3 generally crashes more often than MacPorts LLDB 9.0. Surprisingly= structures work more or less fine. >=20 You can tell lldb to use the older Python like this (from the Terminal.app= ): $ defaults write com.apple.dt.lldb DefaultPythonVersion 2 > 4. Ctrl+C does not produce a valid backtrace. When I break with a breakp= oint, I see a proper stacktrace with more than one entry, with function pro= totypes and values. When I break with Ctrl+C I only see some weird backtrac= e with most of the entries missing regardless of frame position: >=20 > (lldb) bt > * thread #1, stop reason =3D signal SIGTRAP > * frame #0: 0x000000007fe4c5f3 DxeCore.dll >=20 > Probably more and all the unintuitive stuff like the lack of more functi= onal TUI, but it is hard to remember all the trials. >=20 For the macOS API clang emits frame pointers, so you can walk the stack wi= thout symbols. You could try adding the compiler flag to emit the frame poi= nters.=20 > [1] https://github.com/acidanthera/OpenCorePkg/blob/master/Debug/Scripts= /lldb_uefi.py >=20 On macOS the Mach-O and dSYM have a UUID (dwarfdump -u) that is indexed by= Spotlight (mdfind "com_apple_xcode_dsym_uuids =3D=3D *") [2] This should be the UUID in the debug directory entry and you can use that = to lookup the symbols like this: module =3D target.AddModule (None, None, uuid) SBError =3D target.SetModuleLoadAddress (module, LoadAddress + TeAdjust) Also lldb has built in help for commands, but it is kind of terse since it= is autogenerated from the C++ swig.=20 (lldb) script help (lldb.target.AddModule) Help on method AddModule in module lldb: AddModule(self, *args) method of lldb.SBTarget instance AddModule(SBTarget self, SBModule module) -> bool AddModule(SBTarget self, char const * path, char const * triple, char = const * uuid) -> SBModule AddModule(SBTarget self, char const * path, char const * triple, char = const * uuid_cstr, char const * symfile) -> SBModule AddModule(SBTarget self, SBModuleSpec module_spec) -> SBModule The minimum you need to symbolicate a frame is uuid, LoadAddress, and PC.= = =20 [1] http://lldb.llvm.org/use/map.html [2] http://lldb.llvm.org/use/symbols.html Thanks, Andrew Fish > Best wishes, > Vitaly >=20 >> 20 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 2020 =D0=B3., =D0=B2 22:14, Andrew Fi= sh > =D0=BD=D0=B0=D0=BF=D0=B8=D1= =81=D0=B0=D0=BB(=D0=B0): >>=20 >>=20 >>=20 >>> On Mar 20, 2020, at 8:13 AM, Vitaly Cheptsov > wrote: >>>=20 >>> Hello, >>>=20 >>> We noticed that the original bugzilla, which intended to add new LLVM = toolchain support[1], also wanted to bring ELF format support with DWARF de= bugging information. For some reason this did not make its way into EDK II,= and we are currently wondering, how can one debug binaries built with LLVM= 9.0. >>>=20 >>> For macOS and XCODE5 toolchain we use GDB scripts based on Andrei Wark= entin=E2=80=99s work, which allow us to integrate with QEMU and VMware[2]. = It is likely that they should work with little to no work on Linux with CLA= NG38/GCC5 with GDB once again. However, CLANGPDB apparently is using PDB de= bugging information, which I believe is not handled with GDB. >>>=20 >>> Could you please provide the details on the matter and let us know abo= ut the recommended route? >>> =E2=80=94 Is dropping CLANGELF just a temporary measure and it should = be resubmitted again? >>> =E2=80=94 Should LLDB, which seems to be aware of PDB, be used instead= of GDB, when building with CLANGPDB? If so, did anybody try that? >>>=20 >>=20 >> Vitaly, >>=20 >> I've not tried the CLANGPDB path, but if you want to connect lldb to QE= MU you need to set plugin.process.gdb-remote.target-definition-file [1] to= [2].=20 >>=20 >> [1] lldb -o "settings set plugin.process.gdb-remote.target-definition-= file x86_64_target_definition.py" -o "gdb-remote 9000" >> [2] https://github.com/llvm-mirror/lldb/blob/master/examples/python/x86= _64_target_definition.py >>=20 >> Thanks, >>=20 >> Andrew Fish >>=20 >>> Thanks! >>>=20 >>> Best regards, >>> Vitaly >>>=20 >>> [1] https://bugzilla.tianocore.org/show_bug.cgi?id=3D1603 >>> [2] https://github.com/acidanthera/OpenCorePkg/blob/master/Debug/Scrip= ts/gdb_uefi.py >>>=20 >=20 >=20 --Apple-Mail=_6517C3C0-C420-4505-8017-42A6662F6C15 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Mar 21, 2= 020, at 3:28 AM, Vitaly Cheptsov <cheptsov@ispras.ru> wrote:

Hello,


Unfortunately the experience was close to terrible. I may certa= inly do something wrong, but it is clear that PDB and LLDB do not support e= ach other well enough. After spending several hours on playing with the too= ls my conclusion is that LLDB is simply not suited for UEFI PDB debugging, = and we really want DWARF  as there is no other opensource debugger tha= t supports PDB on macOS and Linux

In case so= mebody knows workarounds here are the issues I faced:

1. All integer alias typedefs are discarded in favour of underlyi= ng types. This way EFI_STATUS and EFI_TPL become unsigned long long, CHAR8 = becomes char, and CHAR16 becomes unsigned short. It does not look like LLDB= has the original types anywhere at all, and it also does not have them reg= istered.

=
    frame #0: 0= x000000007fe242aa DxeCore.dll`CoreAllocatePoolPagesI(PoolType=3DEfiBoo= tServicesData, NoPages=3D1, Granularity=3D4096, NeedGuard=3D'\0') at P= ool.c:322
   319      return NULL;<= br class=3D"">   320    }
   321&nb= sp;
-> 322    Buffer =3D CoreAllocatePoolPages (PoolType, No= Pages, Granularity, NeedGuard);
   323    Cor= eReleaseMemoryLock ();
   324 
  &n= bsp;325    if (Buffer !=3D NULL) {
(lldb) p S= tatus
(unsigned long long) $3 =3D 0

Structures work= more or less fine, but for simpler types like strings we are out of even p= otential pretty-printing.


V= italy,

You can teach lldb about types. = There is some example code here: https://github.= com/tianocore/edk2/blob/master/EmulatorPkg/Unix/lldbefi.py

2. Global variables are not accessible. I am not = sure what happens, but they either seem to not relocate or conflict with th= e other names:

=
(lldb) p gST
error: Couldn't materialize: couldn't get the value of variable ::gST: r= ead memory from 0x6e18 failed
error: errored out in DoExecute= , couldn't PrepareToExecuteJITExpression
(lldb) p &g= ST
error: Couldn't materialize: couldn't get the value of var= iable ::gST: read memory from 0x6e18 failed
error: errored ou= t in DoExecute, couldn't PrepareToExecuteJITExpression


That is strange as globals usually work best? The comm= on issue I've seen is getting the slide wrong. The EFI modules are linked a= t a value near zero and relocated into memory, so the slide represents that= adjustment. 

You can use `image d= ump sections` and ` image dump symtab` to see lldb's view of symbols. = More info here [1]. 

3. Quite a = number of crashes.

In most cases autocomp= letion by tab press causes a crash. E.g.

b I= <TAB>

So will do printing of a GUID, e= .g. 
This may have to do with Python compatibili= ty as Xcode 11 LLDB that uses Python 3 generally crashes more often than Ma= cPorts LLDB 9.0. Surprisingly structures work more or less fine.


You can tell lldb to use the older Python= like this (from the Terminal.app):
$ defaults write com.app= le.dt.lldb DefaultPythonVersion 2

4. Ctrl+C = does not produce a valid backtrace. When I break with a breakpoint, I see a= proper stacktrace with more than one entry, with function prototypes and v= alues. When I break with Ctrl+C I only see some weird backtrace with most o= f the entries missing regardless of frame position:

(lldb) bt=
* thread #1, stop reason =3D signal SIGTRAP
  * frame #0: 0x000000007fe4c5f3 DxeCore.dll<= /div>

Probably more and all the unin= tuitive stuff like the lack of more functional TUI, but it is hard to remem= ber all the trials.

For the= macOS API clang emits frame pointers, so you can walk the stack without sy= mbols. You could try adding the compiler flag to emit the frame pointers.&n= bsp;


[1] https://github.com/acidanthera/= OpenCorePkg/blob/master/Debug/Scripts/lldb_uefi.py


On macOS the Mach-O and dSYM have a UUID (dwarfdump -u= ) that is indexed by Spotlight (mdfind "com_apple_xcode_dsym_uuids =3D=3D *= ") [2]
This should be the UUID in the debug directory entry and y= ou can use that to lookup the symbols like this:

<= /div>
module =3D target.AddModule (None, None, uuid)
SBError = = =3D target.SetModuleLoadAddress (module, LoadAddress + TeAdjust)

Also lldb has built in help for commands, but it= is kind of terse since it is autogenerated from the C++ swig. 
<= div>
(lldb) script help (lldb.target.AddModule)
Help on method AddModul= e in module lldb:

AddModule(self, *args) meth= od of lldb.SBTarget instance
    AddModule(SBTarget self, SBModule = module) -> bool
    AddModule(SBTarget self, char const * path, = char const * triple, char const * uuid) -> SBModule
    AddModul= e(SBTarget self, char const * path, char const * triple, char const * uuid_= cstr, char const * symfile) -> SBModule
    AddModule(SBTarget s= elf, SBModuleSpec module_spec) -> SBModule
<= span style=3D"font-variant-ligatures: no-common-ligatures" class=3D"">
The minimum  you need to symbolicat= e a frame is uuid, LoadAddress, and PC. 


Thanks,

Andrew Fish

=

Best wishes,
Vitaly

20 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 2020 =D0=B3., =D0=B2 22:= 14, Andrew Fish <afish@app= le.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0):


On Mar 20, 2020, at 8:13 AM, Vitaly Chep= tsov <cheptsov@ispras.r= u> wrote:

Hello,

We noticed that the original bugzilla, which intended to = add new LLVM toolchain support[1], also wanted to bring ELF format support = with DWARF debugging information. For some reason this did not make its way into EDK II, an= d we are currently wondering, how can one debug binaries built with LLVM 9.= 0.

For macOS and XCODE5 toolchain we use GDB = scripts based on Andrei Warkentin=E2=80=99s wo= rk, which allow us to integrate with QEMU and VMware[2]. It is likely that = they should work with little to no work on Linux with CLANG38/GCC5 with GDB= once again. However, CLANGPDB apparently is using PDB debugging informatio= n, which I believe is not handled with GDB.
Could you please provide the details on t= he matter and let us know about the recommended route?
=E2=80=94 Is dropping CLANGELF just a temporary measure and it should be r= esubmitted again?
=E2=80=94 Should LLDB, which seems to be aware of PD= B, be used instead of GDB, when building with CLANGPDB? If so, did anybody = try that?


Vitaly,
<= br class=3D"">
I've not tried = the CLANGPDB path, but if you want to connect lldb to QEMU you need to set =  plugin.process.gdb-remote.target-definition-file [1] to [2]. 

[1]  lldb -o "settings set plugin.= process.gdb-remote.target-definition-file x86_64_target_definition.py" -o "= gdb-remote 9000"
[2] https://github.com/llvm-mirror/lldb/blob/master/examples/python/x86_= 64_target_definition.py
Thanks,

Andrew Fish

Thanks!

Best regards,<= /span>
Vitaly
=
[1] https://bugzilla.tianocore.= org/show_bug.cgi?id=3D1603
[2] https://github.com/acidanthera/OpenCorePkg/blob/master/Debug= /Scripts/gdb_uefi.py



--Apple-Mail=_6517C3C0-C420-4505-8017-42A6662F6C15--