Andrew, Thanks once again, but unfortunately it is not that simple. Below I answered inline explaining the particular issues, which mostly seem to be specific to CLANGPDB. LLVM stack emits PDB debug files, and even though LLDB does support them to some level, it is unlikely that this will be working well enough soon. We should really stick to more or less native debug formats, ideally those that have proper open specifications, on all platforms, and for Unix that’s DWARF. I am pretty sure LLVM can be taught to emit DWARF debug information even for PE files. Perhaps we can either make some option or provide a separate toolchain for this? Another way would be recovering CLANGELF as originally suggested. > You can teach lldb about types. There is some example code here: https://github.com/tianocore/edk2/blob/master/EmulatorPkg/Unix/lldbefi.py This code works just fine with LLDB and DWARF (e.g. XCODE5), though I have not yet completed these changes for my scripts for LLDB, only for GDB. However, with CLANGPDB generated files it is not functional. The reason for this is because LLDB is unaware of the underlying type, i.e. it does not know what is EFI_STATUS or UINT32. I can implement pretty-printing when LLDB knows about a typedef, but it is not possible to do this when the debug information is already gone or not parsed: (lldb) p Status (unsigned long long) $1 = 0 (lldb) p &Status (unsigned long long *) $2 = 0x000000007fe19ad8 (lldb) p (EFI_STATUS)Status error: use of undeclared identifier 'EFI_STATUS' Just in case I tried using exactly your code, and other stuff like source level debugging works just fine and symbolication works fine, so it should be some bug with PDB in particular. > That is strange as globals usually work best? The common issue I've seen is getting the slide wrong. The EFI modules are linked at a value near zero and relocated into memory, so the slide represents that adjustment. > > You can use `image dump sections` and ` image dump symtab` to see lldb's view of symbols. More info here [1]. Yes, this one is a bit complicated, once again due to PDB most likely. It knows about global symbols, but does not list them in symtab: (lldb) image dump symtab Dumping symbol table for 91 modules. Symtab, file = GdbSyms/Bin/X64_CLANGPDB/GdbSyms.dll, num_symbols = 0 Symtab, file = /Users/user/Documents/UefiWorkspace/Build/OvmfX64/NOOPT_CLANGPDB/X64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll, num_symbols = 0 Symtab, file = /Users/user/Documents/UefiWorkspace/Build/OvmfX64/NOOPT_CLANGPDB/X64/MdeModulePkg/Universal/DevicePathDxe/DevicePathDxe/DEBUG/DevicePathDxe.dll, num_symbols = 0 … The slides are correct, but there are two nuances that collide with it. 1. There are multiple instances of the globals with the same name (e.g. gBS), but for some reason LLDB always tries to print the globals from the first module. This happens even when I am source-level debugging, and I see a gBS symbol from another module (e.g. DxeCore) used right at the same line. With GDB the closest symbol is used, but with LLDB it is always coming from the first module. I tried checking expr help to find whether I can pass it a module explicitly, but also failed. 2. To be able to get EFI types to locate the EFI_SYSTEM_TABLE_POINTER I add a dummy GdbSyms image, which is not loaded to the firmware. So basically I cannot slide what is not in the memory, and this is also my first image. I tried deleting it anyhow, but it failed for me. (lldb) image dump sections Dumping sections for 91 modules. Sections for 'GdbSyms/Bin/X64_CLANGPDB/GdbSyms.dll' (x86_64): SectID Type Load Address Perm File Off. File Size Flags Section Name ---------- ---------------- --------------------------------------- ---- ---------- ---------- ---------- ---------------------------- 0xffffffffffffffff container [0x0000000000000000-0x0000000000006ec0)* --- 0x00000000 0x00000000 0x00000000 GdbSyms.dll. 0x00000001 code [0x0000000000000220-0x0000000000005bd6)* --- 0x00000220 0x000059c0 0x60000020 GdbSyms.dll...text 0x00000002 data [0x0000000000005be0-0x0000000000006d79)* --- 0x00005be0 0x000011a0 0x40000040 GdbSyms.dll...rdata 0x00000003 data [0x0000000000006d80-0x0000000000006e30)* --- 0x00006d80 0x00000060 0xc0000040 GdbSyms.dll...data 0x00000004 regular [0x0000000000006e40-0x0000000000006ea4)* --- 0x00006de0 0x00000080 0x42000040 GdbSyms.dll...reloc Sections for '/Users/user/Documents/UefiWorkspace/Build/OvmfX64/NOOPT_CLANGPDB/X64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll' (x86_64): SectID Type Load Address Perm File Off. File Size Flags Section Name ---------- ---------------- --------------------------------------- ---- ---------- ---------- ---------- ---------------------------- 0xffffffffffffffff container [0x0000000000000000-0x00000000000523a0)* --- 0x00000000 0x00000000 0x00000000 DxeCore.dll. 0x00000001 code [0x000000007fe1b220-0x000000007fe61e34) --- 0x00000220 0x00046c20 0x60000020 DxeCore.dll...text 0x00000002 data [0x000000007fe61e40-0x000000007fe68065) --- 0x00046e40 0x00006240 0x40000040 DxeCore.dll...rdata 0x00000003 data [0x000000007fe68080-0x000000007fe6d160) --- 0x0004d080 0x000018a0 0xc0000040 DxeCore.dll...data 0x00000004 regular [0x000000007fe6d160-0x000000007fe6d398) --- 0x0004e920 0x00000240 0x42000040 DxeCore.dll...reloc Sections for '/Users/user/Documents/UefiWorkspace/Build/OvmfX64/NOOPT_CLANGPDB/X64/MdeModulePkg/Universal/DevicePathDxe/DevicePathDxe/DEBUG/DevicePathDxe.dll' (x86_64): SectID Type Load Address Perm File Off. File Size Flags Section Name ---------- ---------------- --------------------------------------- ---- ---------- ---------- ---------- ---------------------------- 0xffffffffffffffff container [0x0000000000000000-0x0000000000014420)* --- 0x00000000 0x00000000 0x00000000 DevicePathDxe.dll. 0x00000001 code [0x000000007f986220-0x000000007f996cc6) --- 0x00000220 0x00010ac0 0x60000020 DevicePathDxe.dll...text 0x00000002 data [0x000000007f996ce0-0x000000007f999b04) --- 0x00010ce0 0x00002e40 0x40000040 DevicePathDxe.dll...rdata 0x00000003 data [0x000000007f999b20-0x000000007f99a1a2) --- 0x00013b20 0x00000660 0xc0000040 DevicePathDxe.dll...data 0x00000004 regular [0x000000007f99a1c0-0x000000007f99a404) --- 0x00014180 0x00000260 0x42000040 DevicePathDxe.dll…reloc … So, all in all, unique global variables work, but there is no way to access duplicating variables. They either resolve to GdbSyms or just cause a crash: (lldb) p mDebugInfoTableHeader (EFI_DEBUG_IMAGE_INFO_TABLE_HEADER) $0 = { UpdateStatus = 2 TableSize = 92 EfiDebugImageInfoTable = 0x000000007f814018 } (lldb) p gBS error: Couldn't materialize: couldn't get the value of variable ::gBS: read memory from 0x6df8 failed error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression (lldb) p gEfiGlobalVariableGuid 0 libLLVM.dylib 0x000000010e52ee68 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 40 1 libLLVM.dylib 0x000000010e52f262 SignalHandler(int) + 188 2 libsystem_platform.dylib 0x00007fff6ca5642d _sigtramp + 29 ... > You can tell lldb to use the older Python like this (from the Terminal.app): > $ defaults write com.apple.dt.lldb DefaultPythonVersion 2 Thanks, that helped quite a bit, but for some reason Xcode version still crashes more for me. I attached a couple of stack traces if you feel like having a look, but once again it seems that it is all about the PDB plugin. > For the macOS API clang emits frame pointers, so you can walk the stack without symbols. You could try adding the compiler flag to emit the frame pointers. I am pretty sure stack frames are not disabled with UEFI, as sometimes backtracing works just fine. To me it looks like debug information parsing randomly breaks in LLDB, and once it happens it forgets about other images: (lldb) b CoreLocateHandleBuffer Breakpoint 2: where = DxeCore.dll`CoreLocateHandleBuffer + 31 at Locate.c:649, address = 0x000000007fe36e4f (lldb) c Process 1 resuming Process 1 stopped * thread #1, stop reason = breakpoint 2.1 frame #0: 0x000000007fe36e4f DxeCore.dll`CoreLocateHandleBuffer(SearchType=ByProtocol, Protocol=0x000000007f978160, SearchKey=0x0000000000000000, NumberHandles=0x000000007fe19fd8, Buffer=0x000000007fe19fc0) at Locate.c:649 646 EFI_STATUS Status; 647 UINTN BufferSize; 648 -> 649 if (NumberHandles == NULL) { 650 return EFI_INVALID_PARAMETER; 651 } 652 (lldb) bt * thread #1, stop reason = breakpoint 2.1 * frame #0: 0x000000007fe36e4f DxeCore.dll`CoreLocateHandleBuffer(SearchType=ByProtocol, Protocol=0x000000007f978160, SearchKey=0x0000000000000000, NumberHandles=0x000000007fe19fd8, Buffer=0x000000007fe19fc0) at Locate.c:649 frame #1: 0x000000007fe36816 DxeCore.dll`CoreLocateDevicePath(Protocol=0x000000007f978160, DevicePath=0x000000007fe1a060, Device=0x000000007fe1a068) at Locate.c:466 frame #2: 0x000000007f97479a SecurityStubDxe.dll ——— (lldb) b CopyMem Breakpoint 3: 70 locations. (lldb) c Process 1 resuming Process 1 stopped * thread #1, stop reason = breakpoint 2.53 3.53 frame #0: 0x000000007e5c13b3 MnpDxe.dll`CopyMem(DestinationBuffer=0x000000007fe19b50, SourceBuffer=0x000000007e2aa470, Length=656) at CopyMemWrapper.c:47 44 IN UINTN Length 45 ) 46 { -> 47 if (Length == 0) { 48 return DestinationBuffer; 49 } 50 ASSERT ((Length - 1) <= (MAX_ADDRESS - (UINTN)DestinationBuffer)); (lldb) bt * thread #1, stop reason = breakpoint 2.53 3.53 * frame #0: 0x000000007e5c13b3 MnpDxe.dll`CopyMem(DestinationBuffer=0x000000007fe19b50, SourceBuffer=0x000000007e2aa470, Length=656) at CopyMemWrapper.c:47 (lldb) finish error: Could not create return address breakpoint. (lldb) n Process 1 stopped * thread #1, stop reason = step over frame #0: 0x000000007e5c13ce MnpDxe.dll`CopyMem(DestinationBuffer=0x000000007fe19b50, SourceBuffer=0x000000007e2aa470, Length=656) at CopyMemWrapper.c:50 47 if (Length == 0) { 48 return DestinationBuffer; 49 } -> 50 ASSERT ((Length - 1) <= (MAX_ADDRESS - (UINTN)DestinationBuffer)); 51 ASSERT ((Length - 1) <= (MAX_ADDRESS - (UINTN)SourceBuffer)); 52 53 if (DestinationBuffer == SourceBuffer) { (lldb) ... Process 1 stopped * thread #1, stop reason = step over frame #0: 0x000000007e5c14b4 MnpDxe.dll`CopyMem(DestinationBuffer=0x000000007fe19b50, SourceBuffer=0x000000007e2aa470, Length=656) at CopyMemWrapper.c:57 54 return DestinationBuffer; 55 } 56 return InternalMemCopyMem (DestinationBuffer, SourceBuffer, Length); -> 57 } (lldb) Process 1 stopped * thread #1, stop reason = step over frame #0: 0x000000007e5c726e MnpDxe.dll -> 0x7e5c726e: mov rax, qword ptr [rsp + 0x60] 0x7e5c7273: cmp byte ptr [rax + 0x68], 0x0 0x7e5c7277: jne 0x7e5c7291 0x7e5c727d: movabs rax, -0x7fffffffffffffed (lldb) bt * thread #1, stop reason = step over * frame #0: 0x000000007e5c726e MnpDxe.dll ——— (lldb) c Process 1 resuming Process 1 stopped * thread #1, stop reason = signal SIGINT frame #0: 0x000000007fe4d72e DxeCore.dll -> 0x7fe4d72e: cmp al, 0x0 0x7fe4d730: je 0x7fe4d765 0x7fe4d736: mov rcx, qword ptr [rsp + 0x20] 0x7fe4d73b: call 0x7fe4c4b0 (lldb) bt * thread #1, stop reason = signal SIGINT * frame #0: 0x000000007fe4d72e DxeCore.dll > On macOS the Mach-O and dSYM have a UUID (dwarfdump -u) that is indexed by Spotlight (mdfind "com_apple_xcode_dsym_uuids == *") [2] > This should be the UUID in the debug directory entry and you can use that to lookup the symbols like this: > > module = target.AddModule (None, None, uuid) > SBError = target.SetModuleLoadAddress (module, LoadAddress + TeAdjust) > > Also lldb has built in help for commands, but it is kind of terse since it is autogenerated from the C++ swig. > (lldb) script help (lldb.target.AddModule) > Help on method AddModule in module lldb: > > AddModule(self, *args) method of lldb.SBTarget instance > AddModule(SBTarget self, SBModule module) -> bool > AddModule(SBTarget self, char const * path, char const * triple, char const * uuid) -> SBModule > AddModule(SBTarget self, char const * path, char const * triple, char const * uuid_cstr, char const * symfile) -> SBModule > AddModule(SBTarget self, SBModuleSpec module_spec) -> SBModule > > The minimum you need to symbolicate a frame is uuid, LoadAddress, and PC. > > [1] http://lldb.llvm.org/use/map.html > [2] http://lldb.llvm.org/use/symbols.html Thanks for the links again. Yes, I am using some of these, and in fact for GDB that’s pretty much what I did when I worked with XCODE5. It is very likely that when I get to complete LLDB support for XCODE5 it will work quite fine too. But I am already happy with XCODE5 here, and making it even better will only help myself, but not other people with e.g. Linux or people that want me to use the same compiler with them. Best regards, Vitaly > 21 марта 2020 г., в 20:13, Andrew Fish написал(а): > > > >> On Mar 21, 2020, at 3:28 AM, Vitaly Cheptsov > wrote: >> >> Hello, >> >> Andrey, thanks for the hint, it was very helpful. I rewrote the GDB scripts to work with LLDB[1] and was able to debug OVMF built with CLANGPDB. While it is still quite dirty, at the very least it works. >> >> Unfortunately the experience was close to terrible. I may certainly do something wrong, but it is clear that PDB and LLDB do not support each other well enough. After spending several hours on playing with the tools my conclusion is that LLDB is simply not suited for UEFI PDB debugging, and we really want DWARF as there is no other opensource debugger that supports PDB on macOS and Linux >> >> In case somebody knows workarounds here are the issues I faced: >> >> 1. All integer alias typedefs are discarded in favour of underlying types. This way EFI_STATUS and EFI_TPL become unsigned long long, CHAR8 becomes char, and CHAR16 becomes unsigned short. It does not look like LLDB has the original types anywhere at all, and it also does not have them registered. >> >> frame #0: 0x000000007fe242aa DxeCore.dll`CoreAllocatePoolPagesI(PoolType=EfiBootServicesData, NoPages=1, Granularity=4096, NeedGuard='\0') at Pool.c:322 >> 319 return NULL; >> 320 } >> 321 >> -> 322 Buffer = CoreAllocatePoolPages (PoolType, NoPages, Granularity, NeedGuard); >> 323 CoreReleaseMemoryLock (); >> 324 >> 325 if (Buffer != NULL) { >> (lldb) p Status >> (unsigned long long) $3 = 0 >> >> Structures work more or less fine, but for simpler types like strings we are out of even potential pretty-printing. >> > > Vitaly, > > You can teach lldb about types. There is some example code here: https://github.com/tianocore/edk2/blob/master/EmulatorPkg/Unix/lldbefi.py >> 2. Global variables are not accessible. I am not sure what happens, but they either seem to not relocate or conflict with the other names: >> >> (lldb) p gST >> error: Couldn't materialize: couldn't get the value of variable ::gST: read memory from 0x6e18 failed >> error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression >> (lldb) p &gST >> error: Couldn't materialize: couldn't get the value of variable ::gST: read memory from 0x6e18 failed >> error: errored out in DoExecute, couldn't PrepareToExecuteJITExpression >> > > That is strange as globals usually work best? The common issue I've seen is getting the slide wrong. The EFI modules are linked at a value near zero and relocated into memory, so the slide represents that adjustment. > > You can use `image dump sections` and ` image dump symtab` to see lldb's view of symbols. More info here [1]. > >> 3. Quite a number of crashes. >> >> In most cases autocompletion by tab press causes a crash. E.g. >> >> b I >> >> So will do printing of a GUID, e.g. p gEfiGlobalVariableGuid. >> >> This may have to do with Python compatibility as Xcode 11 LLDB that uses Python 3 generally crashes more often than MacPorts LLDB 9.0. Surprisingly structures work more or less fine. >> > > You can tell lldb to use the older Python like this (from the Terminal.app): > $ defaults write com.apple.dt.lldb DefaultPythonVersion 2 > >> 4. Ctrl+C does not produce a valid backtrace. When I break with a breakpoint, I see a proper stacktrace with more than one entry, with function prototypes and values. When I break with Ctrl+C I only see some weird backtrace with most of the entries missing regardless of frame position: >> >> (lldb) bt >> * thread #1, stop reason = signal SIGTRAP >> * frame #0: 0x000000007fe4c5f3 DxeCore.dll >> >> Probably more and all the unintuitive stuff like the lack of more functional TUI, but it is hard to remember all the trials. >> > > For the macOS API clang emits frame pointers, so you can walk the stack without symbols. You could try adding the compiler flag to emit the frame pointers. > > >> [1] https://github.com/acidanthera/OpenCorePkg/blob/master/Debug/Scripts/lldb_uefi.py >> > > On macOS the Mach-O and dSYM have a UUID (dwarfdump -u) that is indexed by Spotlight (mdfind "com_apple_xcode_dsym_uuids == *") [2] > This should be the UUID in the debug directory entry and you can use that to lookup the symbols like this: > > module = target.AddModule (None, None, uuid) > SBError = target.SetModuleLoadAddress (module, LoadAddress + TeAdjust) > > Also lldb has built in help for commands, but it is kind of terse since it is autogenerated from the C++ swig. > (lldb) script help (lldb.target.AddModule) > Help on method AddModule in module lldb: > > AddModule(self, *args) method of lldb.SBTarget instance > AddModule(SBTarget self, SBModule module) -> bool > AddModule(SBTarget self, char const * path, char const * triple, char const * uuid) -> SBModule > AddModule(SBTarget self, char const * path, char const * triple, char const * uuid_cstr, char const * symfile) -> SBModule > AddModule(SBTarget self, SBModuleSpec module_spec) -> SBModule > > The minimum you need to symbolicate a frame is uuid, LoadAddress, and PC. > > [1] http://lldb.llvm.org/use/map.html > [2] http://lldb.llvm.org/use/symbols.html > > Thanks, > > Andrew Fish > > >> Best wishes, >> Vitaly >> >>> 20 марта 2020 г., в 22:14, Andrew Fish > написал(а): >>> >>> >>> >>>> On Mar 20, 2020, at 8:13 AM, Vitaly Cheptsov > wrote: >>>> >>>> Hello, >>>> >>>> We noticed that the original bugzilla, which intended to add new LLVM toolchain support[1], also wanted to bring ELF format support with DWARF debugging information. For some reason this did not make its way into EDK II, and we are currently wondering, how can one debug binaries built with LLVM 9.0. >>>> >>>> For macOS and XCODE5 toolchain we use GDB scripts based on Andrei Warkentin’s work, which allow us to integrate with QEMU and VMware[2]. It is likely that they should work with little to no work on Linux with CLANG38/GCC5 with GDB once again. However, CLANGPDB apparently is using PDB debugging information, which I believe is not handled with GDB. >>>> >>>> Could you please provide the details on the matter and let us know about the recommended route? >>>> — Is dropping CLANGELF just a temporary measure and it should be resubmitted again? >>>> — Should LLDB, which seems to be aware of PDB, be used instead of GDB, when building with CLANGPDB? If so, did anybody try that? >>>> >>> >>> Vitaly, >>> >>> I've not tried the CLANGPDB path, but if you want to connect lldb to QEMU you need to set plugin.process.gdb-remote.target-definition-file [1] to [2]. >>> >>> [1] lldb -o "settings set plugin.process.gdb-remote.target-definition-file x86_64_target_definition.py" -o "gdb-remote 9000" >>> [2] https://github.com/llvm-mirror/lldb/blob/master/examples/python/x86_64_target_definition.py >>> >>> Thanks, >>> >>> Andrew Fish >>> >>>> Thanks! >>>> >>>> Best regards, >>>> Vitaly >>>> >>>> [1] https://bugzilla.tianocore.org/show_bug.cgi?id=1603 >>>> [2] https://github.com/acidanthera/OpenCorePkg/blob/master/Debug/Scripts/gdb_uefi.py >>>> >> >>