From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ma1-aaemail-dr-lapp01.apple.com (ma1-aaemail-dr-lapp01.apple.com [17.171.2.60]) by mx.groups.io with SMTP id smtpd.web09.25765.1627660682086216885 for ; Fri, 30 Jul 2021 08:58:02 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=uM+BNWkE; spf=pass (domain: apple.com, ip: 17.171.2.60, mailfrom: afish@apple.com) Received: from pps.filterd (ma1-aaemail-dr-lapp01.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp01.apple.com (8.16.0.42/8.16.0.42) with SMTP id 16UFvdck002260; Fri, 30 Jul 2021 08:58:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=50BlfmAfXhlWEF2G2n4+1jNR9UQvE5H6gAJhL3iyr5Q=; b=uM+BNWkEw0AoFYZWQihc1NsYNIys7xAb54oiz+AFN3zyLRmOyp1CJzwfUqzPHA63Rofz sCs1XLMACQMKApieA+utJHfCAxXJHaojWIv5S4bttm6Ay6N/rNdt96bx7hM7pwXQWlk/ wnXOvXrnR8gLzxKW3z/MxZBXEQs+QypR/oVJmnpljFpbj99FxXsiTY0Tfftt3LYwuNrO SzXEDGnQJwO6CiFJEurnz4ytRJcG7bO/2zEdPpuh2PFQgJXOtIdOaMIkUSWT/lq/rMY4 jxMh15Wnszf5oajy1SypaWBt2gU2Gq1RJy2GFvHM2SNH0w3IKBTHKryL7K1LtE+Pvvb1 Ew== Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma1-aaemail-dr-lapp01.apple.com with ESMTP id 3a41bfwr3w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 30 Jul 2021 08:58:01 -0700 Received: from rn-mailsvcp-mmp-lapp01.rno.apple.com (rn-mailsvcp-mmp-lapp01.rno.apple.com [17.179.253.14]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QX200VFZF0O5670@rn-mailsvcp-mta-lapp02.rno.apple.com>; Fri, 30 Jul 2021 08:58:00 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp01.rno.apple.com by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QX200300EZ2N600@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Fri, 30 Jul 2021 08:58:00 -0700 (PDT) X-Va-A: X-Va-T-CD: cb83049425a79c8a5fb9f1dafa0fda92 X-Va-E-CD: 8fc46c176030a47175c9fd2180413777 X-Va-R-CD: b8672115bf73922ac9c53ca660b65ee1 X-Va-CD: 0 X-Va-ID: 334e4643-e632-4f7b-bc88-dd9d61334812 X-V-A: X-V-T-CD: cb83049425a79c8a5fb9f1dafa0fda92 X-V-E-CD: 8fc46c176030a47175c9fd2180413777 X-V-R-CD: b8672115bf73922ac9c53ca660b65ee1 X-V-CD: 0 X-V-ID: 47faceb8-6df4-4b97-a4de-999a3a4619a4 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-07-30_08:2021-07-30,2021-07-30 signatures=0 Received: from [17.235.13.141] (unknown [17.235.13.141]) by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QX20113MF0NUV00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Fri, 30 Jul 2021 08:58:00 -0700 (PDT) From: "Andrew Fish" Message-id: <02DBBE3E-4C71-4765-8C58-929B01739C33@apple.com> MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.1\)) Subject: Re: EmulatorPkg and the state of DlLoadImage() Date: Fri, 30 Jul 2021 08:57:58 -0700 In-reply-to: <3f9e363e-26cd-cce2-21dc-50962043c56c@posteo.de> Cc: "devel@edk2.groups.io" , Ray Ni To: =?utf-8?Q?Marvin_H=C3=A4user?= References: <3f9e363e-26cd-cce2-21dc-50962043c56c@posteo.de> X-Mailer: Apple Mail (2.3654.20.0.2.1) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-07-30_08:2021-07-30,2021-07-30 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_100F95B6-5C6C-4C05-BEA7-351B22648888" --Apple-Mail=_100F95B6-5C6C-4C05-BEA7-351B22648888 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jul 30, 2021, at 3:37 AM, Marvin H=C3=A4user = wrote: >=20 > Good day everyone, >=20 > I'm currently refining the port of EmulatorPkg to my new PE/COFF = loader library instance. > In the process, I found the function DlOpenImage() [1], which loads = UEFI Images via the OS loader to utilise its symbol loading capability. = Theoretically, this should e.g. allow arbitrary debuggers using the OS = APIs to symbolise the backtrace. >=20 > macOS: The function seems to be unused entirely. [2] >=20 > Linux: On my system running Fedora 34, the function neither works = out-of-the-box, nor after significant time of trying to fix it. The = first issue is that it only proceeds if the Image has a PDB path with = ".pdb" extension [3], while the GCC5 toolchain generates Images with = ".dll" files for PDB paths (see errors below). Once this is resolved, = there is an error message indicating insufficient Image section = alignment: >=20 > = [...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Universal/EbcDxe/EbcDx= e/DEBUG/EbcDxe.dll: ELF load command alignment not page-aligned >=20 The requiring *.pdb seems like something that rotted out and could be = fixed.=20 > Resolving this yields an error that executable files cannot be loaded = dynamically: >=20 > = [...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/PeiMain/DEBUG= /PeiCore.dll: cannot dynamically load executable >=20 > With my very limited knowledge about Linux and ELF I tried the naive = approach of building the Images as shared (hoping it would be similar to = DLLs, which are built on Windows), but this just silently crashes. >=20 This code is very very old. Notice the comment about gdb predates gdb = Python support [1]. What happens if you comment out the DlLoadImage path? There seems to be = some gdb scripts? The macOS path sets breakpoints on SecGdbScriptBreak() = in an lldb script and loads symbols via that path. That his probably the = best path forward for gdb too?=20 It looks like if you `build.sh run` you should launch the emulator under = gdb and source the symbol loading file. EmulatorPkg/build.sh:221: /usr/bin/gdb $BUILD_ROOT_ARCH/Host -q = -cd=3D$BUILD_ROOT_ARCH -x $WORKSPACE/EmulatorPkg/Unix/GdbRun.sh If you comment out the dlopen() path does it start working? Looks like = breaking in with gdb should get symbols loaded?=20 > So my questions are: > 1) Does this code currently work for anyone? > 2) Does anyone use a debugging setup that is incompatible with Images = loaded by EDK II rather than the OS? Not a 100% sure what you are asking? In a lot of cases you are debugging = what is compatible with the OS? For example on macOS we build a mach-O = and convert that to PE/COFF. We point the PDB entry at the mach-O file = and that is what the debugger sees. As long as the PE/COFF lines up with = the mach-O it does not really matter, as at the end of the day the = debugger is just processing the dwarf debug info associated with = addresses in system memory.=20 > 3) Are the issues above known and planned to be fixed? >=20 Not likely please file a BZ.=20 Note I=E2=80=99m working on getting a generic gdb debugging script into = the edk2 [2] and that should also work with the Emulator. I think you = could replace the ` -x $WORKSPACE/EmulatorPkg/Unix/GdbRun.sh` with `-ex = efi_gdb.py=E2=80=99. There is not a break hook in those scripts so you = would have to run the `efi` command the 1st time you attach to load = symbols. The efi_gdb.py script works on stock EFI so it does not depend = on any of the hooks in the EmulatorPkg to work.=20 > Thank you for your time! >=20 > Best regards, > Marvin >=20 >=20 > [1] > = https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe762e14= ca8c9/EmulatorPkg/Unix/Host/Host.c#L1065-L1113 >=20 > [2] > = https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe762e14= ca8c9/EmulatorPkg/Unix/Host/Host.c#L1071-L1073 >=20 > [3] > = https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe762e14= ca8c9/EmulatorPkg/Unix/Host/Host.c#L1084-L1086 > = https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe762e14= ca8c9/EmulatorPkg/Unix/Host/Host.c#L1003-L1026 >=20 [1] = https://github.com/tianocore/edk2/blob/be282b14938846960cce30825a9fe762e14= ca8c9/EmulatorPkg/Unix/Host/Host.c#L1179 [2] https://github.com/ajfish/edk2/blob/BZ3500-gdb/efi_gdb.py Thanks, Andrew Fish --Apple-Mail=_100F95B6-5C6C-4C05-BEA7-351B22648888 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Jul 30, 2021, at 3:37 AM, Marvin H=C3=A4user <mhaeuser@posteo.de> = wrote:

Good day everyone,

I'm currently = refining the port of EmulatorPkg to my new PE/COFF loader library = instance.
In the process, I found the function = DlOpenImage() [1], which loads UEFI Images via the OS loader to utilise = its symbol loading capability. Theoretically, this should e.g. allow = arbitrary debuggers using the OS APIs to symbolise the backtrace.

macOS: The function seems to be unused = entirely. [2]

Linux: On my system running = Fedora 34, the function neither works out-of-the-box, nor after = significant time of trying to fix it. The first issue is that it only = proceeds if the Image has a PDB path with ".pdb" extension [3], while = the GCC5 toolchain generates Images with ".dll" files for PDB paths (see = errors below). Once this is resolved, there is an error message = indicating insufficient Image section alignment:

[...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Universal/E= bcDxe/EbcDxe/DEBUG/EbcDxe.dll: ELF load command alignment not = page-aligned


The = requiring *.pdb seems like something that rotted out and could be = fixed. 

Resolving this yields an error that = executable files cannot be loaded dynamically:

[...]/Build/EmulatorX64/DEBUG_GCC5/X64/MdeModulePkg/Core/Pei/Pe= iMain/DEBUG/PeiCore.dll: cannot dynamically load executable

With my very limited knowledge about Linux and = ELF I tried the naive approach of building the Images as shared (hoping = it would be similar to DLLs, which are built on Windows), but this just = silently crashes.


This = code is very very old. Notice the comment about gdb predates gdb Python = support [1].

What happens if you = comment out the DlLoadImage path? There seems to be some gdb scripts? = The macOS path sets breakpoints on SecGdbScriptBreak() in an lldb = script and loads symbols via that path. That his probably the best path = forward for gdb too? 

It looks = like if you `build.sh run` you should launch the emulator under gdb and = source the symbol loading file.
EmulatorPkg/build.sh:221:  /usr/bin/gdb = $BUILD_ROOT_ARCH/Host -q -cd=3D$BUILD_ROOT_ARCH -x = $WORKSPACE/EmulatorPkg/Unix/GdbRun.sh

If you comment out the dlopen() path does it = start working? Looks like breaking in with gdb should get symbols = loaded? 

So my questions are:
1) Does this code currently work for anyone?
2) = Does anyone use a debugging setup that is incompatible with Images = loaded by EDK II rather than the OS?

Not a = 100% sure what you are asking? In a lot of cases you are debugging what = is compatible with the OS? For example on macOS we build a mach-O and = convert that to PE/COFF. We point the PDB entry at the mach-O file and = that is what the debugger sees. As long as the PE/COFF lines up with the = mach-O it does not really matter, as at the end of the day the debugger = is just processing the dwarf debug info associated with addresses in = system memory. 

3) Are the issues above known = and planned to be fixed?


Not = likely please file a BZ. 

Note = I=E2=80=99m working on getting a generic gdb debugging script into the = edk2 [2] and that should also work with the Emulator. I think you could = replace the ` -x $WORKSPACE/EmulatorPkg/Unix/GdbRun.sh` with = `-ex efi_gdb.py=E2=80=99. There is not a break hook in those = scripts so you would have to run the `efi` command the 1st time you = attach to load symbols. The efi_gdb.py script works on stock EFI so it = does not depend on any of the hooks in the EmulatorPkg to = work. 

Thank you for your time!

Best regards,
Marvin


[1]
https://github.com/tianocore/edk2/blob/be282b14938846960cce3082= 5a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1065-L1113

[2]
https://github.com/tianocore/edk2/blob/be282b14938846960cce3082= 5a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1071-L1073

[3]
https://github.com/tianocore/edk2/blob/be282b14938846960cce3082= 5a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1084-L1086
https://github.com/tianocore/edk2/blob/be282b14938846960cce3082= 5a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1003-L1026


[1] https://github.com/tianocore/edk2/blob/be282b14938846960cce3082= 5a9fe762e14ca8c9/EmulatorPkg/Unix/Host/Host.c#L1179
[2] https://github.com/ajfish/edk2/blob/BZ3500-gdb/efi_gdb.py

Thanks,

Andrew Fish

= --Apple-Mail=_100F95B6-5C6C-4C05-BEA7-351B22648888--