Re: [edk2-devel] Help with debugging

["Andrew Fish" <afish@apple.com>]

[-- Attachment #1: Type: text/plain, Size: 19103 bytes --]

Ethin,

USB was designed in the 90’s and people actually worried about the number of Si gates in a mouse, so a vast majority of the complexity of USB is in the USB Host Controller (XHCI is the common  host controller, aka HC). The devices are just end points on the bus. The end points are abstracted via the USB IO protocol. So you just need to match up the USB IO protocol to the USB sub spec for audio devices. The XHCI complexity is abstracted via a USB HC (Host Controller) driver that is consumed by the generic USB Bus driver that enumerates all the devices and produces USB IO. So you can just treat the USB IO protocol as a black box. 

So I think your 1st task is getting your Driver Bindging Supported() function matching on an audio endpoint you want to support. The Supported() will need to return success before your Start() function is called. So maybe take a look at some example USB drivers that sit in the same layer in the stack for different devices? [1]

[1] 
https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Bus/Usb/UsbMouseDxe/UsbMouse.c#L68 <https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Bus/Usb/UsbMouseDxe/UsbMouse.c#L68>
https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Bus/Usb/UsbKbDxe/EfiKey.c#L72 <https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Bus/Usb/UsbKbDxe/EfiKey.c#L72>
https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Bus/Usb/UsbMassStorageDxe/UsbMassImpl.c#L706

Thanks,

Andrew Fish

> On Jun 11, 2021, at 9:47 PM, Ethin Probst <harlydavidsen@gmail.com> wrote:
> 
> Yeah, maybe. Now I just have to figure out where to even begin with
> USB audio. The specs aren't useful in determining where to begin -- or
> at least they aren't from my POV (though that might just be my
> inexperience with USB/XHCI showing).
> 
> On 6/11/21, Andrew Fish <afish@apple.com <mailto:afish@apple.com>> wrote:
>> 
>> 
>>> On Jun 11, 2021, at 4:29 PM, Ethin Probst <harlydavidsen@gmail.com>
>>> wrote:
>>> 
>>> Your suggestion of adding 0x240 worked. I'm able to successfully step
>>> through the code now. Thank you!
>>> 
>> 
>> OK that makes sense. The address in the add-symbol-file command is not the
>> load address of the image, but the start address of the text section. So
>> that is why you had to add 0x240.
>> 
>> Sorry I had to work backwards from how it works, but maybe that info will be
>> helpful for others?
>> 
>> Thanks,
>> 
>> Andrew Fish
>> 
>>> On 6/11/21, Andrew Fish <afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>> wrote:
>>>> 
>>>> 
>>>>> On Jun 11, 2021, at 2:29 PM, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>
>>>>> wrote:
>>>>> 
>>>>> Initial connection and loading symbols:
>>>>> Remote debugging using :1234
>>>>> 0x000000007e4b9517 in ?? ()
>>>>> add symbol table from file
>>>>> "Build/MdeModule/DEBUG_GCC5/X64/UsbAudio.debug"
>>>>> at
>>>>> 	.text_addr = 0x7e4b8000
>>>>> Reading symbols from Build/MdeModule/DEBUG_GCC5/X64/UsbAudio.debug...
>>>>> Expanding full symbols from
>>>>> Build/MdeModule/DEBUG_GCC5/X64/UsbAudio.debug...
>>>>> Backtrace:
>>>>> #0  0x000000007e4b9517 in UefiMain (st=0x7f9ee018,
>>>>> imageHandle=0x7e4f7518) at
>>>>> /home/ethin/source/edk/edk2/MdeModulePkg/Application/UsbAudio/UsbAudio.c:72
>>>>> #1  ProcessModuleEntryPointList (SystemTable=0x7f9ee018,
>>>>> ImageHandle=0x7e4f7518) at
>>>>> /home/ethin/source/edk/edk2/Build/MdeModule/DEBUG_GCC5/X64/MdeModulePkg/Application/UsbAudio/UsbAudio/DEBUG/AutoGen.c:300
>>>>> #2  _ModuleEntryPoint (ImageHandle=0x7e4f7518, SystemTable=0x7f9ee018)
>>>>> at
>>>>> /home/ethin/source/edk/edk2/MdePkg/Library/UefiApplicationEntryPoint/ApplicationEntryPoint.c:59
>>>>> #3  0x000000007fead316 in ?? ()
>>>>> #4  0x000000007e4f7518 in ?? ()
>>>>> #5  0x000000007feab5c7 in ?? ()
>>>>> #6  0x000000007fea3520 in ?? ()
>>>>> #7  0x0000000101000000 in ?? ()
>>>>> #8  0x0000000000000030 in ?? ()
>>>>> #9  0x000000007e4f6018 in ?? ()
>>>>> #10 0x000000007e60a918 in ?? ()
>>>>> #11 0x000000000000011d in ?? ()
>>>>> #12 0x000000007fea3528 in ?? ()
>>>>> #13 0x000000007e4f7818 in ?? ()
>>>>> #14 0x000000007e4f7c98 in ?? ()
>>>>> #15 0x000000007fea3538 in ?? ()
>>>>> #16 0x000000007e3abfca in ?? ()
>>>>> #17 0x000000007e4f7418 in ?? ()
>>>>> #18 0x000000007fea3528 in ?? ()
>>>>> #19 0x0000000000000000 in ?? ()
>>>>> Source-code listing:
>>>>> 1	/** @file
>>>>> 2	  GCC inline implementation of BaseLib processor specific functions.
>>>>> 3	
>>>>> 4	  Copyright (c) 2006 - 2020, Intel Corporation. All rights
>>>>> reserved.<BR>
>>>>> 5	  Portions copyright (c) 2008 - 2009, Apple Inc. All rights
>>>>> reserved.<BR>
>>>>> 6	  SPDX-License-Identifier: BSD-2-Clause-Patent
>>>>> 7	
>>>>> 8	**/
>>>>> 9	
>>>>> 10	
>>>>> Attempt to use "next":
>>>>> 72	} else if (interfaceDescriptor.InterfaceClass == 0x01 &&
>>>>> interfaceDescriptor.InterfaceSubClass == 0x03) {
>>>>> (This is my code but it continuously prints this same line over and
>>>>> over every time "next" is used.)
>>>>> Attempt to use "print Index":
>>>>> No symbol "Index" in current context.
>>>>> info local:
>>>>> UsbIo = 0x0
>>>>> interfaceDescriptor = {Length = 0 '\000', DescriptorType = 8 '\b',
>>>>> InterfaceNumber = 1 '\001', AlternateSetting = 0 '\000', NumEndpoints
>>>>> = 0 '\000', InterfaceClass = 0 '\000', InterfaceSubClass = 0 '\000',
>>>>> InterfaceProtocol = 0 '\000',
>>>>> Interface = 0 '\000'}
>>>>> i = 2118887920
>>>>> numHandles = 264
>>>>> handles = 0x4
>>>>> status = <optimized out>
>>>>> info symbol 0x0007E4B9440:
>>>>> _ModuleEntryPoint + 576 in section .text of
>>>>> /home/ethin/source/edk/edk2/Build/MdeModule/DEBUG_GCC5/X64/UsbAudio.debug
>>>> 
>>>> OK that is interesting…. +576 -> 0x240 witch is about the size of the
>>>> PE/COFF header.
>>>> 
>>>> For mach-O (macOS executables) we have to link at 0x240 to make space
>>>> for
>>>> the PE/COFF header in memory….
>>>> 
>>>> So the PE/COFF header starts at 0x7e4b8000 it is likely the text section
>>>> starts at 0x7e4b8240? So try adding 0x240 to the load address on the
>>>> add-symbol-file command. If that does not work trip subtracting 0x240
>>>> from
>>>> the load address.
>>>> 
>>>> We would need to dump out the UsbAudio.efi file to figure out exactly
>>>> what
>>>> is going on. What distro are you on? Do you have the readpe utility? I’m
>>>> not
>>>> sure what you can dump with objcopy?
>>>> 
>>>> Can you mail me a copy of UsbAudio.efi off list? I can take a quick
>>>> look.
>>>> 
>>>> Thanks,
>>>> 
>>>> Andrew Fish
>>>> 
>>>>> The extra weird thing about this is that CpuDeadLoop() is at the start
>>>>> of the UefiMain function, its not on line 72. The program doesn't even
>>>>> start there -- it starts by attempting to get the list of
>>>>> EFI_USB_IO_PROTOCOL handles available. And GDB is making it look like
>>>>> its skipping all of that.
>>>>> 
>>>>> On 6/11/21, Andrew Fish <afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>> wrote:
>>>>>> 
>>>>>> 
>>>>>>> On Jun 11, 2021, at 1:48 PM, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>
>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> Okay, so I just tried exactly what you told me to do -- use
>>>>>>> CpuDeadLoop() and then just modify index to get out of it. Here's
>>>>>>> what
>>>>>>> I do in GDB:
>>>>>>> - Load the EFI application and connect via target remote :1234
>>>>>>> - type `add-symbol-file Build/MdeModule/DEBUG_GCC5/X64/UsbAudio.debug
>>>>>>> 0x0007E4B8000` and answer yes when it prompts me to do so.
>>>>>>> (0x0007E4B8000 is the image base, the entry point is at
>>>>>>> 0x0007E4B9440.)
>>>>>>> - When I try to print the Index symbol, GDB tells me that it isn't in
>>>>>>> the current context.
>>>>>>> I feel like I'm missing something. I'm also not the best with GDB
>>>>>>> myself.
>>>>>>> :)
>>>>>> 
>>>>>> What do you get from the following gdb commands?
>>>>>> bt
>>>>>> info local
>>>>>> info symbol 0x0007E4B9440
>>>>>> 
>>>>>> What exactly is gdb showing you?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Andrew Fish
>>>>>> 
>>>>>>> 
>>>>>>> On 6/11/21, Andrew Fish <afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>
>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>>> wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jun 11, 2021, at 11:39 AM, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>
>>>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>
>>>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Andrew,
>>>>>>>>> How do you debug the EFI binary with LLDB? Can LLDB use GDB stubs
>>>>>>>>> or
>>>>>>>>> does that work differently?
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Ethin,
>>>>>>>> 
>>>>>>>> Lldb is the command line debugger that comes with Xcode on Mac.
>>>>>>>> There
>>>>>>>> is
>>>>>>>> no
>>>>>>>> gdb with Xcode, so I have to use lldb for my day job.
>>>>>>>> 
>>>>>>>> Lldb can speak the gdb remote serial protocol: lldb -o “gdb-remote
>>>>>>>> 9000”
>>>>>>>> That assumes you passed `-gdb tcp::9000`to QEMU.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Andrew Fish
>>>>>>>> 
>>>>>>>>> On 6/11/21, Andrew Fish <afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>
>>>>>>>>> <mailto:afish@apple.com <mailto:afish@apple.com> <mailto:afish@apple.com <mailto:afish@apple.com>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Jun 11, 2021, at 10:06 AM, Ethin Probst
>>>>>>>>>>> <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>
>>>>>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>>
>>>>>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>
>>>>>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>
>>>>>>>>>>> <mailto:harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hey all,
>>>>>>>>>>> 
>>>>>>>>>>> So Leif and I have discussed this at length but I thought I'd
>>>>>>>>>>> reach
>>>>>>>>>>> out to all of you for more help.
>>>>>>>>>>> 
>>>>>>>>>>> I'm having a lot of trouble debugging my UEFI app. Here's how I
>>>>>>>>>>> do
>>>>>>>>>>> things:
>>>>>>>>>>> 
>>>>>>>>>>> - I load the app using uefi-run
>>>>>>>>>>> (https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>>>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>
>>>>>>>>>>> <https://github.com/Richard-W/uefi-run <https://github.com/Richard-W/uefi-run>>>>) like this (from the
>>>>>>>>>>> main
>>>>>>>>>>> EDK
>>>>>>>>>>> II directory): uefi-run -b Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd
>>>>>>>>>>> Build/OvmfX64/DEBUG_GCC5/X64/Shell.efi -- -M q35 -m 24G -usb
>>>>>>>>>>> -device
>>>>>>>>>>> qemu-xhci -device usb-audio,audiodev=audio -audiodev
>>>>>>>>>>> alsa,id=audio
>>>>>>>>>>> -s
>>>>>>>>>>> -debugcon file:../debug.log -global isa-debugcon.iobase=0x402
>>>>>>>>>>> -nographic
>>>>>>>>>>> Or:
>>>>>>>>>>> uefi-run -b Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd
>>>>>>>>>>> Build/OvmfX64/DEBUG_GCC5/X64/Shell.efi -- -M q35 -m 24G -usb
>>>>>>>>>>> -device
>>>>>>>>>>> qemu-xhci -device usb-audio,audiodev=audio -audiodev
>>>>>>>>>>> alsa,id=audio
>>>>>>>>>>> -s
>>>>>>>>>>> -debugcon stdio -global isa-debugcon.iobase=0x402
>>>>>>>>>>> - I connect to the remote GDB stub (localhost:1234) and wait
>>>>>>>>>>> until
>>>>>>>>>>> OVMF gives me the image base. Then I use:
>>>>>>>>>>> add-symbol-file UsbAudio.debug <image base>
>>>>>>>>>>> Here's where everything breaks down. One of two things happens at
>>>>>>>>>>> this
>>>>>>>>>>> point:
>>>>>>>>>>> 1. Either I get the wrong debug information (I get source code
>>>>>>>>>>> but
>>>>>>>>>>> the
>>>>>>>>>>> image isn't loaded anymore), and resetting the system and placing
>>>>>>>>>>> a
>>>>>>>>>>> breakpoint (either software or hardware) has no effect; or
>>>>>>>>>>> 2. If I use CpuBreakpoint(), the firmware gives me the registers
>>>>>>>>>>> and
>>>>>>>>>>> the image base and entry point addresses, and then appears to
>>>>>>>>>>> just
>>>>>>>>>>> sit
>>>>>>>>>>> there waiting for something. Once I load the symbols using the
>>>>>>>>>>> image
>>>>>>>>>>> base it gives me, I can't actually do anything in the debugger; I
>>>>>>>>>>> can't list code because I get "1 in <artificial>", I can't jump
>>>>>>>>>>> into
>>>>>>>>>>> my code without triggering a general protection exception or not
>>>>>>>>>>> actually causing anything to happen... You get the idea.
>>>>>>>>>>> 
>>>>>>>>>>> So I'm really, really confused on what's going wrong. Do you guys
>>>>>>>>>>> have
>>>>>>>>>>> any advice?
>>>>>>>>>> 
>>>>>>>>>> Ethin,
>>>>>>>>>> 
>>>>>>>>>> Caveat emptor as I use lldb for my daily driver debugger so I
>>>>>>>>>> might
>>>>>>>>>> be
>>>>>>>>>> a
>>>>>>>>>> little off on gdb specifics…. Also my terminology may be lldb
>>>>>>>>>> centric.
>>>>>>>>>> 
>>>>>>>>>> Easy one 1st. When you run on top of a debugger using
>>>>>>>>>> CpuBreakpoint()
>>>>>>>>>> works
>>>>>>>>>> great as the debugger hides its self from you. On x86
>>>>>>>>>> CpuBreakpoint()
>>>>>>>>>> is
>>>>>>>>>> an
>>>>>>>>>> INT 3h instruction (0xCC) and it causes an exception 3. If you
>>>>>>>>>> don’t
>>>>>>>>>> have
>>>>>>>>>> a
>>>>>>>>>> debugger hooked in underneath  the exception 3 is going to get
>>>>>>>>>> handled
>>>>>>>>>> in
>>>>>>>>>> the unexpected exception handler, and that is probably in the CPUD
>>>>>>>>>> DXE
>>>>>>>>>> driver or DXE Core or some such. So you are going to end up with
>>>>>>>>>> the
>>>>>>>>>> PC/IP/RIP in the wrong driver. A lot of times for hardware
>>>>>>>>>> debuggers
>>>>>>>>>> it
>>>>>>>>>> works better to use CpuDeadLoop(). The gdb-remote stub from QEMU
>>>>>>>>>> acts
>>>>>>>>>> a
>>>>>>>>>> lot
>>>>>>>>>> more like a JTAG hardware debugger than a pure software debugger.
>>>>>>>>>> Also
>>>>>>>>>> note
>>>>>>>>>> that CpuDeadLoop() is an infinite loop, so you can modify the loop
>>>>>>>>>> variable
>>>>>>>>>> with the debugger to continue.
>>>>>>>>>> 
>>>>>>>>>> I’d suggest a work flow of run your App/Driver, hit the
>>>>>>>>>> CpuDeadLoop(),
>>>>>>>>>> attach gdb. Now after you have the target established load the
>>>>>>>>>> symbols.
>>>>>>>>>> The
>>>>>>>>>> reason for me suggesting this flow is the debugger has a flexible
>>>>>>>>>> concept
>>>>>>>>>> of
>>>>>>>>>> what the target is. If you load symbols that will create a target
>>>>>>>>>> for
>>>>>>>>>> a
>>>>>>>>>> stock x86-64 image. When you connect to the QEMU gdb-remote there
>>>>>>>>>> is
>>>>>>>>>> a
>>>>>>>>>> handshake that describes the target and what registers are
>>>>>>>>>> available.
>>>>>>>>>> I
>>>>>>>>>> seem
>>>>>>>>>> to remember QEMU exports some of the system registers, like the
>>>>>>>>>> control
>>>>>>>>>> registers, so it is an extended version of the x86-64 target. So
>>>>>>>>>> this
>>>>>>>>>> changing the target definition might confuse the debugger. To be
>>>>>>>>>> safe
>>>>>>>>>> I
>>>>>>>>>> always connect 1st and then load symbols.
>>>>>>>>>> 
>>>>>>>>>> The EFI images are PE/COFF relocatable executables that are linked
>>>>>>>>>> around
>>>>>>>>>> zero. They get loaded into memory and relocated, so that is why
>>>>>>>>>> you
>>>>>>>>>> need
>>>>>>>>>> to
>>>>>>>>>> specify the load address to get the symbols to resolve. One trick
>>>>>>>>>> I
>>>>>>>>>> use
>>>>>>>>>> is
>>>>>>>>>> to load the ELF (or PE/COFF) build output directly into the
>>>>>>>>>> debugger.
>>>>>>>>>> This
>>>>>>>>>> lets you poke around the image at the linked address. You can
>>>>>>>>>> disassemble
>>>>>>>>>> the functions to see what they look like, obviously you can read
>>>>>>>>>> any
>>>>>>>>>> variables. This can be useful if you get the unhandled exception
>>>>>>>>>> and
>>>>>>>>>> it
>>>>>>>>>> prints out the load address and offset (you can use the offset
>>>>>>>>>> directly).
>>>>>>>>>> It
>>>>>>>>>> is also a good way to debug why your symbols are not quite loaded
>>>>>>>>>> at
>>>>>>>>>> the
>>>>>>>>>> correct address, as you can see what bytes/instructions should be
>>>>>>>>>> at
>>>>>>>>>> a
>>>>>>>>>> given
>>>>>>>>>> address.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Andrew Fish
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Signed,
>>>>>>>>>>> Ethin D. Probst
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Signed,
>>>>>>>>> Ethin D. Probst
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Signed,
>>>>>>> Ethin D. Probst
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Signed,
>>>>> Ethin D. Probst
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Signed,
>>> Ethin D. Probst
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> -- 
> Signed,
> Ethin D. Probst
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 34090 bytes --]