EDK II Network Stack Issue

EDK II Network Stack Issue

[Karin Willers]

G'Day!

I'm trying to get networking under edk2 up and running. I tried 
AppPkg/Applications/Sockets/RawIp4Tx
under OVMF. The raw packet is sent out on the network, but the 
application never returns from the
socket close routine.

I'm currently using UDK2017 with the latest security patches (downloaded 
December 18 2018). The network
driver under OVMF is the e1000 driver.

The effect that the socket close never returns is also visible when 
running RawIp4Tx on real hardware,
so I think the behavior has nothing to do with OVMF or the UEFI itself.

Does anyone see similar effects? Any hints on setting up networking 
under edk2 correctly?

Thanks,  Karin

Re: EDK II Network Stack Issue

[Laszlo Ersek]

On 01/04/19 15:02, Karin Willers wrote:
> G'Day!
> 
> I'm trying to get networking under edk2 up and running. I tried
> AppPkg/Applications/Sockets/RawIp4Tx
> under OVMF. The raw packet is sent out on the network, but the
> application never returns from the
> socket close routine.
> 
> I'm currently using UDK2017 with the latest security patches (downloaded
> December 18 2018). The network
> driver under OVMF is the e1000 driver.
> 
> The effect that the socket close never returns is also visible when
> running RawIp4Tx on real hardware,
> so I think the behavior has nothing to do with OVMF or the UEFI itself.
> 
> Does anyone see similar effects? Any hints on setting up networking
> under edk2 correctly?

The socket (= libc-level networking) APIs are not the most robust ones
in edk2, in my -- admittedly limited -- experience. I'd suggest using
applications and shell commands that use UEFI protocols instead, for
networking. (I understand that could be a challenge if you are porting a
standard C program to UEFI.)

Thanks
Laszlo

Re: EDK II Network Stack Issue

[Karin Willers]

On 2019-01-07 20:27, Laszlo Ersek wrote:
> On 01/04/19 15:02, Karin Willers wrote:
>> G'Day!
>> 
>> I'm trying to get networking under edk2 up and running. I tried
>> AppPkg/Applications/Sockets/RawIp4Tx
>> under OVMF. The raw packet is sent out on the network, but the
>> application never returns from the
>> socket close routine.
>> 
>> I'm currently using UDK2017 with the latest security patches 
>> (downloaded
>> December 18 2018). The network
>> driver under OVMF is the e1000 driver.
>> 
>> The effect that the socket close never returns is also visible when
>> running RawIp4Tx on real hardware,
>> so I think the behavior has nothing to do with OVMF or the UEFI 
>> itself.
>> 
>> Does anyone see similar effects? Any hints on setting up networking
>> under edk2 correctly?
> 
> The socket (= libc-level networking) APIs are not the most robust ones
> in edk2, in my -- admittedly limited -- experience. I'd suggest using
> applications and shell commands that use UEFI protocols instead, for
> networking. (I understand that could be a challenge if you are porting 
> a
> standard C program to UEFI.)
> 
> Thanks
> Laszlo

The reason for this undertaking initially was to compile a Python.efi 
that
supports sockets. We wanted that to be able to run the Chipsec suite 
with
external network test partners under UEFI. The Chipsec guys do provide
pre-compiled Pyhon executables, but these do not include the network 
stack.

I think, I have to debug the issue myself ...

Greetings,  Karin

Re: EDK II Network Stack Issue

[Karin Willers]

On 2019-01-10 13:40, Karin Willers wrote:
> On 2019-01-07 20:27, Laszlo Ersek wrote:
>> On 01/04/19 15:02, Karin Willers wrote:
>>> G'Day!
>>> 
>>> I'm trying to get networking under edk2 up and running. I tried
>>> AppPkg/Applications/Sockets/RawIp4Tx
>>> under OVMF. The raw packet is sent out on the network, but the
>>> application never returns from the
>>> socket close routine.
>>> 
>>> I'm currently using UDK2017 with the latest security patches 
>>> (downloaded
>>> December 18 2018). The network
>>> driver under OVMF is the e1000 driver.
>>> 
>>> The effect that the socket close never returns is also visible when
>>> running RawIp4Tx on real hardware,
>>> so I think the behavior has nothing to do with OVMF or the UEFI 
>>> itself.
>>> 
>>> Does anyone see similar effects? Any hints on setting up networking
>>> under edk2 correctly?
>> 
>> The socket (= libc-level networking) APIs are not the most robust ones
>> in edk2, in my -- admittedly limited -- experience. I'd suggest using
>> applications and shell commands that use UEFI protocols instead, for
>> networking. (I understand that could be a challenge if you are porting 
>> a
>> standard C program to UEFI.)
>> 
>> Thanks
>> Laszlo
> 
> The reason for this undertaking initially was to compile a Python.efi 
> that
> supports sockets. We wanted that to be able to run the Chipsec suite 
> with
> external network test partners under UEFI. The Chipsec guys do provide
> pre-compiled Pyhon executables, but these do not include the network 
> stack.
> 
> I think, I have to debug the issue myself ...
> 
> Greetings,  Karin

Could please someone shed some light on how to enable debug prints using 
UefiDebugLibConOut
in EfiSocketLib/Socket.c. Enabling debug prints is, IMHO, not very 
intuitive in edk2 ...

Thanks,  Karin

Re: EDK II Network Stack Issue

[Laszlo Ersek]

On 01/16/19 13:57, Karin Willers wrote:
> On 2019-01-10 13:40, Karin Willers wrote:
>> On 2019-01-07 20:27, Laszlo Ersek wrote:
>>> On 01/04/19 15:02, Karin Willers wrote:
>>>> G'Day!
>>>>
>>>> I'm trying to get networking under edk2 up and running. I tried
>>>> AppPkg/Applications/Sockets/RawIp4Tx
>>>> under OVMF. The raw packet is sent out on the network, but the
>>>> application never returns from the
>>>> socket close routine.
>>>>
>>>> I'm currently using UDK2017 with the latest security patches
>>>> (downloaded
>>>> December 18 2018). The network
>>>> driver under OVMF is the e1000 driver.
>>>>
>>>> The effect that the socket close never returns is also visible when
>>>> running RawIp4Tx on real hardware,
>>>> so I think the behavior has nothing to do with OVMF or the UEFI itself.
>>>>
>>>> Does anyone see similar effects? Any hints on setting up networking
>>>> under edk2 correctly?
>>>
>>> The socket (= libc-level networking) APIs are not the most robust ones
>>> in edk2, in my -- admittedly limited -- experience. I'd suggest using
>>> applications and shell commands that use UEFI protocols instead, for
>>> networking. (I understand that could be a challenge if you are porting a
>>> standard C program to UEFI.)
>>>
>>> Thanks
>>> Laszlo
>>
>> The reason for this undertaking initially was to compile a Python.efi
>> that
>> supports sockets. We wanted that to be able to run the Chipsec suite with
>> external network test partners under UEFI. The Chipsec guys do provide
>> pre-compiled Pyhon executables, but these do not include the network
>> stack.
>>
>> I think, I have to debug the issue myself ...
>>
>> Greetings,  Karin
> 
> Could please someone shed some light on how to enable debug prints using
> UefiDebugLibConOut
> in EfiSocketLib/Socket.c. Enabling debug prints is, IMHO, not very
> intuitive in edk2 ...

Can you build your platform and/or application with the --log and
--report-file options, and share the outputs? The following could all
matter:

- what you pass to "build" with "-b" (NOOPT/DEBUG/RELEASE),
- gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask,
- gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel,
- gEfiMdePkgTokenSpaceGuid.PcdFixedDebugPrintErrorLevel,
- what lib instance you resolve the DebugLib class to,
- whether you apply the above just to your application, or all other DXE
  components in your platform.

Thanks
Laszlo

Re: EDK II Network Stack Issue

[Laszlo Ersek]

Adding the list back in -- I can't really justify spending an hour or
more on emails like this if I don't send them to the list. More
importantly, I could be wrong about some details, and keeping the
discussion on-list gives others a chance to correct me.

Comments below:

On 01/22/19 09:56, Karin Willers wrote:
> On 2019-01-16 17:51, Laszlo Ersek wrote:
>
>>>
>>> Could please someone shed some light on how to enable debug prints
>>> using UefiDebugLibConOut in EfiSocketLib/Socket.c. Enabling debug
>>> prints is, IMHO, not very intuitive in edk2 ...
>>
>> Can you build your platform and/or application with the --log and
>> --report-file options, and share the outputs? The following could all
>> matter:
>
> See attached ZIP-File.
>
>>
>> - what you pass to "build" with "-b" (NOOPT/DEBUG/RELEASE),
>
> DEBUG is selected in target.txt
>
>> - gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask,
>> - gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel,
>> - gEfiMdePkgTokenSpaceGuid.PcdFixedDebugPrintErrorLevel,
>
> In StdLib.dsc I have:
>
> #
> #  Debug output control
> #
>   DEFINE DEBUG_ENABLE_OUTPUT      = TRUE        # Set to TRUE to enable
> debug output
> # DEFINE DEBUG_PRINT_ERROR_LEVEL  = 0x80000000  # Flags to control
> amount of debug output
>   DEFINE DEBUG_PRINT_ERROR_LEVEL  = 0xffffffff  # Flags to control
> amount of debug output
>   DEFINE DEBUG_PROPERTY_MASK      = 0x0f
>
> [PcdsFeatureFlag]
>
> [PcdsFixedAtBuild]
>   gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask|$(DEBUG_PROPERTY_MASK)
>
> gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|$(DEBUG_PRINT_ERROR_LEVEL)
>
> This enables UefiDebugLibConOut.inf (in StdLib.dsc) as follows:
>
>   !if $(DEBUG_ENABLE_OUTPUT)
>     DebugLib|MdePkg/Library/UefiDebugLibConOut/UefiDebugLibConOut.inf
>
> DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
>
>   !else   ## DEBUG_ENABLE_OUTPUT
>     DebugLib|MdePkg/Library/BaseDebugLibNull/BaseDebugLibNull.inf
>   !endif  ## DEBUG_ENABLE_OUTPUT
>
> Just to be sure (don't know whether this is really necessary) in
> MdePkg.dsc I have:
> gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0xffffffff
>
>> - what lib instance you resolve the DebugLib class to,
>
> In 'StdLib.inc' (included from 'StdLib.dsc') I inserted:
>   DebugLib|MdePkg/Library/UefiDebugLibConOut/UefiDebugLibConOut.inf   #
> Debugging Support
>
>> - whether you apply the above just to your application, or all other
>> DXE
>
> I suppose, just my application?!
>
>>   components in your platform.
>
> I compile the parts (which I think I need) individually:
>
> o MdePkg
> o StdLib
> o AppPkg
>
> and then copy everything to a folder for use with OVMF.
>
> The packages were built in this exact order:
>
> build --log MdePkg-log.txt --report-file MdePkg-report-file.txt
> build --log StdLib-log.txt --report-file StdLib-report-file.txt
> build --log AppPkg-log.txt --report-file AppPkg-report-file.txt
>
> When running qemu on the OVMF image (built previously) using the files
> just generated,
> I do not see any debug prints. My qemu sequence is as follows:
>
> #!/bin/sh
>
> set -x
>
> qemu-system-x86_64 -L . --bios ./OVMF.fd \
>     -hda fat:rw:hda-contents \
>     -global e1000.romfile="" \
>     -netdev tap,id=hostnet0,ifname=tap0,script=no,downscript=no \
>     -device e1000,netdev=hostnet0,mac=52:54:00:56:34:12
>
>>
>> Thanks
>> Laszlo
>
> The next thing is to try another network driver - the e1000 driver is
> only available as a binary from Intel
> and not in source form.
>
> Thanks in advance for your help!

In "AppPkg-report-file.txt", the "RawIp4Tx" module's section lists the
following library instances (reformatted for readability):

  DebugPrintErrorLevelLib --> MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
  PcdLib                  --> MdePkg/Library/BasePcdLibNull/BasePcdLibNull.inf
  DebugLib                --> MdePkg/Library/UefiDebugLibConOut/UefiDebugLibConOut.inf
  EfiSocketLib            --> StdLib/EfiSocketLib/EfiSocketLib.inf

The report file also confirms that the build target was DEBUG.

This means that DEBUG macro invocations in StdLib/EfiSocketLib code will
be resolved to MdePkg/Library/UefiDebugLibConOut, in the final
"RawIp4Tx.efi" binary.

The PcdLib class resolution means that dynamic PCDs won't work; all the
relevant PCDs have to be patched-in-module, or fixed-at-build. This is
actually good (see details below).

The chosen DebugLib instance depends on PcdDebugPropertyMask and
PcdFixedDebugPrintErrorLevel. The former PCD is relevant here for
DebugPrintEnabled() and DebugCodeEnabled(), the latter is relevant for
DebugPrintLevelEnabled().

In addition, the chosen DebugPrintErrorLevelLib instance consumes
PcdDebugPrintErrorLevel, via GetDebugPrintErrorLevel().


Now, assume that the DEBUG() function-like macro is invoked within a
DEBUG_CODE() section in StdLib/EfiSocketLib. This is what happens, after
the macros are all expanded:

- DebugCodeEnabled() is called first. This checks, in
  UefiDebugLibConOut, whether DEBUG_PROPERTY_DEBUG_CODE_ENABLED (0x04)
  is set in PcdDebugPropertyMask.

- DebugPrintEnabled() is called. This checks, in UefiDebugLibConOut,
  whether DEBUG_PROPERTY_DEBUG_PRINT_ENABLED (0x02) is set in
  PcdDebugPropertyMask.

- DebugPrintLevelEnabled() is called. This checks, in
  UefiDebugLibConOut, whether the specified error level (or more
  precisely, error mask) intersects with PcdFixedDebugPrintErrorLevel.

- DebugPrint() is called. This function is again implemented in
  UefiDebugLibConOut, and it calls GetDebugPrintErrorLevel(). The retval
  is checked against the caller-specified error mask.

- GetDebugPrintErrorLevel() is implemented in
  BaseDebugPrintErrorLevelLib, and it retrieves PcdDebugPrintErrorLevel.

- If all the checks pass, and you have a non-NULL UEFI console
  (gST->ConOut), then DebugPrint() outputs the message.


That is:
- PcdDebugPropertyMask must have bit value 0x04 set.
- PcdDebugPropertyMask must have bit value 0x02 set.
- PcdFixedDebugPrintErrorLevel must intersect with the error mask that
  the given DEBUG macro invocation uses.
- PcdDebugPrintErrorLevel must intersect with the error mask that the
  given DEBUG macro invocation uses.

In the same section of the AppPkg report file (under RawIp4Tx), I see
the following PCD settings:

>     PcdDebugPropertyMask               :  FIXED    (UINT8) = 0
>
>     PcdFixedDebugPrintErrorLevel       :  FIXED   (UINT32) = 0xFFFFFFFF
>
>  *P PcdDebugPrintErrorLevel            :  FIXED   (UINT32) = 0x80000040
>                                                DEC DEFAULT = 0x80000000

The PcdDebugPropertyMask requirement is not satisfied. You have to set
it to the bitmask 0x06 (or a superset of it) in either AppPkg.dsc, or
else on the "build" command line, with the --pcd switch.

The PcdFixedDebugPrintErrorLevel requirement is satisfied (it has all
imaginable bits set).

The PcdDebugPrintErrorLevel requirement is not satisfied. Although
AppPkg.dsc already overrides the default from MdePkg.dec (hence the "*P"
mark, for "platform override"), value 0x80000040 only stands for
DEBUG_ERROR | DEBUG_INFO (see "MdePkg/Include/Library/DebugLib.h").

However, "StdLib/EfiSocketLib" uses non-standard error masks such as
DEBUG_BIND, DEBUG_ACCEPT, and so on (see
"StdLib/EfiSocketLib/Socket.h"). If you want those to appear as well,
then you'll have to include the corresponding bits in
PcdDebugPrintErrorLevel. Use the --pcd switch again, or modify the
AppPkg.dsc file, or use -D DEBUG_PRINT_ERROR_LEVEL=... on the "build"
command line, when building AppPkg (because AppPkg.dsc sets the PCD to
that macro, as a convenience).

A PCD has several aspects. Any given PCD has:

- A set of permissible PCD storage types. This set comes from the
  package DEC (declaration) file, where the PCD is declared. It
  determines the storage types (plural) that any given platform, or
  module, may choose for the PCD. Storage types include: fixed-at-build
  (basically: a build constant), patchable-in-module (a build constant
  capable of being statically updated in firmware images by an external,
  binary patching tool), dynamic (read/write, stored in memory,
  accessible by almost everything in the firmware), HII (stored in NVRAM
  / UEFI variables).

- A data type, such as UINT32. This comes from the DEC file.

- A default value, such as 0 or 1. Comes from the DEC file.

- In the platform DSC file, the platform chooses the actual storage type
  for the PCD (selecting one of the DEC-permitted types), and may
  provide a platform-specific default value (overriding the default from
  the DEC file).

- In the platform DSC file, individual modules too may choose different
  storage types (overriding the DSC default) and different default
  values (ditto).


In your case:

- Compiling MdePkg and StdLib modules in separation, that is, through
  the "MdePkg.dsc" and "StdLib.dsc" platform description files, is
  entirely superfluous. The build output fom those builds will not be
  used for anything. When you compile AppPkg, the build utility will
  collect dependencies from across the tree, and build everything
  necessary under Build/, such that the build artifacts are going to be
  specific to AppPkg, even if they come from under MdePkg or StdLib.
  And, in this build, the AppPkg.dsc settings will apply. The only PCD
  settings that will take effect from MdePkg and StdLib are those where
  AppPkg doesn't override the MdePkg / StdLib DEC defaults. This is why
  I only looked at the AppPkg build report file, from your attachment.

- If you build EFI binaries from AppPkg, and then run them on an OVMF
  platform firmware, then fixed-at-build PCDs may have entirely
  different values in the AppPkg binaries, from the values that the same
  PCDs have in OVMF modules. The same applies to the resolutions of
  library classes to library instances. For example, modules built into
  OVMF will print their debug messages to the QEMU debug port, not to
  the UEFI console.

- Worse, if both sets of modules use dynamic storage for a given PCD,
  then they might disagree on either the precise storage location itself
  (with catastrophic results [I'm ignoring some details here]), or on
  the meaning of the individual integer values. Dynamic PCDs exist for
  cross-driver communication, and they establish a tight coupling
  between modules. Therefore UEFI applications (which are generally
  considered "portable") should generally not use dynamic PCDs at all.
  Fixed PCDs are fine, but then see the previous bullet -- the same PCDs
  may have different assignments in the platform firmware.

This is why I asked earlier, "whether you apply the above just to your
application, or all other DXE" -- the question referred to whether you
applied the fixed PCD settings to all of OVMF (via the OVMF DSC files),
or just your standalone EFI application (built from AppPkg).


Okay, summary:

- forget the MdePkg and StdLib DSC files, build only AppPkg (or whatever
  platform DSC it is that produces your UEFI application)

- make sure PcdDebugPropertyMask has bitmask 0x06 set when you build
  AppPkg (or the appropriate platform DSC in question); either modify
  the same DSC for this, or use the --pcd command line switch

- make sure PcdDebugPrintErrorLevel has all the DEBUG_* bits set from
  "StdLib/EfiSocketLib/Socket.h" that you care about; the methods for
  setting this PCD are identical to those seen in the previous point.

Hope this helps,
Laszlo

Re: EDK II Network Stack Issue

[Karin Willers]

On 1/22/19 6:34 PM, Laszlo Ersek wrote:
> Okay, summary:
>
> - forget the MdePkg and StdLib DSC files, build only AppPkg (or whatever
>    platform DSC it is that produces your UEFI application)
>
> - make sure PcdDebugPropertyMask has bitmask 0x06 set when you build
>    AppPkg (or the appropriate platform DSC in question); either modify
>    the same DSC for this, or use the --pcd command line switch
>
> - make sure PcdDebugPrintErrorLevel has all the DEBUG_* bits set from
>    "StdLib/EfiSocketLib/Socket.h" that you care about; the methods for
>    setting this PCD are identical to those seen in the previous point.
>
> Hope this helps,
> Laszlo
>
Hello Laszlo,

debug prints are working now - thanks for your advice!

Greetings,  Karin