[staging/branch]: CdePkg - some more details

["Kilian Kegel" <KILIAN_KEGEL@OUTLOOK.COM>]

[-- Attachment #1.1: Type: text/plain, Size: 10040 bytes --]

Resend to <devel@edk2.groups.io>

Hi Marvin, hi UEFI community,

sorry for being late with my reply,
> As far as I understood it, it aims to provide an actual full C standard library implementation
Yes, full with known limitations regarding file access, console access and <locale.h> related requirements
(e.g. all f...() functions or system(“echo Hello world”) won’t be run in POST)
Here is a list of 110 ANSI C functions that already run in PEI, DXE using CdePkg and UEFI Shell using Torito C Library:
https://github.com/tianocore/edk2-staging/blob/CdePkg/implemented.md#validation-status


> Is this planned to land in edk2
This would be my proposal for future UEFI based firmware BIOS products like modernFW.
As UEFI implements a filesystem “FAT32”,
I propose to implement additionally a “C” interface to modernFW to:

  *   make it trustable for security, finance, military and government applications/platforms/products
  *   demystifying UEFI BIOS firmware by providing a “known” API

that shall be extended to secure/bounds checking interface of C11 e.g. snprintf_s()

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf Annex K

  *   improve maintainability by using “standard” functions known to the compiler and to static code analysis tools

(that warns at  parameter mismatch e.g. on sprintf() at buildtime)

  *   allow validation by C validation suites CVS from third party vendors
  *   make it usable and understandable for “normal” IT professionals and application developers,

writing shell apps and OPTION ROMs; developers are forced to understand a lot of the EDK buildprocess

and EDK libraries before they can write a simple flash tool… (at companies like Broadcom, Realtek, LSI, Emulex,

Infineon, PC manufacturers…)

> and if so, in what ways will one be encouraged to use it?
EDK-STAGING is the place on tianocore.org where new features that are not ready for product
integration can be checked in for evaluation by the EDK II community prior to
adding to the edk2 trunk…
https://github.com/tianocore/edk2-staging

> Usually it's considered best practice to keep the code quantity in low levels to a minimum,
this might be true for small microcontrollers.
On x86 architecture UEFI runs on 64 Bit processors that start in 32 Bit mode and use CAR Cash As RAM to get PEI
in C running before memory detection. Usually those platforms  have a multi megabyte cache available for CAR.
For those machines ANSI C shall be considered as the lowest common denominator required for string processing <string.h> / <wchar.h>,
character handling <ctype.h> / <wctype.h>, time and date serviceing <time.h>, string to number conversion and vice versa <stdio.h>,
memory allocation,  global jumps and exit() services <stdlib.h>…

Most of that functionality is already provided in UEFI in ANSI and UNICODE representation each as a
proprietary interface and  lacks compatibility with ANSI C.

> and I do not think even most kernel spaces implement a *full* C standard library for this reason.
If you consider BIOS flash space requirements, it is delightfully small, much smaller than you expect, and
probably smaller than the current implementation.
Because it is consistently divided into wrapper library and worker driver.
The “big” worker code is present once per PEI/DXE/SMM phase only.
The code from the wrapper library is linked into the drivers binary only on demand.
This is the “traditional” library concept.

The full functionality (CdeServices) needed for malloc()/free()/realloc(), entire  printf()-family (narrow and wide),
entire  scanf()-family (narrow and wide), most  string processing functions (narrow and wide) needs
much less than 20KB (DXE) / 15KB (PEI) and could also serve as a C-Library-driver for UEFI Shell programs (DXE driver).
https://github.com/KilianKegel/CdeBinPkg/blob/master/CdeServices/CdeServicesDxe64.efi
https://github.com/KilianKegel/CdeBinPkg/blob/master/CdeServices/CdeServicesPei32.efi

Let’s have a look into some ANSI C functions that interact with the CdeServices driver:
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdio_h/Vsscanf.c#L50
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdio_h/Printf.c#L34
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdlib_h/Realloc.c#L21
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdlib_h/strtol.c#L58
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/string_h/StrTok.c#L25
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/time_h/clock.c#L37

On function entrance they all fetch the application interface, that contains the pointer to
the “CdeServices” protocol:

CDE_APP_IF *pCdeAppIf = _GetCdeAppIf();

That, itself is allocated once during drivers runthrough driverentrypoint/CRT0():
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/OSInterface/OS_DXE/osifUefiDxeEntryPoint.c#L125
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/OSInterface/OS_PEI/osifUefiPeiEntryPoint.c#L97

A lot of functions defined in the ANSI C library don’t need any space optimizing, because they are that small by nature:
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/ctype_h/islower.c#L38
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/ctype_h/isalpha.c#L39
https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/string_h/StrLen.c#L16

That is true for DXE/(SMM) and PEI.

>and is there a chance the rest of edk2 is (very slowly) transfered to a standard C
That´d be my idea, but it depends on the acceptance.
I will provide a demonstration on how to convert an existing MdePkg UEFI driver to
CdePkg extention soon.

> Or will it "just" be a handy set of libraries for porting purposes?
I am talking about one single C library CdeLib (as each normal  C library or LIBC is always
provided as one single library file).
CdePkg does not need multiple libraries. There is one driver and one library per POST phase
only (the command line driver is a temporary addition).
All  built out of the same source code.

EDK2 is only the firmware fundament for final PC products (server, desktop, notebook, industry PC, Tablet)
e.g. https://www.fujitsu.com/global/products/computing/peripheral/mainboards/
The final products needs lots of improvements and extensions to withstand the
market requirements and the competitors offerings and to match the customer demands.

PC products for business applications implements many additional  features e.g.

  *   LAN wakeup for different NICs Marvell, BroadCom, Realtek, Intel…
  *   keyboard power button wake out of S5
  *   LAN Boot
  *   USB Port Protection/Disable for security purpose
  *   authentification via CardReader in POST
  *   power fail recovery including setup wake resources after power fail
  *   console redirection for AMT or DASH
  *   hard disk security around security freeze lock
  *   boot device selection by boot menu / hot key
  *   HDD SMART
  *   HSTI Hardware Security Test Interface
  *   TPM
  *   OA OEM activation
  *   (Server runtime) error logging
  *   (remote) BIOS configuration / setup settings,
  *   graphical setup
  *   Server RAS features
  *   IPMI support
  *   NVRAM backup
  *   LVDS display support for industrial applications
  *   brightness slider support
  *   password protection for boot/ HDD/ setup,
  *   fan control
  *   industry requirements:

forced preference of “signed” USB flash/CDROM/HDD drives in

the BIOS boot order for maintenance and service

  *   runtime counter
  *   exchangeable customer logo
(this is a selection of features I worked on the past 10 years doing UEFI BIOS projects).

> Especially in latter case, this, to me intuitively, sounds like a maintenance and reviewing nightmare
Absolutely not, because it is easy to compare each function´s behavior against its corresponding
Microsoft LIBCMT.LIB pendant, once you have started the CdeValidationPkg.sln Visual Studio solution:
[cid:image004.png@01D5AEB1.24B7B000]

Each sample provided in the CdeValidationPkg can be translated for and run on Windows Console, UEFI Shell, DXE and PEI
(except the SystemInterfaceDxe/PEI projects)

All implemented ANSI C functions are already tested and validated comprehensively that way.

Therefore I am pretty sure, you´ll have to try hard to find one single bug beside
https://github.com/KilianKegel/torito-C-Library#known-bugs
if any…

To get the validation modules  running in POST you have to use the traditional EDK2 build process:

  1.  clone the edk2-staging repository
  2.  checkout CdePkg
  3.  run LAUNCH.BAT
  4.  run build -p EmulatorPkg\EmulatorPkg.dsc -t VS2015x86 -a IA32
  5.  run DBGEMU.BAT to start emulation (EmulatorPkg)
  6.  run build -a IA32 -a X64 -n 5 -t VS2015x86 -b DEBUG -p Vlv2TbltDevicePkg\PlatformPkgX64.dsc
  7.  update MinnowBoard with Build/Vlv2TbltDevicePkgX64\DEBUG_VS2015x86\FV\VLV.fd

>curious about its purpose and future
with a “C” interface provided as CdeServices it would:

  1.  allow Shell apps / DXE/SMM/PEI driver to share the same sourcecode (beside API specific parts)
  2.  Share or reuse source code with open source
  3.  allow use of automatically generated source code by syntactical/lexical analysis tools (lex/yacc)
  4.  ease programming tasks that deal with text processing (e.g. device path, setup strings), time and date handling…

because ANSI C is standardized

  1.  allow prototyping as a UEFI Shell application or as a Windows Console application to  be debugged with

superb Windows debug tools

  1.  dispense the need of reading the source code to get an idea about exact behavior of a particular function as

(https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BasePrintLib/PrintLib.c#L26

Thanks for your curiosity and best regards,
Kilian






[-- Attachment #1.2: Type: text/html, Size: 37356 bytes --]

[-- Attachment #2: D2F0AB7C08694605BD466B5B6A17CB46.png --]
[-- Type: image/png, Size: 153130 bytes --]

[-- Attachment #3: F38E7D314A8740749CDED2CA2BB88463.png --]
[-- Type: image/png, Size: 135 bytes --]