Re: [staging/branch]: CdePkg - some more details

["Marvin Häuser" <mhaeuser@outlook.de>]

Good day Kilian,

Comments are inline.
Thank you very much for your very comprehensive explainations!

Please do not understand my sceptical tone as if I didn't want this 
project to succeed, my concerns are a lot more organisation-centered 
than on the work on the library itself.

I wish you the best of luck with your project!

Best regards,
Marvin

Am 10.12.2019 um 08:23 schrieb Kilian Kegel:
> Resend to <devel@edk2.groups.io>
> 
> Hi Marvin, hi UEFI community,
> 
> sorry for being late with my reply,
> 
>> As far as I understood it, it aims to provide an actual full C standard library implementation
> 
> Yes, full with known limitations regarding file access, console access 
> and <locale.h> related requirements
> 
> (e.g. all *f...()* functions or *system(“echo Hello world”) *won’t be 
> run in POST)

Got it, thanks!

> 
> Here is a list of 110 ANSI C functions that already run in PEI, DXE 
> using CdePkg and UEFI Shell using Torito C Library:
> 
> https://github.com/tianocore/edk2-staging/blob/CdePkg/implemented.md#validation-status
> 
>> Is this planned to land in edk2
> 
> This would be my proposal for future UEFI based firmware BIOS products 
> like modernFW.
> 
> As UEFI implements a filesystem “FAT32”,
> 
> I propose to implement additionally a “C” interface to modernFW to:
> 
>   * make it trustable for security, finance, military and government
>     applications/platforms/products

I do not get your point here - a standard compliant interface itself is 
supposed to make it more trustable? I think I misunderstand you here.

>   * demystifying UEFI BIOS firmware by providing a “known” API >
> that shall be extended to secure/bounds checking interface of C11 e.g. 
> snprintf*_s*()
> 
> http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf Annex K
> 
>   * improve maintainability by using “standard” functions known to the
>     compiler and to static code analysis tools
> 
> (that warns at  parameter mismatch e.g. on sprintf() at buildtime)
> 
>   * allow validation by C validation suites CVS from third party vendors
>   * make it usable and understandable for “normal” IT professionals and
>     application developers, 

The points about existing validation suites, static analysis support and 
such are all good.

> 
> writing shell apps and OPTION ROMs; developers are forced to understand 
> a lot of the EDK buildprocess
> 
> and EDK libraries before they can write a simple flash tool… (at 
> companies like Broadcom, Realtek, LSI, Emulex,
> 
> Infineon, PC manufacturers…)
> 
>> and if so, in what ways will one be encouraged to use it?
> 
> EDK-STAGING is the place on tianocore.org where new features that are 
> not ready for product
> 
> integration can be checked in for evaluation by the EDK II community 
> prior to
> 
> adding to the edk2 trunk…
> 
> https://github.com/tianocore/edk2-staging

Thank you, but I am aware of this. I am more interested in the future 
mainline usage, i.e. the final goal of this initiative.

> 
>> Usually it's considered best practice to keep the code quantity in low levels to a minimum,
> 
> this might be true for small microcontrollers.
> 
> On x86 architecture UEFI runs on 64 Bit processors that start in 32 Bit 
> mode and use CAR Cash As RAM to get PEI
> 
> in C running before memory detection. Usually those platforms  have a 
> multi megabyte cache available for CAR.

This point was not about code size or resource consumption, but that is 
also a concern. When making a point about such, you cannot take the best 
case (x86-based products which are optimised for performance and 
support) but must take the worst case (e.g. I have seen edk2 has been 
ported to Raspberry devices and other embedded systems), where the 
resources are far more limited. For CDE itself, I'm not worrying about 
the resource consumption though, at most so when combining it with edk2.

The actual point is about validation. We should not strive for a 
feature-rich framework to get everyone happy with the interface, but for 
a rock-solid fundament that is easy to verify and validate. 110 ANSI C 
functions which overlap with edk2 functions mean 220 functions to 
validate and maintain, and this, in my opinion, is disastrous. I do not 
know about how well C standard functions work out in firmware 
implementations, but if they work out as well as edk2-specific stuff, 
I'd strongly suggest and favour deprecating edk2 concepts in favour of 
ANSI C step by step.

I have also seen efforts to allow Rust support for edk2. *Please* engage 
with the different people proposing different modernisations for the 
project to not end up with a suboptimal middle ground, a lot of scrapped 
work, or, at worst, many alternatives. Supporting traditional edk2, CDE 
*and* Rust at once sounds absolutely horrible to me. Currently, it 
sounds a bit chaotic to me where edk2 is heading.

This does make me wonder, why was edk2 designed with this set of 
proprietary functions and types, manifested in even the specification, 
in the first place? Is there a practical point regarding ANSI C design 
or just the compiler situation when the original EDK was written? If 
former, that should surely be considered in the CDE design.

> 
> For those machines ANSI C shall be considered as the lowest common 
> denominatorrequired for string processing <string.h> / <wchar.h>,
> 
> character handling <ctype.h> / <wctype.h>, time and date serviceing 
> <time.h>, string to number conversion and vice versa <stdio.h>,
> 
> memory allocation,  global jumps and exit() services <stdlib.h>…
> 
> Most of that functionality is already provided in UEFI in ANSI and 
> UNICODE representation each as a
> 
> proprietary interface and  lacks compatibility with ANSI C.

I'm all for moving to ANSI C for all *existing* concepts. A full C 
standard library consists of a lot of concepts that we probably do not 
want in firmware, including but not limited to advanced (or any really) 
floating point support, math operations, signals, advanced 
multithreading, advanced file and console I/O, advanced environment APIs 
(e.g. locale, time). I know a bunch of those are optional and were 
probably not considered in the first place, but I wanted to stress to 
keep it as simple as possible - and if that means to deviate or just 
ignore parts of the standard (library), I think that should be done.

But just for the basic stuff, I would really appreciate ANSI C support. 
This is for basic APIs (malloc, str*, etc) and especially types. Integer 
Promotion is an essential aspect of C, I don't know how one can just 
banish that type from their vocabulary like that. This can make things 
like the print functions a bit inconvenient. I do not see how UEFI's 
types make any sense over standard fixed-width types either.
Did any of the edk2 designers (or Stewards? Sorry, I don't understand 
who is in charge of what very well) comment on this kind of stuff before?

> 
>> and I do not think even most kernel spaces implement a *full* C standard library for this reason.
> 
> If you consider BIOS flash space requirements, it is delightfully small, 
> much smaller than you expect, and
> 
> probably smaller than the current implementation.
> 
> Because it is consistently divided into wrapper library and worker driver.
> 
> The “big” worker code is present once per PEI/DXE/SMM phase only.
> 
> The code from the wrapper library is linked into the drivers binary only 
> on demand.
> 
> This is the “traditional” library concept.
> 
> The full functionality (CdeServices) needed for 
> malloc()/free()/realloc(), entire  printf()-family (narrow and wide),
> 
> entire  scanf()-family (narrow and wide), most  string processing 
> functions (narrow and wide) needs
> 
> much less than 20KB (DXE) / 15KB (PEI) and could also serve as a 
> C-Library-driver for UEFI Shell programs (DXE driver).
> 
> https://github.com/KilianKegel/CdeBinPkg/blob/master/CdeServices/CdeServicesDxe64.efi
> 
> https://github.com/KilianKegel/CdeBinPkg/blob/master/CdeServices/CdeServicesPei32.efi
> 
> Let’s have a look into some ANSI C functions that interact with the 
> *CdeServices* driver:
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdio_h/Vsscanf.c#L50
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdio_h/Printf.c#L34
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdlib_h/Realloc.c#L21
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/stdlib_h/strtol.c#L58
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/string_h/StrTok.c#L25
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/time_h/clock.c#L37
> 
> On function entrance they all fetch the application interface, that 
> contains the pointer to
> 
> the “*CdeServices*” protocol:
> 
> **
> 
> *CDE_APP_IF *pCdeAppIf = _GetCdeAppIf();*
> 
> **
> 
> That, itself is allocated once during drivers runthrough 
> driverentrypoint/CRT0():
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/OSInterface/OS_DXE/osifUefiDxeEntryPoint.c#L125
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/OSInterface/OS_PEI/osifUefiPeiEntryPoint.c#L97
> 
> A lot of functions defined in the ANSI C library don’t need any space 
> optimizing, because they are that small by nature:
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/ctype_h/islower.c#L38
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/ctype_h/isalpha.c#L39
> 
> https://github.com/KilianKegel/CdeBinPkgSrc/blob/master/CdeLib/Library/string_h/StrLen.c#L16
> 
> That is true for DXE/(SMM) and PEI.
> 
>>and is there a chance the rest of edk2 is (very slowly) transfered to a standard C
> 
> That´d be my idea, but it depends on the acceptance.

That was my biggest concern, really. I think this project can do a lot 
of damage if it is not fully accepted but only provided as a mere 
alternative by choice. It makes review (quantity) and understanding the 
control flow (two separate API designs) harder, overcomplicates the 
codebase (e.g. folder layout) and may lead to multiple functions per 
purpose (MDE *and* CDE) per module, which will degrade performance 
(binary size -> code cache, load time, etc).

But I also think it can do a lot of good if it is accepted and executed 
well, so I hope this gains some actual traction. So far I only have seen 
you posting about it, and that worries me. Are there disucussions going 
on in the background?

> 
> I will provide a demonstration on how to convert an existing MdePkg UEFI 
> driver to
> 
> CdePkg extention soon.
> 
>> Or will it "just" be a handy set of libraries for porting purposes?
> 
> I am talking about one single C library CdeLib (as each normal  C 
> library or LIBC is always
> 
> provided as one single library file).
> 
> CdePkg does not need multiple libraries. There is one driver and one 
> library per POST phase

Sorry, this was unclear, I did not mean it in a edk2 module sense, but 
the separation into groups (stdio, stddef, stdlib, ...). The actual 
point was the "just", as in an alternative, as discussed above.

> 
> only (the command line driver is a temporary addition).
> 
> All  built out of the same source code.
> 
> EDK2 is only the firmware fundament for final PC products (server, 
> desktop, notebook, industry PC, Tablet)
> 
> e.g. 
> https://www.fujitsu.com/global/products/computing/peripheral/mainboards/
> 
> The final products needs lots of improvements and extensions to 
> withstand the
> 
> market requirements and the competitors offerings and to match the 
> customer demands.
> 
> PC products for business applications implements many additional
> features e.g.
> 
>   * LAN wakeup for different NICs Marvell, BroadCom, Realtek, Intel…
>   * keyboard power button wake out of S5
>   * LAN Boot
>   * USB Port Protection/Disable for security purpose
>   * authentification via CardReader in POST
>   * power fail recovery including setup wake resources after power fail
>   * console redirection for AMT or DASH
>   * hard disk security around security freeze lock
>   * boot device selection by boot menu / hot key
>   * HDD SMART
>   * HSTI Hardware Security Test Interface
>   * TPM
>   * OA OEM activation
>   * (Server runtime) error logging
>   * (remote) BIOS configuration / setup settings,
>   * graphical setup
>   * Server RAS features
>   * IPMI support
>   * NVRAM backup
>   * LVDS display support for industrial applications
>   * brightness slider support
>   * password protection for boot/ HDD/ setup,
>   * fan control
>   * industry requirements: 
> 
> forced preference of “signed” USB flash/CDROM/HDD drives in
> 
> the BIOS boot order for maintenance and service
> 
>   * runtime counter
>   * exchangeable customer logo
> 
> (this is a selection of features I worked on the past 10 years doing 
> UEFI BIOS projects).
> 
>> Especially in latter case, this, to me intuitively, sounds like a maintenance and reviewing nightmare
> 
> Absolutely not, because it is easy to compare each function´s behavior 
> against its corresponding
> 
> Microsoft LIBCMT.LIB pendant, once you have started the 
> */CdeValidationPkg.sln /*Visual Studio solution:
> 
> cid:image004.png@01D5AEB1.24B7B000
> 
> Each sample provided in the CdeValidationPkg can be translated for and 
> run on Windows Console, UEFI Shell, DXE and PEI
> 
> (except the SystemInterfaceDxe/PEI projects)
> 
> All implemented ANSI C functions are already tested and validated 
> comprehensively that way.

Well, yes, but this seems to be basically unit testing, I meant in a 
sense of formal review. The problem is not the characteristics of your 
project, but the fact that it duplicates existing concepts. If you 
believe in the advantages of this approach, I hope you aim to deprecate 
existing edk2 pendants step by step.

> 
> Therefore I am pretty sure, you´ll have to try hard to find one single 
> bug beside
> 
> https://github.com/KilianKegel/torito-C-Library#known-bugs
> 
> if any…
> 
> To get the validation modules  running in POST you have to use the 
> traditional EDK2 build process:
> 
>  1. clone the *edk2-staging* repository
>  2. checkout *CdePkg*
>  3. run *LAUNCH.BAT*
>  4. run |*build -p EmulatorPkg\EmulatorPkg.dsc -t VS2015x86 -a IA32*|
>  5. run|*DBGEMU.BAT *||to start emulation||*(EmulatorPkg)*|
>  6. run|*build -a IA32 -a X64 -n 5 -t VS2015x86 -b DEBUG -p
>     Vlv2TbltDevicePkg\PlatformPkgX64.dsc*|
>  7. |update MinnowBoard
>     with||*Build/Vlv2TbltDevicePkgX64\DEBUG_VS2015x86\FV\VLV.fd*|
> 
>>curious about its purpose and future
> 
> with a “C” interface provided as CdeServices it would:
> 
>  1. allow Shell apps / DXE/SMM/PEI driver to share the same sourcecode
>     (beside API specific parts)
>  2. Share or reuse source code with open source
>  3. allow use of automatically generated source code by
>     syntactical/lexical analysis tools (lex/yacc)
>  4. ease programming tasks that deal with text processing (e.g. device
>     path, setup strings), time and date handling…
> 
> because ANSI C is standardized
> 
>  5. allow prototyping as a UEFI Shell application or as a Windows
>     Console application to  be debugged with
> 
> superb Windows debug tools
> 
>  6. dispense the need of reading the source code to get an idea about
>     exact behavior of a particular function as
> 
> (https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BasePrintLib/PrintLib.c#L26
> 
> Thanks for your curiosity and best regards,
> 
> Kilian
> 
> 
>