[edk2-devel] [edk2-rfc] [RFC] EDK2 Testing Formalization

["Bret Barkelew" <bret.barkelew@microsoft.com>]

[-- Attachment #1: Type: text/plain, Size: 4529 bytes --]

In light of the parallel discussion happening right now about CI options, I wanted to highlight a conversation that has been ongoing about Testing. There has been a lot of work contributed recently to expand that types and availability of automated tests in the EDK2 tree. This discussion ranges from topics on test technologies and approaches to types/definitions and code maintenance.

A full RFC doc can be found here:
https://github.com/spbrogan/edk2-staging/blob/edk2-host-test/Readme-RFC.md
and the latest community version (though slightly older) can be found here:
https://github.com/tianocore/edk2-staging/blob/edk2-host-test/Readme-RFC.md

All of the discussed technologies can be seen in the corresponding branches. I’ve pasted an abridged version of the RFC doc below for those who want to skim, but I encourage everyone to take a look at the full docs. We would love community feedback on the topics of:

  *   Where should various bits of the code live?
  *   What types of tests should be approached first?
  *   How should all of this be documented and maintained?

Thanks!
- Bret

==================================================

# Testing

TianoCore has been exploring a few types of testing and this page will help provide some high level info and links for more information.

## Testing Infrastructure - Definitions and Descriptions

### Host-Based Unit Tests (cmocka)

Host-based unit tests let you compile your unit tests to run as an application in the host.  This method can also leverage "mocking" objects/functions so that unit tests can validate functionality in isolation and force unexpected edge cases.  These unit tests call C functions directly with known parameter sets to force good and bad conditions.  They are linked directly against C code in either a library instance or module. These tests leverage a UnitTest Library.  For Project Mu we have chosen cmocka.

### Edk2- and UEFI-specific Codebase Analysis

_Mu_Build_ provides a framework for running static tests on the code base.  Simple tests like character encoding are examples.  In Project Mu we are working to expand this set of tests to include checking guids, checking for library classes, etc.

### Basic Compilation

Each package must have a _*Pkg.ci.dsc_ file.  This DSC should list every module and library instance inf in the components section for the appropriate architectures.  This forces compilation when doing a CI build.  Since this is only to validate code builds successfully the library classes used to resolve dependencies should leverage null library instances whenever possible.  These null libs should minimize dependencies and make DSC management minimal.

### Compile-Time Asserts

Compile-time asserts can be used to check assertions in the code for build-time defined data.  An example could be confirming the size of a pixel array for an image matches the width x length.

* C code (header and code)
  * Leverage C11 Static Assert feature for compile time verification in C code.
  * Read here for more details https://docs.microsoft.com/en-us/cpp/cpp/static-assert?view=vs-2019
  * And here https://en.cppreference.com/w/cpp/language/static_assert

Another great reason for these types of tests is that many IDEs will verify inline and show issues without the compiler.

### Runtime Debug Asserts

Not really a "testing" tool but more of a debug and development practice.

### Host-Based Fuzz Testing (INTEL)

> Some of the info that Intel has published.
> * Read here for some intel information. https://firmware.intel.com/sites/default/files/Intel_UsingHBFAtoImprovePlatformResiliency.pdf
> * First iteration from Intel: https://github.com/tianocore/edk2-staging/tree/HBFA/HBFA

### UEFI Shell-Based Functional Tests

Some tests are best run from within the UEFI environment.  These tests might be for APIs that leverage platform and global state.

> Review Project Mu Unit Test framework: https://github.com/Microsoft/mu_basecore/tree/release/201903/MsUnitTestPkg

### UEFI Shell-Based Audit Tests

These tests are run from UEFI to collect information in a machine-parsible format and then post-processed to compare against a "Golden Copy".  These tests often contain a UEFI shell application as well as a script for intelligent comparison against a known good "Golden Copy".  The "Golden Copy" could be device specific and is often curated and managed by a developer of that platform.

### SCT Framework

> Documented elsewhere.

[-- Attachment #2: Type: text/html, Size: 11206 bytes --]