From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=pass header.i=@microsoft.com header.s=selector1 header.b=e+X/pNoB; spf=pass (domain: microsoft.com, ip: 40.107.78.94, mailfrom: bret.barkelew@microsoft.com) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (NAM03-BY2-obe.outbound.protection.outlook.com [40.107.78.94]) by groups.io with SMTP; Fri, 06 Sep 2019 17:07:42 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DBOcOEVw/xZNCy0DYIRgljGZF8hlRQ8rhZLzK6J7qn6arIh19GPfxdxhPJgzDfv96qCoAVesyewdPQXZS0Ir90RkGRtvMZVTUC0dPPwBKpgW3bo+cxImqVHRgo9rgrb8qTYWg+UtR3OfzS2TEAJwl5FiStW7biwUc+I7Eg3v02oxV2ayYnfmvtO3SL1pBfxFgxnHbjFcXNXdO3yfroG4iVFVh9ufZ3KzUmZPelfFAd/gpj5+T654S8U0clQ4BXxYJ97inOTtArNakkDtQtXUXFtYR9BNcjP6GoB6URmAkQZoSEwk+KUugkPKGzRqNY5tCNBAhorIZRXQyBk4j1Kd4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bYk0PHo0xvq4lVjvrAX8H+iyhbnYvgatEbO4BTIEzm8=; b=TquIcU3Zga9h1XbCt/kJVzPG9nG1c7pIyCjESZCzM2zRm3dsu7cWjU6KasclcbYXDDK1dyXymkNfu2k/ilS5vz13pNsPjU2IPBNd+BAeQdDEJrkU+oBWGn5JgNN8Z8IOCnxLl89pUativ6f4Defk/v8Fjz42O3+Uf4XzGPEGN5rxC1x1urrmJUX7f/ZYTNRpf2AaU0J4XqasnF4rvkVsCbXXbpJvnP7NSPep4F+nLmLLOiVgRedqJlh/Arur1z0BM0adoVowCZJP5NIPa/2kifwRvRlMW6xTZl53P9rB6l0MI/WIdITEooq0jO0EFSwBbRtmkNHQsa321IVrJeunFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bYk0PHo0xvq4lVjvrAX8H+iyhbnYvgatEbO4BTIEzm8=; b=e+X/pNoBCQ7RVNxceoeRHaJaxnigikoPFbKfmOdCDEx+q1m1oDjwQra6WBdoTDlAZhs6wvpbA96zaroaBNSnqEA6w/KhYYj6jhrb3PwVWmAzLNDwHUlFoEebXJxUHBKcb50lYdzJqPrycTOtiiFFcej3XYPflTmX2CxZhygwStk= Received: from DM5PR2101MB1000.namprd21.prod.outlook.com (52.132.133.34) by DM5PR2101MB0887.namprd21.prod.outlook.com (52.132.132.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.9; Sat, 7 Sep 2019 00:07:41 +0000 Received: from DM5PR2101MB1000.namprd21.prod.outlook.com ([fe80::5845:6242:1bca:e44]) by DM5PR2101MB1000.namprd21.prod.outlook.com ([fe80::5845:6242:1bca:e44%2]) with mapi id 15.20.2263.005; Sat, 7 Sep 2019 00:07:41 +0000 From: "Bret Barkelew" To: "devel@edk2.groups.io" , Sean Brogan , "Kinney, Michael D" Subject: [edk2-devel] [edk2-rfc] [RFC] EDK2 Testing Formalization Thread-Topic: [edk2-devel] [edk2-rfc] [RFC] EDK2 Testing Formalization Thread-Index: AQHVZQ9aH5c+nePb6kuQxFlODzR51g== Date: Sat, 7 Sep 2019 00:07:40 +0000 Message-ID: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bret.Barkelew@microsoft.com; x-originating-ip: [166.255.248.60] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0ce3bfc6-c92f-4b9a-6f6c-08d733276663 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020);SRVR:DM5PR2101MB0887; x-ms-traffictypediagnostic: DM5PR2101MB0887:|DM5PR2101MB0887:|DM5PR2101MB0887: x-ms-exchange-transport-forked: True x-ms-exchange-purlcount: 9 x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0153A8321A x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(4636009)(366004)(136003)(39860400002)(396003)(346002)(376002)(199004)(189003)(6506007)(66476007)(66556008)(1511001)(33656002)(6116002)(7696005)(66446008)(66066001)(478600001)(26005)(3846002)(102836004)(236005)(5660300002)(9686003)(476003)(86362001)(256004)(54896002)(6306002)(66946007)(52536014)(64756008)(486006)(25786009)(76116006)(606006)(14454004)(14444005)(6436002)(22452003)(71190400001)(74316002)(71200400001)(186003)(53936002)(55016002)(110136005)(316002)(8936002)(99286004)(10290500003)(10090500001)(2501003)(8990500004)(966005)(7736002)(2906002)(81166006)(8676002)(81156014);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR2101MB0887;H:DM5PR2101MB1000.namprd21.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: wi9JkV1V4Wn8aSHmDXZV5sT0UJPf5EJRmaQ6IMmKaEHvYQTEJoX5H/B6s/8sg+oOylDU3hOAOcA+fSbiS5V0l3R9lUMvAIqNSqeBcxYcYeDAJrQzOUN+qzwIHw5zXgRNwoeXR4FTsWDW2d3rqIKyU9y2bhZ/2w9+YeQ8KMvFdq/hGtqjV0XymN9Rp60C+he339mIz8rv1gvSeZBPTI45p1nnW311rd3IGS0MrKAWe1wkKsw1cn6qh8nn+C0PHo5vUT1nJKfFW3lBeHIOhhMhPZqCltmbMhG8C9V6iqPItjBh5CWWojjyirtNg6mMPDVzpOCuDvJGAkalV723mUJVWcUs/sK556CrzNcnaKTWVagGtcwOlVhMVqcnMw90a3dH+oUWk9lWcWmJWV9dNL6ynYJUJ5NZmMVadey5Ejp3JBo= MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ce3bfc6-c92f-4b9a-6f6c-08d733276663 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Sep 2019 00:07:40.8199 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: LVD1mtTluPRdTjL+IaVIYBVmVuS2TVCKSHYHUBKr9KVobPdrQeh8jfZqwQY3bw1LVpYizNyYW0n3fk/lI2yorw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR2101MB0887 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_DM5PR2101MB1000C38F951954B5532A8658EFBA0DM5PR2101MB1000_" --_000_DM5PR2101MB1000C38F951954B5532A8658EFBA0DM5PR2101MB1000_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable In light of the parallel discussion happening right now about CI options, I= wanted to highlight a conversation that has been ongoing about Testing. Th= ere has been a lot of work contributed recently to expand that types and av= ailability of automated tests in the EDK2 tree. This discussion ranges from= topics on test technologies and approaches to types/definitions and code m= aintenance. A full RFC doc can be found here: https://github.com/spbrogan/edk2-staging/blob/edk2-host-test/Readme-RFC.md and the latest community version (though slightly older) can be found here: https://github.com/tianocore/edk2-staging/blob/edk2-host-test/Readme-RFC.md All of the discussed technologies can be seen in the corresponding branches= . I=92ve pasted an abridged version of the RFC doc below for those who want= to skim, but I encourage everyone to take a look at the full docs. We woul= d love community feedback on the topics of: * Where should various bits of the code live? * What types of tests should be approached first? * How should all of this be documented and maintained? Thanks! - Bret =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # Testing TianoCore has been exploring a few types of testing and this page will help= provide some high level info and links for more information. ## Testing Infrastructure - Definitions and Descriptions ### Host-Based Unit Tests (cmocka) Host-based unit tests let you compile your unit tests to run as an applicat= ion in the host. This method can also leverage "mocking" objects/functions= so that unit tests can validate functionality in isolation and force unexp= ected edge cases. These unit tests call C functions directly with known pa= rameter sets to force good and bad conditions. They are linked directly ag= ainst C code in either a library instance or module. These tests leverage a= UnitTest Library. For Project Mu we have chosen cmocka. ### Edk2- and UEFI-specific Codebase Analysis _Mu_Build_ provides a framework for running static tests on the code base. = Simple tests like character encoding are examples. In Project Mu we are w= orking to expand this set of tests to include checking guids, checking for = library classes, etc. ### Basic Compilation Each package must have a _*Pkg.ci.dsc_ file. This DSC should list every mo= dule and library instance inf in the components section for the appropriate= architectures. This forces compilation when doing a CI build. Since this= is only to validate code builds successfully the library classes used to r= esolve dependencies should leverage null library instances whenever possibl= e. These null libs should minimize dependencies and make DSC management mi= nimal. ### Compile-Time Asserts Compile-time asserts can be used to check assertions in the code for build-= time defined data. An example could be confirming the size of a pixel arra= y for an image matches the width x length. * C code (header and code) * Leverage C11 Static Assert feature for compile time verification in C c= ode. * Read here for more details https://docs.microsoft.com/en-us/cpp/cpp/sta= tic-assert?view=3Dvs-2019 * And here https://en.cppreference.com/w/cpp/language/static_assert Another great reason for these types of tests is that many IDEs will verify= inline and show issues without the compiler. ### Runtime Debug Asserts Not really a "testing" tool but more of a debug and development practice. ### Host-Based Fuzz Testing (INTEL) > Some of the info that Intel has published. > * Read here for some intel information. https://firmware.intel.com/sites/= default/files/Intel_UsingHBFAtoImprovePlatformResiliency.pdf > * First iteration from Intel: https://github.com/tianocore/edk2-staging/t= ree/HBFA/HBFA ### UEFI Shell-Based Functional Tests Some tests are best run from within the UEFI environment. These tests migh= t be for APIs that leverage platform and global state. > Review Project Mu Unit Test framework: https://github.com/Microsoft/mu_ba= secore/tree/release/201903/MsUnitTestPkg ### UEFI Shell-Based Audit Tests These tests are run from UEFI to collect information in a machine-parsible = format and then post-processed to compare against a "Golden Copy". These t= ests often contain a UEFI shell application as well as a script for intelli= gent comparison against a known good "Golden Copy". The "Golden Copy" coul= d be device specific and is often curated and managed by a developer of tha= t platform. ### SCT Framework > Documented elsewhere. --_000_DM5PR2101MB1000C38F951954B5532A8658EFBA0DM5PR2101MB1000_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

In light of the parallel discussion happening right = now about CI options, I wanted to highlight a conversation that has been on= going about Testing. There has been a lot of work contributed recently to e= xpand that types and availability of automated tests in the EDK2 tree. This discussion ranges from topics on= test technologies and approaches to types/definitions and code maintenance= .

 

A full RFC doc can be found here:

https://github.com/spbrogan/edk2-staging= /blob/edk2-host-test/Readme-RFC.md

and the latest community version (though slightly ol= der) can be found here:

https://github.com/tianocore/edk2-stagi= ng/blob/edk2-host-test/Readme-RFC.md

 

All of the discussed technologies can be seen in the= corresponding branches. I=92ve pasted an abridged version of the RFC doc b= elow for those who want to skim, but I encourage everyone to take a look at= the full docs. We would love community feedback on the topics of:

  • Where should various bits of the code live?
  • What types of t= ests should be approached first?
  • How should all of this be docu= mented and maintained?

 

Thanks!

- Bret

 

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D

 

# Testing

 

TianoCore has been exploring a few types of testing = and this page will help provide some high level info and links for more inf= ormation.

 

## Testing Infrastructure - Definitions and Descript= ions

 

### Host-Based Unit Tests (cmocka)

 

Host-based unit tests let you compile your unit test= s to run as an application in the host.  This method can also leverage= "mocking" objects/functions so that unit tests can validate func= tionality in isolation and force unexpected edge cases.  These unit tests call C functions directly with known paramet= er sets to force good and bad conditions.  They are linked directly ag= ainst C code in either a library instance or module. These tests leverage a= UnitTest Library.  For Project Mu we have chosen cmocka.

 

### Edk2- and UEFI-specific Codebase Analysis

 

_Mu_Build_ provides a framework for running static t= ests on the code base.  Simple tests like character encoding are examp= les.  In Project Mu we are working to expand this set of tests to incl= ude checking guids, checking for library classes, etc.

 

### Basic Compilation

 

Each package must have a _*Pkg.ci.dsc_ file.  T= his DSC should list every module and library instance inf in the components= section for the appropriate architectures.  This forces compilation w= hen doing a CI build.  Since this is only to validate code builds successfully the library classes used to resolve depe= ndencies should leverage null library instances whenever possible.  Th= ese null libs should minimize dependencies and make DSC management minimal.

 

### Compile-Time Asserts

 

Compile-time asserts can be used to check assertions= in the code for build-time defined data.  An example could be confirm= ing the size of a pixel array for an image matches the width x length. = ;

 

* C code (header and code)

  * Leverage C11 Static Assert feature for comp= ile time verification in C code.

  * Read here for more details https://docs.mic= rosoft.com/en-us/cpp/cpp/static-assert?view=3Dvs-2019

  * And here https://en.cppreference.com/w/cpp/= language/static_assert

 

Another great reason for these types of tests is tha= t many IDEs will verify inline and show issues without the compiler. 

 

### Runtime Debug Asserts

 

Not really a "testing" tool but more of a = debug and development practice.

 

### Host-Based Fuzz Testing (INTEL)

 

> Some of the info that Intel has published.=

> * Read here for some intel information. https:/= /firmware.intel.com/sites/default/files/Intel_UsingHBFAtoImprovePlatformRes= iliency.pdf

> * First iteration from Intel: https://github.co= m/tianocore/edk2-staging/tree/HBFA/HBFA

 

### UEFI Shell-Based Functional Tests

 

Some tests are best run from within the UEFI environ= ment.  These tests might be for APIs that leverage platform and global= state.

 

> Review Project Mu Unit Test framework: https://= github.com/Microsoft/mu_basecore/tree/release/201903/MsUnitTestPkg

 

### UEFI Shell-Based Audit Tests

 

These tests are run from UEFI to collect information= in a machine-parsible format and then post-processed to compare against a = "Golden Copy".  These tests often contain a UEFI shell appli= cation as well as a script for intelligent comparison against a known good "Golden Copy".  The "Golden Copy&= quot; could be device specific and is often curated and managed by a develo= per of that platform.

 

### SCT Framework

 

> Documented elsewhere.

--_000_DM5PR2101MB1000C38F951954B5532A8658EFBA0DM5PR2101MB1000_--