Unit tests and the EDK2

Unit tests and the EDK2

[Matt Lazarowitz]

I would like to find out if anyone has experience with an off the shelf
unit test framework in the EDK2.
I'm currently experimenting with embUnit and have some of the project's
sample code running without using the StdLib package. My eventual goal is
to see if this can be used as an automated unit test system similar to what
is used in other areas of software development.

I'm at a point where I believe I could write some basic tests for library
function calls, but before I continue, I wanted to check if anyone else has
done any development with the same goal in mind.

Matt

Re: Unit tests and the EDK2

[Blibbet]

On 11/06/2016 01:57 AM, Matt Lazarowitz wrote:
> I would like to find out if anyone has experience with an off the shelf
> unit test framework in the EDK2.[...]

If you haven't looked at it, two months ago Microsoft open-sourced some
EDK2-centric unit tests.

https://firmwaresecurity.com/2016/09/23/microsoft-uefi-unit-tests/

Lee Fisher

Re: Unit tests and the EDK2

[Kinney, Michael D]

Hi,

A test framework for EDK2 is one of my highest priorities to
complete before the end of the year.

I am evaluating a number of options and hope to put together 
a complete proposal as an RFC for consideration in the next 
few weeks.

Thanks,

Mike


> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Blibbet
> Sent: Sunday, November 6, 2016 9:26 AM
> To: edk2-devel@lists.01.org
> Subject: Re: [edk2] Unit tests and the EDK2
> 
> On 11/06/2016 01:57 AM, Matt Lazarowitz wrote:
> > I would like to find out if anyone has experience with an off the shelf
> > unit test framework in the EDK2.[...]
> 
> If you haven't looked at it, two months ago Microsoft open-sourced some
> EDK2-centric unit tests.
> 
> https://firmwaresecurity.com/2016/09/23/microsoft-uefi-unit-tests/
> 
> Lee Fisher
> 
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: Unit tests and the EDK2

[Marvin H?user]

Hey Mike,

Is the framework you plan to RFC a framework within the UEFI environment (UEFI Shell) or within the OS?
Using the OS implementations of UEFI (Nt32 & Emulator) to run Unit Tests without the need for a separate UEFI device or a reboot sounds pretty compelling to me, to be honest.

Thank you very much!

Regards,
Marvin.

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Kinney, Michael D
> Sent: Sunday, November 6, 2016 8:04 PM
> To: Blibbet <blibbet@gmail.com>; edk2-devel@lists.01.org; Kinney, Michael
> D <michael.d.kinney@intel.com>
> Subject: Re: [edk2] Unit tests and the EDK2
> 
> Hi,
> 
> A test framework for EDK2 is one of my highest priorities to complete before
> the end of the year.
> 
> I am evaluating a number of options and hope to put together a complete
> proposal as an RFC for consideration in the next few weeks.
> 
> Thanks,
> 
> Mike
> 
> 
> > -----Original Message-----
> > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> > Blibbet
> > Sent: Sunday, November 6, 2016 9:26 AM
> > To: edk2-devel@lists.01.org
> > Subject: Re: [edk2] Unit tests and the EDK2
> >
> > On 11/06/2016 01:57 AM, Matt Lazarowitz wrote:
> > > I would like to find out if anyone has experience with an off the
> > > shelf unit test framework in the EDK2.[...]
> >
> > If you haven't looked at it, two months ago Microsoft open-sourced
> > some EDK2-centric unit tests.
> >
> > https://firmwaresecurity.com/2016/09/23/microsoft-uefi-unit-tests/
> >
> > Lee Fisher
> >
> >
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: Unit tests and the EDK2

[Kinney, Michael D]

Marvin,

UEFI environment.

Emulated environments are great for some types of testing.

OVMF is also interesting for something that is a bit closer to 
real HW without requiring the real hardware.

And types of testing always requires a real HW target.


Mike

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Marvin H?user
> Sent: Sunday, November 6, 2016 11:58 AM
> To: edk2-devel@lists.01.org
> Cc: Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2] Unit tests and the EDK2
> 
> Hey Mike,
> 
> Is the framework you plan to RFC a framework within the UEFI environment (UEFI Shell)
> or within the OS?
> Using the OS implementations of UEFI (Nt32 & Emulator) to run Unit Tests without the
> need for a separate UEFI device or a reboot sounds pretty compelling to me, to be
> honest.
> 
> Thank you very much!
> 
> Regards,
> Marvin.
> 
> > -----Original Message-----
> > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> > Kinney, Michael D
> > Sent: Sunday, November 6, 2016 8:04 PM
> > To: Blibbet <blibbet@gmail.com>; edk2-devel@lists.01.org; Kinney, Michael
> > D <michael.d.kinney@intel.com>
> > Subject: Re: [edk2] Unit tests and the EDK2
> >
> > Hi,
> >
> > A test framework for EDK2 is one of my highest priorities to complete before
> > the end of the year.
> >
> > I am evaluating a number of options and hope to put together a complete
> > proposal as an RFC for consideration in the next few weeks.
> >
> > Thanks,
> >
> > Mike
> >
> >
> > > -----Original Message-----
> > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> > > Blibbet
> > > Sent: Sunday, November 6, 2016 9:26 AM
> > > To: edk2-devel@lists.01.org
> > > Subject: Re: [edk2] Unit tests and the EDK2
> > >
> > > On 11/06/2016 01:57 AM, Matt Lazarowitz wrote:
> > > > I would like to find out if anyone has experience with an off the
> > > > shelf unit test framework in the EDK2.[...]
> > >
> > > If you haven't looked at it, two months ago Microsoft open-sourced
> > > some EDK2-centric unit tests.
> > >
> > > https://firmwaresecurity.com/2016/09/23/microsoft-uefi-unit-tests/
> > >
> > > Lee Fisher
> > >
> > >
> > > _______________________________________________
> > > edk2-devel mailing list
> > > edk2-devel@lists.01.org
> > > https://lists.01.org/mailman/listinfo/edk2-devel
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: Unit tests and the EDK2

[Matt Lazarowitz]

I decided to give embUnit a try since it offers basic functionality without
the C standard libraries. I have my work available on github at
https://github.com/mattlazarowitz/UnitTestNoStdLibPkg

The embUnitSamples folder may give you an idea of what I am trying. I
create a new folder in the package and set up a new component in the
package. I'll add the library with a function I want to unit test and
create the tests. The inf is set up to build an application so it stands
alone and doe not necessarily need to be run on the target platform.
Drivers will potentially be more difficult to create unit tests for. Either
I need to be able to incorporate the obj files into an application in my
unit test package, or a unit test inf and associated C files that live next
to the driver would work but make for a messier file structure.
An engineer could architect a driver in such a way that most of it is built
as a library and pulled into a driver framework as another alternative.
If all the test can live in the package it should be somewhat easy for a
build automation system to gather and deploy the unit tests automatically.
A simulated environment would be great as a place to execute the tests for
fast feedback.

Hardware level tests will always be needed. Unit testing is just part of
the test pyramid which also has system level and integration tests. The
unit tests I have in mind are meant to provide rapid feedback to a
programmer before sinking time into longer tests.

I don't have any test doubles or any way to get inversion of control yet. I
hope a unit test package will be able to borrow from SCT as much as
possible.


A Microsoft authored unit test harness was mentioned earlier in this thread
which might be worth investigating as well. I took a quick look and it
seems very similar to what I was able to accomplish with embUnit. I'll have
to take a longer look to really compare it. One advantage of embUnit is
that is is fairly well documented and is stable code even if it doesn't
conform the the EDK2 coding standard.

I would also like to evaluate the StdLib package and see how well it will
work with some of the more advanced embUnit features that require the C
standard libraries.
I'm glad to hear I'm not the only one thinking about this.

Matt

On Sun, Nov 6, 2016 at 1:06 PM, Kinney, Michael D <
michael.d.kinney@intel.com> wrote:

> Marvin,
>
> UEFI environment.
>
> Emulated environments are great for some types of testing.
>
> OVMF is also interesting for something that is a bit closer to
> real HW without requiring the real hardware.
>
> And types of testing always requires a real HW target.
>
>
> Mike
>
> > -----Original Message-----
> > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Marvin H?user
> > Sent: Sunday, November 6, 2016 11:58 AM
> > To: edk2-devel@lists.01.org
> > Cc: Kinney, Michael D <michael.d.kinney@intel.com>
> > Subject: Re: [edk2] Unit tests and the EDK2
> >
> > Hey Mike,
> >
> > Is the framework you plan to RFC a framework within the UEFI environment
> (UEFI Shell)
> > or within the OS?
> > Using the OS implementations of UEFI (Nt32 & Emulator) to run Unit Tests
> without the
> > need for a separate UEFI device or a reboot sounds pretty compelling to
> me, to be
> > honest.
> >
> > Thank you very much!
> >
> > Regards,
> > Marvin.
> >
> > > -----Original Message-----
> > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> > > Kinney, Michael D
> > > Sent: Sunday, November 6, 2016 8:04 PM
> > > To: Blibbet <blibbet@gmail.com>; edk2-devel@lists.01.org; Kinney,
> Michael
> > > D <michael.d.kinney@intel.com>
> > > Subject: Re: [edk2] Unit tests and the EDK2
> > >
> > > Hi,
> > >
> > > A test framework for EDK2 is one of my highest priorities to complete
> before
> > > the end of the year.
> > >
> > > I am evaluating a number of options and hope to put together a complete
> > > proposal as an RFC for consideration in the next few weeks.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > >
> > > > -----Original Message-----
> > > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf
> Of
> > > > Blibbet
> > > > Sent: Sunday, November 6, 2016 9:26 AM
> > > > To: edk2-devel@lists.01.org
> > > > Subject: Re: [edk2] Unit tests and the EDK2
> > > >
> > > > On 11/06/2016 01:57 AM, Matt Lazarowitz wrote:
> > > > > I would like to find out if anyone has experience with an off the
> > > > > shelf unit test framework in the EDK2.[...]
> > > >
> > > > If you haven't looked at it, two months ago Microsoft open-sourced
> > > > some EDK2-centric unit tests.
> > > >
> > > > https://firmwaresecurity.com/2016/09/23/microsoft-uefi-unit-tests/
> > > >
> > > > Lee Fisher
> > > >
> > > >
> > > > _______________________________________________
> > > > edk2-devel mailing list
> > > > edk2-devel@lists.01.org
> > > > https://lists.01.org/mailman/listinfo/edk2-devel
> > > _______________________________________________
> > > edk2-devel mailing list
> > > edk2-devel@lists.01.org
> > > https://lists.01.org/mailman/listinfo/edk2-devel
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
>

Re: Unit tests and the EDK2

[Blibbet]

> I am evaluating a number of options and hope to put together
> a complete proposal as an RFC for consideration in the next
> few weeks.

For your evaluation, please also check out the U-Boot and coreboot
projects, they both have some built-in tests.

Thanks,
Lee Fisher