From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ma1-aaemail-dr-lapp02.apple.com (ma1-aaemail-dr-lapp02.apple.com [17.171.2.68]) by mx.groups.io with SMTP id smtpd.web11.8264.1575481853650057922 for ; Wed, 04 Dec 2019 09:50:54 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=ZxeyR2Qn; spf=pass (domain: apple.com, ip: 17.171.2.68, mailfrom: afish@apple.com) Received: from pps.filterd (ma1-aaemail-dr-lapp02.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp02.apple.com (8.16.0.27/8.16.0.27) with SMTP id xB4HVqCv042976; Wed, 4 Dec 2019 09:50:52 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=sender : from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=xJacRNdkoKQa6pC6uoZOVcpYvK4uQACIPfDGY1eVDfM=; b=ZxeyR2Qnhe8SwLJJJJDZZIiz5EdNEk9XpGuow3BU2s45arHKufpbwJ6sY5KiomZQDDas RUtZ0hRELSdmMmEbEA+zu+W23tI3Dl8B9/GLwoA/PuMo6Mjf+JP1V+iY1rife1MHBVON 8izTJAirdTE39gBJp3P3WMpNDekP4dVBO936X4mrtWyz2PLi4zbH1pAEXKmMIaBFIx0D 03TLDZ8Mjh39byA74LO0IrGty1pAZiHkvNKEQXYUXKdP2UrKFKu0SfVk0o0/hZV2tt1j uz/ejhDf2JmeT8S4fvuUemoo4LbFz/XdWDb+SjfJaHoOAv7gCGu2FlY80XJ0yI0Bucos Zg== Received: from mr2-mtap-s03.rno.apple.com (mr2-mtap-s03.rno.apple.com [17.179.226.135]) by ma1-aaemail-dr-lapp02.apple.com with ESMTP id 2wknyxwgra-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 04 Dec 2019 09:50:51 -0800 Received: from nwk-mmpp-sz12.apple.com (nwk-mmpp-sz12.apple.com [17.128.115.204]) by mr2-mtap-s03.rno.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPS id <0Q2000IU01KRGK00@mr2-mtap-s03.rno.apple.com>; Wed, 04 Dec 2019 09:50:51 -0800 (PST) Received: from process_milters-daemon.nwk-mmpp-sz12.apple.com by nwk-mmpp-sz12.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) id <0Q2000H001KNW700@nwk-mmpp-sz12.apple.com>; Wed, 04 Dec 2019 09:50:50 -0800 (PST) X-Va-A: X-Va-T-CD: e48e8dc3f6c377b8dc939b4126ad19f3 X-Va-E-CD: eb178d38e7b80098287a9ee137c4a794 X-Va-R-CD: ad9d6f5dd9e246f356e289950801c66b X-Va-CD: 0 X-Va-ID: 4c89714b-7a83-4ccc-bc38-d8464fcbec51 X-V-A: X-V-T-CD: e48e8dc3f6c377b8dc939b4126ad19f3 X-V-E-CD: eb178d38e7b80098287a9ee137c4a794 X-V-R-CD: ad9d6f5dd9e246f356e289950801c66b X-V-CD: 0 X-V-ID: b692a8bf-f1ba-4153-8826-896cad9c8eb4 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-12-04_03:,, signatures=0 Received: from da0602a-dhcp197.apple.com (da0602a-dhcp197.apple.com [17.226.23.197]) by nwk-mmpp-sz12.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPSA id <0Q20006951KAYY20@nwk-mmpp-sz12.apple.com>; Wed, 04 Dec 2019 09:50:34 -0800 (PST) Sender: afish@apple.com From: "Andrew Fish" Message-id: <0F460E19-2A3B-4FEE-8B6F-BE65B9C5F1D4@apple.com> MIME-version: 1.0 (Mac OS X Mail 13.0 \(3594.4.17\)) Subject: Re: [edk2-devel] EDK2 Host-Based Unit Test RFC (Now with docs!) Date: Wed, 04 Dec 2019 09:50:33 -0800 In-reply-to: Cc: "bret.barkelew@microsoft.com" , "rfc@edk2.groups.io" To: devel@edk2.groups.io, Mike Kinney References: X-Mailer: Apple Mail (2.3594.4.17) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-12-04_03:,, signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_A4C9B025-4C7B-4B4F-AB1D-E05DBF90075B" --Apple-Mail=_A4C9B025-4C7B-4B4F-AB1D-E05DBF90075B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Mike, I like and agree with your comments. On the UnitTestPkg(s) dependency issue I think it would make sense to move= as much as possible into the *Pkg/Test/ ( *Pkg/HostLibrary, MdePkg/MdePkg= Test.dsc, etc.) directory structure. It looks like for implementation speci= fic tests we skip the Test directory and drop the UnitTest or FuzzTest dire= ctly into modules directory. Maybe we should follow the same pattern as *Pk= g/Test and have a Test directory? This will help minimize the number of "re= served" directories we need for managing the testing. Have a standardized d= irectory structure will make it easier to differentiate the code from tests= while doing `git grep` or other forms of code spelunking.=20 A Host-Based Unit Test for a Library makes sense as you can link directly = against the library and test it. A Host-Based Unit Test for Protocol/PPI/GU= ID [1] seems a little more complex. It is easy enough to write tests for th= e interfaces but what APIs do you call to get access to the protocol?=20 Per our conversation at the Stewards meeting I think make sense to try to = roll out the testing of libraries as the 1st phase. The mocking required fo= r drivers could get quite complex and let us not get bogged down in all tha= t to get something working.=20 [1] *Pkg/Test/UnitTest/[Library|Protocol|Ppi|Guid]=20 Thanks, Andrew Fish > On Dec 2, 2019, at 3:12 PM, Michael D Kinney wrote: >=20 > Hi Bret, > > Thanks for posting this content. Host based unit testing is a very valu= able addition to the CI checks. > > I have the following comments: > > I see that MdePkg adds a dependency on UnitTestPkg. This makes UnitTest= Pkg the root package when building and running host based unit tests. This= makes sense, but good to highlight that all packages that use host based t= ests will introduce a new package dependency on UnitTestPkg. > Since the dependency only applies to unit tests, can we update the Depen= dencyCheck plugin to support listing dependencies for FW components separat= e from dependencies for unit tests? > I see UnitTestPkg declares 6 new lib classes. Are all 6 classes intende= d to be used directly from a unit test case? If some of these are only int= ended to be used from the framework inside the UnitTestPkg can we make them= private? > In the MdePkg, I see a new top level directory called =E2=80=98HostLibra= ry=E2=80=99. Since these lib instances are only used from a host based test= , can they be moved into the =E2=80=98Test=E2=80=99 directory? > MdePkg/MdePkgTest.dsc.=20 > Can this DSC file be moved into the =E2=80=98Test=E2=80=99 directory?=20 > I see this DSC file only supports VS today. How much work is it to add = GCC support? > MdePkg/HostLibrary/BaseLibHost > Why are there 2 INFs. One with ASM and one without ASM? Can we just us= e the one with ASM and assume NASM is installed? > I see the purpose of this lib instance is to call the > SetJump()/LongJump(). So these implementations always work? It looks l= ike it assumes BASE_LIBRARY_JUMP_BUFFER is identical to the host OS user mo= de applicationsetjmp()/longjmp() state. > Why are DRx and CRx registers emulated? I would think and code that dep= ends on read/writing these registers would not be compatible with host base= d testing. Can we change to ASSERT (FALSE)? > PatchInstructionX86() =E2=80=93 I suspect this will not work in a host b= ased environment because it is self modifying code. Should it be ASSERT (F= ALSE)? > MdePkg/HostLibrary/DebugLibHost > What is =E2=80=98#ifndef TEST_WITH_KLEE=E2=80=99 > What is the =E2=80=98PatchFormat()=E2=80=99 function? It is always disa= bled with if (0). > Are the PCDs to set the debug message levels disabled on purpose? (Debug= PrintEnabled(), DebugPrintLevelEnabled(), DebugCodeEnabled()) > MdePkg/HostLibrary/BaseMemoryLibHost > Why can=E2=80=99t we use MdePkg/Library/BaseMemoryLib/BaseMemoryLib/inf = instead and reduce the number of host specific lib instances? > MdePkg/HostLibrary/MemoryAllocationLibHost > Why is are memcpy(), assert(), memset() used in this lib? I would expec= t this lib instance to match the UefiMemoryAllocationLib with the only the = use of malloc() and free() to replace the UEFI Boot Services calls. > Host library instance naming conventions. > The current naming convention uses the environment as a prefix(e.g. Pei,= Smm, Uefi) and the services the lib instance uses as a post fix. Would it= make more sense to use =E2=80=98Host=E2=80=99 as a prefix instead of a pos= tfix in the lib instance names? > Unicode vs ASCII strings > I see InitUnitTestFramework(), CreateUnitTestSuite(), and AddTestCase() = all take Unicode strings for the labels. I also see extra code to convert = gEfiCallerBaseName from ASCII to Unicode to use it as a short name of a tes= t framework. I think it would be simpler if the parameters to these APIs w= ere ASCII and the framework can convert to Unicode if needed. > Will InitUnitTestFramework() and CreateUnitTestSuite() always be called = in pairs? If so, can we combine these to a single API?=20 > I see the SafeIntLib example create a single framework and multiple test= suites. Perhaps we can have a single CreateUnitTestSuite() API where Fram= ework is an IN/OUT and if it is passed in as NULL, the Framework handle is = created. > I see a pattern where the CreateUnitTestSuite() =E2=80=98Package=E2=80= =99 parameter is used as a prefix to every call to AddTestCase() in the = =E2=80=98ClassName=E2=80=99 parameter. Can we simplify AddTestCase() so i= t only need to pass in the name of the unit test case, and the framework ca= n append Package and the unit test case name? > I see the use of the =E2=80=98Fw=E2=80=99 variable as a shorthand for = =E2=80=98Framework=E2=80=99. Can we use something other than =E2=80=98Fw= =E2=80=99. It may be confused with =E2=80=98Firmware=E2=80=99. > UnitTestPkg/Include/UnitTestTypes.h > I see several hard coded string lengths. Since this runs in a host envi= ronment and strings can always be allocated, can the hard coded lengths be = removed? Update the structs to use pointers to strings instead of string a= rrays, and the framework can manage alloc() and free()? > How are Fingerprints used? The idea of using as hash as a unique identi= fier is a good idea. How is the hash calculated? What unit test code arti= facts are used so developers know what parameters must be unique? Can we j= ust decide to use a specific hash algorithm/size and the structure can be a= pointer to an allocated buffer instead of a fixed size array to make it ea= sy to change the algorithm/size in the future? > Update all the strings to be ASCII? See Unicode vs ASCII above. > UNIT_TEST_SAVE_TEST =E2=80=93 The last field is a variable sized array, = so it can be a formal field. > UNIT_TEST_SAVE_CONTEXT - =E2=80=93 The last field is a variable sized ar= ray, so it can be a formal field. > UNIT_TEST_SAVE_HEADER =E2=80=93 Can the last 3 fields be changed to poin= ters and be formal fields? > Do the structures really need to be packed? Specially with the changes = suggested above? Is the intent to potentially share data stored on disk be= tween different host execution environments that may have different width a= rchitectures? > UnitTestPkg =E2=80=93 UnitTestLib.h > Can you provide more context for the APIs SaveFrameworkState(), SaveFram= eworkStateAndQuit(), SaveFrameworkStateAndReboot(), SetFrameworkBootNextDev= ice(). I do not see any Load() functions, so how would a set of tests be r= esumed? If these do not apply to host based tests, should these be split o= ut to a different lib class? > UnitTestPkg/Library/UnitTestLib > I see an implementation of MD5. We should not do our own. We should us= e an approved implementation such as OpenSSL. > UnitTestPkg/Library/UnitTestTerminationLibTbd > Do we really need this lib instance now? > > Thanks, > > Mike > > From: devel@edk2.groups.io > On Behalf Of Bret Barkelew via Groups= .Io > Sent: Thursday, November 21, 2019 11:39 PM > To: devel@edk2.groups.io > Subject: [edk2-devel] EDK2 Host-Based Unit Test RFC (Now with docs!) > > Now that CI has landed in edk2/master, we'd like to get the basic framew= ork for unittesting staged as well. Target intercept date would be immediat= ely after the 2019/11 stabilization, so we wanted to go ahead and get comme= nts now. >=20 > The host unittest framework consists of five primary pieces: > - The test library (Cmocka) > - Support libraries (Found in UnitTestPkg) > - The test support plugins (HostUnitTestComilerPlugin, HostUnitTestDxeCo= mpleteCheck, HostBasedUnitTestRunner) > - The configuration in the package-based *.ci.yaml file and package-base= d Test.dsc > - The tests themselves >=20 > We have a demo branch set up at: > https://github.com/corthon/edk2-staging/tree/edk2-host-test_v2 > We also have a demo build pipeline at: > https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=3D36&_a= = =3Dsummary >=20 > The current demo branch contains a single test in MdePkg for SafeIntLib = (module file MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib= .inf). This test is automatically detected by the HostUnitTestComilerPlugin= and run by the HostBasedUnitTestRunner as part of the CI process. >=20 > A few notes about the current demo: > 1) The demo currently only works on Windows build chains, but there's no= reason to believe that it can't work equally well on Linux build chains, a= nd are happy to work with anyone to get it there. >=20 > 2) The demo currently has four failing conditions that can be seen towar= ds the end of MdePkg "Build and Test" log file for this build: > https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=3D28= 13 > "WARNING - Test SafeInt16ToChar8 - Status > d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.= c:302: error: Failure! > WARNING - TestBaseSafeIntLib.exe Test Failed > WARNING - Test SafeInt32ToChar8 - Status > d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.= c:638: error: Failure! > WARNING - TestBaseSafeIntLib.exe Test Failed > WARNING - Test SafeIntnToChar8 - Status > d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.= c:1051: error: Failure! > WARNING - TestBaseSafeIntLib.exe Test Failed > WARNING - Test SafeInt64ToChar8 - Status > d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.= c:1456: error: Failure!" > These failures seem to be legitimate and further work should be done by = the community to decide whether the test case is correct or the library is = correct, but one of them needs to change. >=20 > 3) Current demo pulls in test collateral from a fork of the edk2-test re= po. This repo is currently *very* heavy due to the history of the UEFI SCT = project and the number of binaries that it pulls down. It's possible that w= e (the community) need to select a better place for this code to live. Mayb= e in edk2 primary (though it's not explicitly firmware code, so it seems un= necessary). Maybe in a new edk2-test2 repo or something like that. >=20 > There=E2=80=99s an RFC doc here: https://github.com/corthon/edk2-staging= /blob/edk2-host-test_v2/Readme-RFC.md > And a usage guide here: https://github.com/corthon/edk2-staging/blob/edk= 2-host-test_v2/UnitTestPkg/ReadMe.md >=20 > Once again, would love to get this into EDK2 master after stabilization,= and most of this has previously been shopped around in other discussion th= reads. This is just where the rubber meets the road and is the minimal subs= et of code that needs to land for folks to start contributing unittests aga= inst the core libraries that can be run as part of any CI pass. >=20 > Thanks! > - Bret >=20 >=20 --Apple-Mail=_A4C9B025-4C7B-4B4F-AB1D-E05DBF90075B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Mike,

I like and agree with your comments.
=

On the UnitTestPkg(s) = dependency issue I think it would make sense to move as much as possible in= to the *Pkg/Test/ ( *Pkg/HostLibrary,  MdePkg/MdePkgTest.dsc= , etc.) directory structure. It looks like for implementation specific test= s we skip the Test directory and drop the UnitTest or FuzzTest directly int= o modules directory. Maybe we should follow the same pattern as *Pkg/Test a= nd have a Test directory? This will help minimize the number of "reserved" = directories we need for managing the testing. Have a standardized directory= structure will make it easier to differentiate the code from tests while d= oing `git grep` or other forms of code spelunking. 

A Host-Based Unit Test for a Librar= y makes sense as you can link directly against the library and test it. A H= ost-Based Unit Test for Protocol/PPI/GUID [1] seems a little more complex. = It is easy enough to write tests for the interfaces but what APIs do you ca= ll to get access to the protocol? 

Per our conversation at the Stewards meeting I think= make sense to try to roll out the testing of libraries as the 1st phase. T= he mocking required for drivers could get quite complex and let us not get = bogged down in all that to get something working. 

[1] *Pkg/Test/UnitTest/[Library= |Protocol|Ppi|Guid] 

Thanks,

= Andrew Fish


On Dec 2, 2019, at 3:12 PM, M= ichael D Kinney <michael.d.kinney@intel.com> wrote:

Hi = Bret,
 
Thanks for posting this content.&n= bsp; Host based un= it testing is a very valuable addition to the CI checks.
 
I h= ave the following comments:
 
  • I see t= hat MdePkg adds a de= pendency on UnitTestPkg.  This makes UnitTestPkg the root package when building and= running host based unit tests.  This makes sense, but good to highligh= t that all packages that use host based tests will introduce a new package = dependency on UnitTestPkg.
    1. Since the dependenc= y only applies to unit tests, can we update the DependencyCheck plugin to support listing depen= dencies for FW components separate from dependencies for unit tests?
  • I see <= /span>UnitTestPkg declares 6 new lib classes.  Are all 6 classes i= ntended to be used directly from a unit test case?  If some of these ar= e only intended to be used from the framework inside the UnitTestPkg can we make them private?<= o:p class=3D"">
  • In the = MdePkg, I see a new top level directory calle= d =E2=80=98HostLibrary=E2=80=99. Since these = lib instances are only used from a host based test, can they be moved into<= span class=3D"Apple-converted-space">  the =E2=80=98Test=E2=80=99 directory?
  • MdePkg/MdePk= gTest.dsc. 
    1. Can = this DSC file be moved into the =E2=80=98Test=E2=80=99 directory? 
    2. I see this DSC file only supports VS= today.  <= /span>How much work is it to add GCC support?<= /span>
  • MdePkg/HostLibrary/BaseLibHost
    1. Why are there 2 INFs. <= span class=3D"Apple-converted-space"> One with ASM and o= ne without ASM?  Can we just use the one with ASM and assume NASM is in= stalled?
    2. I see the purpose of this lib instance is to cal= l the
    3. SetJump()/LongJump().&nb= sp; So these imple= mentations always work?  It looks like it assumes BASE_LIBRARY_JUMP_BUF= FER is identical to the host OS user mode applicationsetjmp()/longjmp() state.
    4. Why are DRx and CRx registers = emulated?  = ;I would think and code that depends on read/writing these re= gisters would not be compatible with host based testing.&n= bsp; Can we change= to ASSERT (FALSE)?
    5. PatchInstructionX86() =E2=80=93 I sus= pect this will not work in a host based environment because it is self modify= ing code.  Sho= uld it be ASSERT (FALSE)?
  • MdePkg/HostLibrary<= /span>/DebugLibHost
    1. Wh= at is =E2=80=98#ifndef TEST_WITH_KLEE=E2=80=99
    2. What = is the =E2=80=98PatchFormat()=E2=80=99 functi= on?  It is always disabled with if (0).
    3. Are the PCDs = to set the debug message levels disabled on purpose? (DebugPrintEnabled(), DebugPrintLevelEnabled(), DebugCodeEnabled= ())
  • MdePkg= /HostLibrary/BaseMemoryLibHost
    1. Why can=E2= =80=99t we use MdePkg/Library/BaseMemoryLib/BaseMemoryLib/inf instead and reduce the= number of host specific lib instances?
    2. <= span class=3D"">MdePkg/HostLibrary/MemoryAllocationLibHost<= o:p class=3D"">
      1. Why is are&n= bsp;memcpy(), assert(), memset() = used in this lib?  I would expect this lib instance to match the UefiMemo= ryAllocationLib wi= th the only the use of malloc() and free() to replace the UEFI Boot Service= s calls.
    3. Host library instance naming conventio= ns.
      1. The current naming convention uses the environme= nt as a prefix(e.g. Pei, = Smm,&nb= sp;Uefi) and the services the lib inst= ance uses as a post fix.  Would it make more sense to use =E2=80=98Host= = =E2=80=99 as a prefix instead of a postfix in the lib instance names?
    4. Unicode vs ASCII strings
      1. I = see InitUnitTestFramework(),&nbs= p;CreateUnitTestSuite(), and AddTestCase= () all take Unicode strings for the labels.  I also see extra co= de to convert gEfiCallerBaseName from ASCII to Unicode to use it as a short name of a test fra= mework.  <= /span>I think it would be simpler if the parameters to these APIs we= re ASCII and the framework can convert to Unicode if needed.
    5. Will InitUnitTestFramework() and CreateUnitTestSuite() always be called in pairs?  If so, can we combine these to a single= API? 
      1. I see the SafeI= ntLib example crea= te a single framework and multiple test suites.  Perhaps we can have a = single CreateUnitTestSuite() API where Framework is an IN/OUT and if i= t is passed in as NULL, the Framework handle is created.
      2. = I see a pattern where the CreateUnitTestSuite() =E2=80=98Package=E2=80= = =99 parameter is used as a prefix to every call to AddTestCase() in t= he =E2=80=98ClassName=E2=80=99 parameter.  Can we simplify AddTestCase() so it only need to pass in the name of= the unit test case, and the framework can append Package&= nbsp; and the unit= test case name?
    6. I see the use of the =E2=80=98= Fw=E2=80=99 variable as a shorthand for =E2= =80=98Framework=E2=80=99.  Can we use something other than =E2=80=98Fw=E2=80=99.  It may be confused with =E2= = =80=98Firmware=E2=80=99.  
    7. = UnitTestPkg/Include/U= nitTestTypes.h
      1. I see several hard coded strin= g lengths. &nbs= p;Since this runs in a host environment and strings can alway= s be allocated, can the hard coded lengths be removed?&nbs= p; Update the stru= cts to use pointers to strings instead of string arrays, and the framework = can manage alloc() and free()?
      2. How are Fingerprints = used?  The idea of using as hash as a unique identifier is a good idea.=   <= /span>How is the hash calculated?  What unit test code artifacts are us= ed so developers know what parameters must be unique? = ; Can we just deci= de to use a specific hash algorithm/size and the structure can be a pointer= to an allocated buffer instead of a fixed size array to make it easy to ch= ange the algorithm/size in the future?
        3. Update all the str= ings to be ASCII?  See Unicode vs ASCII above.
      3. UNIT_T= EST_SAVE_TEST =E2=80=93 The last field is a variable sized array, so it can= be a formal field.
      4. UNIT_TEST_SAVE_CONTEXT - =E2=80=93 Th= e last field is a variable sized array, so it can be a formal field.
      5. UNIT_TEST_SAVE_HEADER =E2=80=93 Can the last 3 fields be change= d to pointers and be formal fields?
      6. Do the structures rea= lly need to be packed?  Specially with the changes suggested above?  Is the intent to potentially share data stored on disk between different = host execution environments that may have different width architectures?
    8. UnitTestPkg<= span class=3D""> =E2=80= =93 UnitTestLib.h
      1. Can you provide more context= for the APIs SaveFrameworkState(), SaveFrameworkStateAndQuit(),=  S= aveFrameworkStateAndReboot(),&= nbsp;SetFrameworkBootNextDevice().  I do not see any Load() functions, so how would a set of tests be resumed= ?  = If these do not apply to host based tests, should these be split out= to a different lib class?
    9. UnitTestPkg/Library/UnitTestLib
      1. I see an implementation of MD5.=   <= /span>We should not do our own.  We should use an approved implementati= on such as OpenSSL.
    10. Unit= TestPkg/Library/UnitT= estTerminationLibTbd
      1. Do we really need this = lib instance now?
     
    Thanks,
    =  
    Mike<= o:p class=3D"">
     
    From: devel@edk2.groups.io <devel@edk2.groups= .io> On Behalf Of Bret Ba= rkelew via Groups.Io
    Sent: Thursday, November 21, 2019 11:39 PM
    To: = devel@edk2.groups.io
    Subject:=  [edk2-devel] EDK2 Host-Based Unit Test RFC (Now with docs!)
     

    Now that CI h= as landed in edk2/master, we'd like to get the basic framework for unittest= ing staged as well. Target intercept date would be immediately after the 20= 19/11 stabilization, so we wanted to go ahead and get comments now.

    The host unittest framework consists of five primary = pieces:
    - The test library (Cmocka)
    - Support l= ibraries (Found in UnitTestPkg)
    - The test support plugins (H= ostUnitTestComilerPlugin, HostUnitTestDxeCompleteCheck, HostBasedUnitTestRu= nner)
    - The configuration in the package-based *.ci.yaml file= and package-based Test.dsc
    - The tests themselves

    We have a demo branch set up at:
    http= s://github.com/corthon/edk2-staging/tree/edk2-host-test_v2
    We also have a demo build pipeline at:
    https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId= =3D36&_a=3Dsummary

    The current demo b= ranch contains a single test in MdePkg for SafeIntLib (module file MdePkg\T= est\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.inf). This test is a= utomatically detected by the HostUnitTestComilerPlugin and run by the HostB= asedUnitTestRunner as part of the CI process.

    = A few notes about the current demo:
    1) The demo currently onl= y works on Windows build chains, but there's no reason to believe that it c= an't work equally well on Linux build chains, and are happy to work with an= yone to get it there.

    2) The demo currently ha= s four failing conditions that can be seen towards the end of MdePkg "Build= and Test" log file for this build:
    = https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=3D2813<= /a>
    "WARNING -   Test SafeInt16ToChar8 - Status
    d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSaf= eIntLib.c:302: error: Failure!
    WARNING - TestBaseSafeIntLib.e= xe Test Failed
    WARNING -   Test SafeInt32ToChar8 - = Status
    d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\T= estBaseSafeIntLib.c:638: error: Failure!
    WARNING - TestBaseSa= feIntLib.exe Test Failed
    WARNING -   Test SafeIntnT= oChar8 - Status
    d:\a\1\s\MdePkg\Test\UnitTest\Library\BaseSaf= eIntLib\TestBaseSafeIntLib.c:1051: error: Failure!
    WARNING - = TestBaseSafeIntLib.exe Test Failed
    WARNING -   Test= SafeInt64ToChar8 - Status
    d:\a\1\s\MdePkg\Test\UnitTest\Libr= ary\BaseSafeIntLib\TestBaseSafeIntLib.c:1456: error: Failure!"
    These failures seem to be legitimate and further work should be done by t= he community to decide whether the test case is correct or the library is c= orrect, but one of them needs to change.

    3) Cu= rrent demo pulls in test collateral from a fork of the edk2-test repo. This= repo is currently *very* heavy due to the history of the UEFI SCT project = and the number of binaries that it pulls down. It's possible that we (the c= ommunity) need to select a better place for this code to live. Maybe in edk= 2 primary (though it's not explicitly firmware code, so it seems unnecessar= y). Maybe in a new edk2-test2 repo or something like that.<= /o:p>

    There=E2=80=99s an RFC doc here: https://githu= b.com/corthon/edk2-staging/blob/edk2-host-test_v2/Readme-RFC.md

    And a usage guide here: http= s://github.com/corthon/edk2-staging/blob/edk2-host-test_v2/UnitTestPkg/Read= Me.md


    Once again, would love to get this into EDK2 master after stabilizat= ion, and most of this has previously been shopped around in other discussio= n threads. This is just where the rubber meets the road and is the minimal = subset of code that needs to land for folks to start contributing unittests= against the core libraries that can be run as part of any CI pass.

    Thanks!
    - Bret

    • <= /blockquote>

    --Apple-Mail=_A4C9B025-4C7B-4B4F-AB1D-E05DBF90075B--