From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) by mx.groups.io with SMTP id smtpd.web12.4704.1590633605357855745 for ; Wed, 27 May 2020 19:40:05 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@corthon-com.20150623.gappssmtp.com header.s=20150623 header.b=qCFrnNEN; spf=none, err=permanent DNS error (domain: corthon.com, ip: 209.85.216.65, mailfrom: bret@corthon.com) Received: by mail-pj1-f65.google.com with SMTP id s69so2415503pjb.4 for ; Wed, 27 May 2020 19:40:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corthon-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KGA+IOYt0BNf6YcJfXPTS6Cx1lhGugKdmJLfK6bXdOM=; b=qCFrnNENQ0FMJQJvfz3SmLqnDk0/VdBzsRvFGezjBapUYmmaFLnK1gOwHq0nuON4qq j8AtRSVrXmEY4gKB/I+YM9HnL6lFz9OR8Mbzdgor8WqDujCN0WhaJR/A+b+ffUEDtfok mAYIsP+0ogpNV06Nmfw+QQGnpeBIaeKXdui42E+WXXNHXnYwxhd0M3hFDxaA/jsI+8Ga siR4SqoSqt+aMwazraMHVdlxUs13MLmXGrfb5WMp1spn7LmPTntBPG4IK9L5mI+oT2vO bQNIDQPib3vZeM4d+OdaPcEc0ptd8Fjh/nSUQPoAVt4KSKZGBMxyKIJbW7yz9Uz0tAhz 2x8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KGA+IOYt0BNf6YcJfXPTS6Cx1lhGugKdmJLfK6bXdOM=; b=rq/+upAfi8PtlYNNr5TD/itixz9WQ7d86ltpBSUxCkXGx8GWvKqIYeVD6wrbOQT1/T ZtTftNDjltzm8eZQi56f0W5hZ7tHA6f5BZJhWTzPnaILSSsR2gvxkZ8ApdpczfjuhO9h h197r3h70Tf0Y7WCs4ExI80bMpX9RVkUBkln5cJhNHatswPqjyfVkpsSnMknV37nPNos f/thiOqur+bgx/0fRTwWDUEyTxVZdATMtoSkoCfXzAI/iLF8mWL+CdwpUeRbBR3fFKus F9gbozEhG2V391fjliAeyl495POAAIxxaHsBGeFAtLUnYDyVs37p3fQXUh7cK32qbdy8 d/7g== X-Gm-Message-State: AOAM532hyYitaE6874vl1VjpXPXRSGBLcoNG7G2Wk/gRQo1OfykQfyDg fcLuSVwH2HIG8+VULwl9WOmoae9uabA= X-Google-Smtp-Source: ABdhPJw6fJ9NNxg7qQeU9mkUmfvo62myCKOEf8vQzt4/MQmisuxUHm768sts52DJ85JmvyO94pJotg== X-Received: by 2002:a17:90a:ad08:: with SMTP id r8mr1478257pjq.154.1590633604259; Wed, 27 May 2020 19:40:04 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([71.212.144.72]) by smtp.gmail.com with ESMTPSA id a10sm2998168pgv.72.2020.05.27.19.40.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 May 2020 19:40:03 -0700 (PDT) From: Bret Barkelew X-Google-Original-From: Bret Barkelew To: devel@edk2.groups.io Cc: Michael D Kinney Subject: [PATCH v1 1/1] UnitTestFrameworkPkg: Add info to readme about working with UnitTests Date: Wed, 27 May 2020 19:38:23 -0700 Message-Id: <20200528023823.1776-2-brbarkel@microsoft.com> X-Mailer: git-send-email 2.26.2.windows.1.8.g01c50adf56.20200515075929 In-Reply-To: <20200528023823.1776-1-brbarkel@microsoft.com> References: <20200528023823.1776-1-brbarkel@microsoft.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Cc: Michael D Kinney Signed-off-by: Bret Barkelew --- .pytool/Readme.md | 125 +++++++++++++++----- UnitTestFrameworkPkg/ReadMe.md | 82 ++++++++++++- 2 files changed, 172 insertions(+), 35 deletions(-) diff --git a/.pytool/Readme.md b/.pytool/Readme.md index c7dce3b64ca0..c401dba18fbf 100644 --- a/.pytool/Readme.md +++ b/.pytool/Readme.md @@ -2,31 +2,32 @@ =0D ## Basic Status=0D =0D -| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/AR= M/AARCH64) | Known Issues |=0D -| :---- | :----- | :---- = | :--- |=0D -| ArmPkg |=0D -| ArmPlatformPkg |=0D -| ArmVirtPkg | SEE PACKAGE README | SEE PACKAGE README |=0D -| CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch= ecking in audit mode=0D -| DynamicTablesPkg |=0D -| EmbeddedPkg |=0D -| EmulatorPkg | SEE PACKAGE README | SEE PACKAGE README | Spell ch= ecking in audit mode=0D -| FatPkg | :heavy_check_mark: | :heavy_check_mark: |=0D -| FmpDevicePkg | :heavy_check_mark: | :heavy_check_mark: |=0D -| IntelFsp2Pkg |=0D -| IntelFsp2WrapperPkg |=0D -| MdeModulePkg | :heavy_check_mark: | :heavy_check_mark: | DxeIpl d= ependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit mo= de=0D -| MdePkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch= ecking in audit mode=0D -| NetworkPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch= ecking in audit mode=0D -| OvmfPkg | SEE PACKAGE README | SEE PACKAGE README | Spell ch= ecking in audit mode=0D -| PcAtChipsetPkg | :heavy_check_mark: | :heavy_check_mark: |=0D -| SecurityPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch= ecking in audit mode=0D -| ShellPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch= ecking in audit mode, 3 modules are not being built by DSC=0D -| SignedCapsulePkg |=0D -| SourceLevelDebugPkg |=0D -| StandaloneMmPkg |=0D -| UefiCpuPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch= ecking in audit mode, 2 binary modules not being built by DSC=0D -| UefiPayloadPkg |=0D +| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/A= RM/AARCH64) | Known Issues |=0D +| :---- | :----- | :---- = | :--- |=0D +| ArmPkg |=0D +| ArmPlatformPkg |=0D +| ArmVirtPkg | SEE PACKAGE README | SEE PACKAGE README |=0D +| CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c= hecking in audit mode=0D +| DynamicTablesPkg |=0D +| EmbeddedPkg |=0D +| EmulatorPkg | SEE PACKAGE README | SEE PACKAGE README | Spell c= hecking in audit mode=0D +| FatPkg | :heavy_check_mark: | :heavy_check_mark: |=0D +| FmpDevicePkg | :heavy_check_mark: | :heavy_check_mark: |=0D +| IntelFsp2Pkg |=0D +| IntelFsp2WrapperPkg |=0D +| MdeModulePkg | :heavy_check_mark: | :heavy_check_mark: | DxeIpl = dependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit m= ode=0D +| MdePkg | :heavy_check_mark: | :heavy_check_mark: | Spell c= hecking in audit mode=0D +| NetworkPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c= hecking in audit mode=0D +| OvmfPkg | SEE PACKAGE README | SEE PACKAGE README | Spell c= hecking in audit mode=0D +| PcAtChipsetPkg | :heavy_check_mark: | :heavy_check_mark: |=0D +| SecurityPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c= hecking in audit mode=0D +| ShellPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c= hecking in audit mode, 3 modules are not being built by DSC=0D +| SignedCapsulePkg |=0D +| SourceLevelDebugPkg |=0D +| StandaloneMmPkg |=0D +| UefiCpuPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c= hecking in audit mode, 2 binary modules not being built by DSC=0D +| UefiPayloadPkg |=0D +| UnitTestFrameworkPkg | :heavy_check_mark: | :heavy_check_mark: |=0D =0D For more detailed status look at the test results of the latest CI run on = the=0D repo readme.=0D @@ -88,7 +89,7 @@ that a few steps should be followed. Details of EDKII To= ols can be found in the * VS 2017 or VS 2019=0D * Windows SDK (for rc)=0D * Windows WDK (for capsules)=0D - * Ubuntu 16.04=0D + * Ubuntu 18.04 or Fedora=0D * GCC5=0D * Easy to add more but this is the current state=0D 2. Python 3.7.x or newer on path=0D @@ -137,11 +138,31 @@ location makes more sense for the community. =0D ### Module Inclusion Test - DscCompleteCheck=0D =0D -This test scans all available modules (via INF files) and compares them to= the=0D -package-level DSC file for the package each module is contained within. Th= e test=0D -considers it an error if any module does not appear in the `Components` se= ction=0D -of at least one package-level DSC (indicating that it would not be built i= f the=0D -package were built).=0D +This scans all INF files from a package and confirms they are=0D +listed in the package level DSC file. The test considers it an error if an= y INF=0D +does not appear in the `Components` section of the package-level DSC (indi= cating=0D +that it would not be built if the package were built). This is critical be= cause=0D +much of the CI infrastructure assumes that all modules will be listed in t= he DSC=0D +and compiled.=0D +=0D +This test will ignore INFs in the following cases:=0D +=0D +1. When `MODULE_TYPE` =3D `HOST_APPLICATION`=0D +2. When a Library instance **only** supports the `HOST_APPLICATION` enviro= nment=0D +=0D +### Host Module Inclusion Test - HostUnitTestDscCompleteCheck=0D +=0D +This test scans all INF files from a package for those related to host=0D +based unit tests and confirms they are listed in the unit test DSC file fo= r the package.=0D +The test considers it an error if any INF meeting the requirements does no= t appear=0D +in the `Components` section of the unit test DSC. This is critical because= =0D +much of the CI infrastructure assumes that modules will be listed in the = DSC=0D +and compiled.=0D +=0D +This test will only require INFs in the following cases:=0D +=0D +1. When `MODULE_TYPE` =3D `HOST_APPLICATION`=0D +2. When a Library instance explicitly supports the `HOST_APPLICATION` envi= ronment=0D =0D ### Code Compilation Test - CompilerPlugin=0D =0D @@ -150,6 +171,46 @@ all package-level DSCs were built, the Code Compilatio= n Test simply runs through and builds every package-level DSC on every toolchain and for every archit= ecture=0D that is supported. Any module that fails to build is considered an error.= =0D =0D +### Host Unit Test Compilation and Run Test - HostUnitTestCompilerPlugin=0D +=0D +A test that compiles the dsc for host based unit test apps.=0D +On Windows this will also enable a build plugin to execute that will run t= he unit tests and verify the results.=0D +=0D +These tools will be invoked on any CI=0D +pass that includes the NOOPT target. In order for these tools to do their = job,=0D +the package and tests must be configured in a particular way...=0D +=0D +#### Including Host-Based Tests in the Package YAML=0D +=0D +For example, looking at the `MdeModulePkg.ci.yaml` config file, there are = two=0D +config options that control HostBased test behavior:=0D +=0D +```json=0D + ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin=0D + "HostUnitTestCompilerPlugin": {=0D + "DscPath": "Test/MdeModulePkgHostTest.dsc"=0D + },=0D +```=0D +=0D +This option tell the test builder to run. The test builder needs to know w= hich=0D +modules in this package are host-based tests, so that DSC path is provided= .=0D +=0D +#### Configuring the HostBased DSC=0D +=0D +The HostBased DSC for `MdeModulePkg` is located at=0D +`MdeModulePkg/Test/MdeModulePkgHostTest.dsc`.=0D +=0D +To add automated host-based unit test building to a new package, create a= =0D +similar DSC. The new DSC should make sure to have the `NOOPT` BUILD_TARGET= =0D +and should include the line:=0D +=0D +```=0D +!include UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc=0D +```=0D +=0D +All of the modules that are included in the `Components` section of this=0D +DSC should be of type HOST_APPLICATION.=0D +=0D ### GUID Uniqueness Test - GuidCheck=0D =0D This test works on the collection of all packages rather than an individua= l=0D @@ -207,6 +268,8 @@ few standard scopes. | global-nix | edk2_invocable++ | Runn= ing on Linux based OS |=0D | edk2-build | | This= indicates that an invocable is building EDK2 based UEFI code |=0D | cibuild | set in .pytool/CISettings.py | Sugg= ested target for edk2 continuous integration builds. Tools used for CiBuil= ds can use this scope. Example: asl compiler |=0D +| host-based-test | set in .pytool/CISettings.py | Turn= s on the host based tests and plugin |=0D +| host-test-win | set in .pytool/CISettings.py | Enab= les the host based test runner for Windows |=0D =0D ## Future investments=0D =0D diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.md index 7296f0a45c5f..64386941cb4b 100644 --- a/UnitTestFrameworkPkg/ReadMe.md +++ b/UnitTestFrameworkPkg/ReadMe.md @@ -58,7 +58,7 @@ header for the `UnitTestLib` is located in `MdePkg`, so y= ou shouldn't need to de packages. As long as your DSC file knows where to find the lib implementat= ion that you want to use,=0D you should be good to go.=0D =0D -See this example in 'SampleUnitTestApp.inf'...=0D +See this example in 'SampleUnitTestUefiShell.inf'...=0D =0D ```=0D [Packages]=0D @@ -72,6 +72,14 @@ See this example in 'SampleUnitTestApp.inf'... PrintLib=0D ```=0D =0D +Also, if you want you test to automatically be picked up by the Test Runne= r plugin, you will need=0D +to make sure that the module `BASE_NAME` contains the word `Test`...=0D +=0D +```=0D +[Defines]=0D + BASE_NAME =3D SampleUnitTestUefiShell=0D +```=0D +=0D ### Requirements - Code=0D =0D Not to state the obvious, but let's make sure we have the following includ= e before getting too far along...=0D @@ -221,9 +229,11 @@ https://api.cmocka.org/ =0D ## Development=0D =0D -When using the EDK2 Pytools for CI testing, the host-based unit tests will= be built and run on any build that includes the `NOOPT` build target.=0D +When using the EDK2 Pytools for CI testing, the host-based unit tests will= be built and run on any build that includes=0D +the `NOOPT` build target.=0D =0D -If you are trying to iterate on a single test, a convenient pattern is to = build only that test module. For example, the following command will build = only the SafeIntLib host-based test from the MdePkg...=0D +If you are trying to iterate on a single test, a convenient pattern is to = build only that test module. For example,=0D +the following command will build only the SafeIntLib host-based test from = the MdePkg...=0D =0D ```bash=0D stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2017 -p MdePkg= -t NOOPT BUILDMODULE=3DMdePkg/Test/UnitTest/Library/BaseSafeIntLib/TestBas= eSafeIntLib.inf=0D @@ -250,8 +260,72 @@ reporting lib. This isn't currently possible with host= -based. Only the assertion =0D We will continue trying to make these as similar as possible.=0D =0D +## Unit Test Location/Layout Rules=0D +=0D +Code/Test | Location=0D +--------- | --------=0D +Host-Based Unit Tests for a Library/Protocol/PPI/GUID Interface | If wha= t's being tested is an interface (e.g. a library with a public header file,= like DebugLib), the test should be scoped to the parent package.
Examp= le: `MdePkg/Test/UnitTest/[Library/Protocol/Ppi/Guid]/`

A real-wor= ld example of this is the BaseSafeIntLib test in MdePkg.
`MdePkg/Test/U= nitTest/Library/BaseSafeIntLib/TestBaseSafeIntLibHost.inf`=0D +Host-Based Unit Tests for a Library/Driver (PEI/DXE/SMM) implementation = | If what's being tested is a specific implementation (e.g. BaseDebugLibSer= ialPort for DebugLib), the test should be scoped to the implementation dire= ctory itself, in a UnitTest subdirectory.

Module Example: `MdeModu= lePkg/Universal/EsrtFmpDxe/UnitTest/`
Library Example: `MdePkg/Library/= BaseMemoryLib/UnitTest/`=0D +Host-Based Tests for a Functionality or Feature | If you're writing a fu= nctional test that operates at the module level (i.e. if it's more than a s= ingle file or library), the test should be located in the package-level Tes= ts directory under the HostFuncTest subdirectory.
For example, if you w= ere writing a test for the entire FMP Device Framework, you might put your = test in:
`FmpDevicePkg/Test/HostFuncTest/FmpDeviceFramework`

I= f the feature spans multiple packages, it's location should be determined b= y the package owners related to the feature.=0D +Non-Host-Based (PEI/DXE/SMM/Shell) Tests for a Functionality or Feature = | Similar to Host-Based, if the feature is in one package, should be locate= d in the `*Pkg/Test/[Shell/Dxe/Smm/Pei]Test` directory.

If the fea= ture spans multiple packages, it's location should be determined by the pac= kage owners related to the feature.

USAGE EXAMPLES
PEI Example= : MP_SERVICE_PPI. Or check MTRR configuration in a notification function. SMM Example: a test in a protocol callback function. (It is different w= ith the solution that SmmAgent+ShellApp)
DXE Example: a test in a UEFI = event call back to check SPI/SMRAM status.
Shell Example: the SMM han= dler audit test has a shell-based app that interacts with an SMM handler to= get information. The SMM paging audit test gathers information about both = DXE and SMM. And the SMM paging functional test actually forces errors into= SMM via a DXE driver.=0D +=0D +### Example Directory Tree=0D +=0D +```text=0D +Pkg/=0D + ComponentY/=0D + ComponentY.inf=0D + ComponentY.c=0D + UnitTest/=0D + ComponentYHostUnitTest.inf # Host-Based Test for Driver Module= =0D + ComponentYUnitTest.c=0D +=0D + Library/=0D + GeneralPurposeLibBase/=0D + ...=0D +=0D + GeneralPurposeLibSerial/=0D + ...=0D +=0D + SpecificLibDxe/=0D + SpecificLibDxe.c=0D + SpecificLibDxe.inf=0D + UnitTest/ # Host-Based Test for Specific Librar= y Implementation=0D + SpecificLibDxeHostUnitTest.c=0D + SpecificLibDxeHostUnitTest.inf=0D + Test/=0D + HostTest.dsc # Host-Based Test Apps=0D + UnitTest/=0D + InterfaceX=0D + InterfaceXHostUnitTest.inf # Host-Based App (should be in Test/= HostTest.dsc)=0D + InterfaceXPeiUnitTest.inf # PEIM Target-Based Test (if applica= ble)=0D + InterfaceXDxeUnitTest.inf # DXE Target-Based Test (if applicab= le)=0D + InterfaceXSmmUnitTest.inf # SMM Target-Based Test (if applicab= le)=0D + InterfaceXShellUnitTest.inf # Shell App Target-Based Test (if ap= plicable)=0D + InterfaceXUnitTest.c # Test Logic=0D +=0D + GeneralPurposeLib/ # Host-Based Test for any implementa= tion of GeneralPurposeLib=0D + GeneralPurposeLibTest.c=0D + GeneralPurposeLibHostUnitTest.inf=0D +=0D + Pkg.dsc # Standard Modules and any Target-Based Test A= pps (including in Test/)=0D +=0D +```=0D +=0D +### Future Locations in Consideration=0D +=0D +We don't know if these types will exist or be applicable yet, but if you w= rite a support library or module that matches the following, please make su= re they live in the correct place.=0D +=0D +Code/Test | Location=0D +--------- | --------=0D +Host-Based Library Implementations | Host-Based Implementa= tions of common libraries (eg. MemoryAllocationLibHost) should live in the = same package that declares the library interface in its .DEC file in the `*= Pkg/HostLibrary` directory. Should have 'Host' in the name.=0D +Host-Based Mocks and Stubs | Mock and Stub libraries should live in the `= UefiTestFrameworkPkg/StubLibrary` with either 'Mock' or 'Stub' in the libra= ry name.=0D +=0D +### If still in doubt...=0D +=0D +Hop on GitHub and ask @corthon, @mdkinney, or @spbrogan. ;)=0D +=0D ## Copyright=0D =0D Copyright (c) Microsoft Corporation.=0D SPDX-License-Identifier: BSD-2-Clause-Patent=0D -=0D --=20 2.25.1.windows.1