From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (NAM11-BN8-obe.outbound.protection.outlook.com [40.92.20.30]) by mx.groups.io with SMTP id smtpd.web11.3749.1596771778482317010 for ; Thu, 06 Aug 2020 20:42:58 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@outlook.com header.s=selector1 header.b=JHrAw8Hr; spf=pass (domain: outlook.com, ip: 40.92.20.30, mailfrom: spbrogan@outlook.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lESdcvvpBS+nlnMKA71pzw5+f6cBfUvTAumVikI5H3URtOvZCmOaYXSBfiflxrXXRv27+MCmAY2U7Hy+WME9zsOby11B/HW9FvZFSQ9NWStuqR6DDOMloztFEHBxZLBi+z8QlPXNMxluJHcGSPvrRARPJRu0mQFO8dCpcYFOd+Cnn0wnMSBfCiV0s1Z6sq2QBf76oV7NK3mhAocxZnvRcTbpkg7kSxfaLPh9S5NuMv6qjZw142q9MzdPoLsAyFsadatMw1rgzfgAkMlKNrVLCnmdLwenY99dcjZP/8PKDQC33a3IOJT9VrpxlDjUuQWKJx65YA2KV1M1tXEHErtZIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rQKoOWMi+lVF2fdCdQ95zfbkfE+/rALfcdfcXHIu9YI=; b=KkJjnn6IJ/qw8ddg6LyIPkBGeYMt13TBYBP3hgWUi7kmfxLev/tP2/39bnFbY7t/wr2RIzN9lQxUHi1eTAL20c9ysFkgWugWsEWNLqvY6uwD4KmHR4h0zNuoNqqTCMnXOVqOQYNYT08aNn1CuLTFMbJGEWuqNv90SNKIUbwiC2RA+qXbYPWNnrAi6gDZBOyMfiyJAdhjqCL5TEEilbqWuzifH61iq0KP/7rQzHlD4xyVO/8+cMj/XK5ilTZDE9Q5zlUVGzeJiow9jWs/qf4M1uIGMbsBtJbf0OfAIZLHl67+KDy/6U1NSBiqePGImGCcLudUwk3y3sXRIcrGfuOhqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rQKoOWMi+lVF2fdCdQ95zfbkfE+/rALfcdfcXHIu9YI=; b=JHrAw8HrHrF0DbS+ted2qdCKBChgFovW+Xqprk0JDocFvwByCDgv2ywsrOi6SnyumcDuXo1toCUOcGX/5+ryFlETp1NcmsjNNUXHGGWKarK1CFy72HjuYirUL1SkV8NNM/4T043mD3nSjluTC/a2WXtbYyx4sL893hI+wHCIPXQGFfjtNbO2+V29twtSg1OocPFX4+8J3rYggv+voPr2P5zySoxISKslA+kt8KPtwHoTv4uMhn93djZ/R7PQNVYWnSdjubScwGjZJlzEEftkpq8ZXBugYilEGUKB+/J2XWUD+LHOIG40mUQxZW3aLOrhHHI3k8L4wukDMR4QfYjIuQ== Received: from DM6NAM11FT061.eop-nam11.prod.protection.outlook.com (2a01:111:e400:fc4d::51) by DM6NAM11HT187.eop-nam11.prod.protection.outlook.com (2a01:111:e400:fc4d::455) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3261.16; Fri, 7 Aug 2020 03:42:56 +0000 Received: from BN8PR07MB6962.namprd07.prod.outlook.com (2a01:111:e400:fc4d::49) by DM6NAM11FT061.mail.protection.outlook.com (2a01:111:e400:fc4d::394) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3261.16 via Frontend Transport; Fri, 7 Aug 2020 03:42:56 +0000 X-IncomingTopHeaderMarker: OriginalChecksum:9995259247B40AA402310558F18263585609AFF303C890BA5E11B1982CD93896;UpperCasedChecksum:A556907570179DCEFA228B68F4BEDCF78A9675A897270228ADE7CA27357F1359;SizeAsReceived:8891;Count:49 Received: from BN8PR07MB6962.namprd07.prod.outlook.com ([fe80::a8fd:820:a09d:b878]) by BN8PR07MB6962.namprd07.prod.outlook.com ([fe80::a8fd:820:a09d:b878%5]) with mapi id 15.20.3261.019; Fri, 7 Aug 2020 03:42:56 +0000 Subject: Re: [edk2-devel] [Patch] UnitTestFrameworkPkg/Readme.md: Update documentation for latest features To: devel@edk2.groups.io, michael.d.kinney@intel.com Cc: Bret Barkelew , Sean Brogan , Bret Barkelew References: <20200807025656.38956-1-michael.d.kinney@intel.com> From: "Sean" Message-ID: Date: Thu, 6 Aug 2020 20:42:53 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 In-Reply-To: <20200807025656.38956-1-michael.d.kinney@intel.com> X-ClientProxiedBy: MWHPR18CA0041.namprd18.prod.outlook.com (2603:10b6:320:31::27) To BN8PR07MB6962.namprd07.prod.outlook.com (2603:10b6:408:d6::11) Return-Path: spbrogan@outlook.com X-Microsoft-Original-Message-ID: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.2.78] (50.47.113.221) by MWHPR18CA0041.namprd18.prod.outlook.com (2603:10b6:320:31::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3261.18 via Frontend Transport; Fri, 7 Aug 2020 03:42:55 +0000 X-Microsoft-Original-Message-ID: X-TMN: [hBMw1Y7n8asb4GjtN0NcKsnXam3GFZYB] X-MS-PublicTrafficType: Email X-IncomingHeaderCount: 49 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-Correlation-Id: 5ff2be97-00a5-4062-2a9d-08d83a83f8c1 X-MS-TrafficTypeDiagnostic: DM6NAM11HT187: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Xt7ifiXQmj6K5FC9klmH91CtFmquiOSpM4SusBtZGYg0ut4j/p0nw9HWB1m58W2c5RwsOXZ0jtpzFVj4kc8NGM9P4MUcttA+CzVmYfJO+Sp8tzk36Z1ebvouyUmO7j4/iLc9qWslPmxAwwV9HL236sqdbrHda4CGG7Fja6BzoX+l5UOrubJKnOJ9U9h5get2n3N4WW72798zuwLYTM+7o3mrqUnv0/ADgRha38EB3RnuRrvDvNByiuWVGLZUuu1E X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:0;SRV:;IPV:NLI;SFV:NSPM;H:BN8PR07MB6962.namprd07.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:;DIR:OUT;SFP:1901; X-MS-Exchange-AntiSpam-MessageData: iHcsWLM6cFLHtkQ3JlQtz5haP/pC2p7+B7vSiQd6JcvF+bLzDTYgWL8ep9Otwv9K12uIxjIy23ZYAzhD7b0Er5wGsCB5vBgqfh+sfkigOqDOdpxFJ2oahO8jJhXt7eQCtqmcQJ/3/ZZeOQz8IgIHaA== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5ff2be97-00a5-4062-2a9d-08d83a83f8c1 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Aug 2020 03:42:56.4009 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT061.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6NAM11HT187 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Reviewed-by: Sean Brogan On 8/6/2020 7:56 PM, Michael D Kinney wrote: > From: Bret Barkelew > > * Add additional documentation about running tests locally > * Add a note about XML formatting > * Update readme with BaseLib and UNIT_TESTING_DEBUG > > Cc: Michael D Kinney > Cc: Sean Brogan > Signed-off-by: Bret Barkelew > Reviewed-by: Michael D Kinney > --- > UnitTestFrameworkPkg/ReadMe.md | 165 +++++++++++++++++++++++++++++++++ > 1 file changed, 165 insertions(+) > > diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.md > index 64386941cb..70656a422c 100644 > --- a/UnitTestFrameworkPkg/ReadMe.md > +++ b/UnitTestFrameworkPkg/ReadMe.md > @@ -229,6 +229,8 @@ https://api.cmocka.org/ > > ## Development > > +### Iterating on a Single Test > + > When using the EDK2 Pytools for CI testing, the host-based unit tests will be built and run on any build that includes > the `NOOPT` build target. > > @@ -239,6 +241,169 @@ the following command will build only the SafeIntLib host-based test from the Md > stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2017 -p MdePkg -t NOOPT BUILDMODULE=MdePkg/Test/UnitTest/Library/BaseSafeIntLib/TestBaseSafeIntLib.inf > ``` > > +### Hooking BaseLib > + > +Most unit test mocking can be performed by the functions provided in the UnitTestFramework libraries, but since > +BaseLib is consumed by the Framework itself, it requires different techniques to substitute parts of the > +functionality. > + > +To solve some of this, the UnitTestFramework consumes a special implementation of BaseLib for host-based tests. > +This implementation contains a [hook table](https://github.com/tianocore/edk2/blob/e188ecc8b4aed8fdd26b731d43883861f5e5e7b4/MdePkg/Test/UnitTest/Include/Library/UnitTestHostBaseLib.h#L507) > +that can be used to substitute test functionality for any of the BaseLib functions. By default, this implementation > +will use the underlying BaseLib implementation, so the unit test writer only has to supply minimal code to test a > +particular case. > + > +### Debugging the Framework Itself > + > +While most of the tests that are produced by the UnitTestFramework are easy to step through in a debugger, the Framework > +itself consumes code (mostly Cmocka) that sets its own build flags. These flags cause parts of the Framework to not > +export symbols and captures exceptions, and as such are harder to debug. We have provided a Stuart parameter to force > +symbolic debugging to be enabled. > + > +You can run a build by adding the `BLD_*_UNIT_TESTING_DEBUG=TRUE` parameter to enable this build option. > + > +```bash > +stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -p MdePkg -t NOOPT BLD_*_UNIT_TESTING_DEBUG=TRUE > +``` > + > +## Building and Running Host-Based Tests > + > +The EDK2 CI infrastructure provides a convenient way to run all host-based tests -- in the the entire tree or just > +selected packages -- and aggregate all the the reports, including highlighting any failures. This functionality is > +provided through the Stuart build system (published by EDK2-PyTools) and the `NOOPT` build target. > + > +### Building Locally > + > +First, to make sure you're working with the latest PyTools, run the following command: > + > +```bash > +# Would recommend to run this in a Python venv, but that's out of scope for this doc. > +python -m pip install --upgrade -r ./pip-requirements.txt > +``` > + > +After that, the following commands will set up the build and run the host-based tests. > + > +```bash > +# Setup repo for building > +# stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG= > +stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 > + > +# Update all binary dependencies > +# stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG= > +stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 > + > +# Build and run the tests > +# stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG= -t NOOPT [-p ] > +stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -t NOOPT -p MdePkg > +``` > + > +### Evaluating the Results > + > +In your immediate output, any build failures will be highlighted. You can see these below as "WARNING" and "ERROR" messages. > + > +```text > +(edk_env) PS C:\_uefi\edk2> stuart_ci_build -c .\.pytool\CISettings.py TOOL_CHAIN_TAG=VS2019 -t NOOPT -p MdePkg > + > +SECTION - Init SDE > +SECTION - Loading Plugins > +SECTION - Start Invocable Tool > +SECTION - Getting Environment > +SECTION - Loading plugins > +SECTION - Building MdePkg Package > +PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT -- > +WARNING - Allowing Override for key TARGET_ARCH > +PROGRESS - Start time: 2020-07-27 17:18:08.521672 > +PROGRESS - Setting up the Environment > +PROGRESS - Running Pre Build > +PROGRESS - Running Build NOOPT > +PROGRESS - Running Post Build > +SECTION - Run Host based Unit Tests > +SUBSECTION - Testing for architecture: X64 > +WARNING - TestBaseSafeIntLibHost.exe Test Failed > +WARNING - Test SafeInt8ToUint8 - UT_ASSERT_EQUAL(0x5b:5b, Result:5c) > +c:\_uefi\edk2\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:35: error: Failure! > +ERROR - Plugin Failed: Host-Based Unit Test Runner returned 1 > +CRITICAL - Post Build failed > +PROGRESS - End time: 2020-07-27 17:18:19.792313 Total time Elapsed: 0:00:11 > +ERROR - --->Test Failed: Host Unit Test Compiler Plugin NOOPT returned 1 > +ERROR - Overall Build Status: Error > +PROGRESS - There were 1 failures out of 1 attempts > +SECTION - Summary > +ERROR - Error > + > +(edk_env) PS C:\_uefi\edk2> > +``` > + > +If a test fails, you can run it manually to get more details... > + > +```text > +(edk_env) PS C:\_uefi\edk2> .\Build\MdePkg\HostTest\NOOPT_VS2019\X64\TestBaseSafeIntLibHost.exe > + > +Int Safe Lib Unit Test Application v0.1 > +--------------------------------------------------------- > +------------ RUNNING ALL TEST SUITES -------------- > +--------------------------------------------------------- > +--------------------------------------------------------- > +RUNNING TEST SUITE: Int Safe Conversions Test Suite > +--------------------------------------------------------- > +[==========] Running 71 test(s). > +[ RUN ] Test SafeInt8ToUint8 > +[ ERROR ] --- UT_ASSERT_EQUAL(0x5b:5b, Result:5c) > +[ LINE ] --- c:\_uefi\edk2\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:35: error: Failure! > +[ FAILED ] Test SafeInt8ToUint8 > +[ RUN ] Test SafeInt8ToUint16 > +[ OK ] Test SafeInt8ToUint16 > +[ RUN ] Test SafeInt8ToUint32 > +[ OK ] Test SafeInt8ToUint32 > +[ RUN ] Test SafeInt8ToUintn > +[ OK ] Test SafeInt8ToUintn > +... > +``` > + > +You can also, if you are so inclined, read the output from the exact instance of the test that was run during > +`stuart_ci_build`. The ouput file can be found on a path that looks like: > + > +`Build//HostTest//...result.xml` > + > +A sample of this output looks like: > + > +```xml > + > + > + > + > + > + +c:\_uefi\mu_ci\mu_basecore\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntLib.c:35: error: Failure!]]> > + > + > + > + > + > + > + > +``` > + > +### XML Reporting Mode > + > +Since these applications are built using the CMocka framework, they can also use the following env variables to output > +in a structured XML rather than text: > + > +```text > +CMOCKA_MESSAGE_OUTPUT=xml > +CMOCKA_XML_FILE= > +``` > + > +This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view. > + > +### Important Note > + > +This works on both Windows and Linux, but is currently limited to x64 architectures. Working on getting others, but we > +also welcome contributions. > + > ## Known Limitations > > ### PEI, DXE, SMM >