From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 8295A941D86 for ; Tue, 13 Feb 2024 18:20:55 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=NfBhghty1Lk91rhY9A3tiRahkByRSGchR7P7jsR0Yoc=; c=relaxed/simple; d=groups.io; h=Message-ID:Date:MIME-Version:User-Agent:Subject:To:References:From:In-Reply-To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1707848454; v=1; b=VQvDBInkashZ3oBrcr7JRhce9KN1Ep3nHd5IhHpHrofB6a+vogbBXezDoCRLyC3MIHO/SW+L ldGLps7Gh7i4CeXaTZDj1yzQzSCVq+9th0aBgnlmYe+IOfwMzhJk61Kwv2QWhlp0SR34397XPJe pUe1ZnudJlbf9KiN0mNrJr5k= X-Received: by 127.0.0.2 with SMTP id x2joYY7687511xkJwS3B355q; Tue, 13 Feb 2024 10:20:54 -0800 X-Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by mx.groups.io with SMTP id smtpd.web11.20496.1707848453358505254 for ; Tue, 13 Feb 2024 10:20:53 -0800 X-Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 41DGoVe5032458; Tue, 13 Feb 2024 18:20:53 GMT X-Received: from nasanppmta03.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3w7hewbkew-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Feb 2024 18:20:52 +0000 (GMT) X-Received: from nasanex01c.na.qualcomm.com (nasanex01c.na.qualcomm.com [10.45.79.139]) by NASANPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 41DIKqHp011852 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Feb 2024 18:20:52 GMT X-Received: from [10.111.132.144] (10.80.80.8) by nasanex01c.na.qualcomm.com (10.45.79.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Tue, 13 Feb 2024 10:20:50 -0800 Message-ID: <63110078-f84a-412a-9bd1-fd3940732ee9@quicinc.com> Date: Tue, 13 Feb 2024 18:20:33 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [edk2-devel] [edk2-stable202402][Patch v4 0/7] EDK II CI misses UnitTestFrameworkPkg failures To: , References: <20240213172612.636-1-michael.d.kinney@intel.com> From: "Leif Lindholm" In-Reply-To: <20240213172612.636-1-michael.d.kinney@intel.com> X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nasanex01c.na.qualcomm.com (10.45.79.139) X-QCInternal: smtphost X-Proofpoint-GUID: o-5Bk_sBkMxCBQO_O_feISt3wSY6OBav X-Proofpoint-ORIG-GUID: o-5Bk_sBkMxCBQO_O_feISt3wSY6OBav Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,quic_llindhol@quicinc.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: AD7dP7Mk11MfHlRdRJCtqUa4x7686176AA= Content-Language: en-GB Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=VQvDBInk; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=quicinc.com (policy=none); spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io This series look safe, as in if the only change that affects the output=20 image was buggy, it would blow up spectacularly at build time. I'm happy for it to go in the stable tag. On 2024-02-13 17:26, Michael D Kinney wrote: > The main bug is that EDK II CI can miss a unit test failure when > a unit test generates an exception. Addressing this one issue > required the following chain of changes to the UnitTestFrameworkPkg > that are included in this patch series. >=20 > New in v4 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > * Moved GoogleTestLib.inf /EHsc -> EHs from Patch #2 ->Patch #3 > so the /EHsc change is not mixed with /MT changes in Patch #2. > * Fixed typo in Patch #1 commit message >=20 > New in V3 > =3D=3D=3D=3D=3D=3D=3D=3D=3D > * Add EXPECT_THROW_MESSAGE() and ASSERT_THROW_MESSAGE() to > GoogleTestLib.h to simplify unit test case statements testing for > an ASSERT() for a specific trigger expressiom. Uncrustify format > of EXPECT_THAT() with [](){stmt} make code harder to read. Adding > macros makes the source content in unit test cases match the > source style of EXPECT_THROW() statements. > * Update GoogTest samples to use EXPECT_THROW_MESSAGE() instead of > EXPECT_THAT(). >=20 > New in V2 > =3D=3D=3D=3D=3D=3D=3D=3D=3D > * Include Throws() APIs in GoogleTestLib.h > * Update GoogleTest samples to demonstrate use of EXPECT_THROW() > and EXPECT_THAT() to test for more specific ASSERT() conditions > including the ability to verify the specific ASSERT() expression > that was triggered. > =3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > * For host-base VX20xx builds, use release builds by default to > make sure unexpected exceptions are caught as a test failure. > Using debug libraries may generate a popup window to prompt > running a debugger, which is value for debug, but that mode > does not generate an error in CI. > * Update UNIT_TEST_DEBUG to enable use of VS20xx debug libraries > to enable popup windows to run debugger. > * Remove DebugLib.h internal macro name collision with windows.h > that was introduced when release libraries are used by default. > * Update MSFT and GCC flags to consistently enable structured > exception handling so Google Test host-based unit tests that > check for expected ASSERT() conditions can be properly detected. > The additional benefit of this bug fix is that catching thrown > exceptions is much faster than EXPECT_DEATH() tests. The gtest > sample unit tests are updated to use ASSERT_ANY_THROW() instead > of EXPECT_DEATH(). > * In order to catch C++ exceptions for google test, the > UnitTestDebugAssertLib has to be split into a target version > and a host version and had to be able to detect the difference > between cmock host-based tests and gtest host-based tests. This > required minor changes to the UnitTestLib to be able to use the > UnitTestLib GetFrameworkHandle() API to make that determination. > It is NULL for gtest and non-NULL for cmocka. > * In order to test the unit test failing conditions, unit tests > the fail on purpose and unit tests that generate exceptions are > added. These are not added to CI because that would cause CI to > always fail. Instead, they can be run to make sure that reports > generated provide the right level of detail for a developer to > quickly identify the source of the unit test failure or the > source of the exception. >=20 > Cc: Michael Kubacki mikuback@linux.microsoft.com > Cc: Sean Brogan sean.brogan@microsoft.com > Cc: Liming Gao gaoliming@byosoft.com.cn > Cc: Zhiguang Liu zhiguang.liu@intel.com > Cc: Andrew Fish afish@apple.com > Cc: Leif Lindholm quic_llindhol@quicinc.com > Signed-off-by: Michael D Kinney michael.d.kinney@intel.com >=20 > Michael D Kinney (7): > MdePkg/Include: Rename _DEBUG() to address name collision > UnitTestFrameworkPkg: MSFT CC_FLAGS add /MT to for host builds > UnitTestFrameworkPkg: Expand host-based exception handling and gcov > UnitTestFrameworkPkg/UnitTestLib: GetActiveFrameworkHandle() no > ASSERT() > UnitTestFrameworkPkg/UnitTestDebugAssertLib: Add GoogleTest support > UnitTestFrameworkPkg/SampleGoogleTest: Use EXPECT_ANY_THROW() > UnitTestFrameworkPkg: Add DSC and host tests that always fail >=20 > MdePkg/Include/Library/DebugLib.h | 6 +- > .../Include/Library/GoogleTestLib.h | 20 + > .../Library/GoogleTestLib/GoogleTestLib.inf | 6 +- > .../UnitTestDebugAssertLibHost.cpp | 63 ++ > .../UnitTestDebugAssertLibHost.inf | 36 + > .../UnitTestDebugAssertLibHost.uni | 11 + > .../Library/UnitTestLib/Assert.c | 4 + > .../Library/UnitTestLib/Log.c | 4 + > .../Library/UnitTestLib/RunTests.c | 1 - > .../Library/UnitTestLib/RunTestsCmocka.c | 1 - > .../Library/UnitTestLib/UnitTestLib.c | 4 + > UnitTestFrameworkPkg/ReadMe.md | 2 +- > .../SampleGoogleTest/SampleGoogleTest.cpp | 36 +- > .../SampleGoogleTestExpectFail.cpp} | 125 ++- > .../SampleGoogleTestHostExpectFail.inf | 36 + > .../SampleGoogleTestGenerateException.cpp | 54 ++ > .../SampleGoogleTestHostGenerateException.inf | 39 + > .../SampleUnitTestDxeExpectFail.inf | 41 + > .../SampleUnitTestExpectFail.c | 861 ++++++++++++++++++ > .../SampleUnitTestHostExpectFail.inf | 35 + > .../SampleUnitTestPeiExpectFail.inf | 41 + > .../SampleUnitTestSmmExpectFail.inf | 42 + > .../SampleUnitTestUefiShellExpectFail.inf | 38 + > .../SampleUnitTestDxeGenerateException.inf | 43 + > .../SampleUnitTestGenerateException.c | 204 +++++ > .../SampleUnitTestHostGenerateException.inf | 37 + > .../SampleUnitTestPeiGenerateException.inf | 43 + > .../SampleUnitTestSmmGenerateException.inf | 44 + > ...mpleUnitTestUefiShellGenerateException.inf | 40 + > .../Test/UnitTestFrameworkPkgHostTest.dsc | 1 + > ...UnitTestFrameworkPkgHostTestExpectFail.dsc | 44 + > .../UnitTestFrameworkPkg.ci.yaml | 18 +- > UnitTestFrameworkPkg/UnitTestFrameworkPkg.dsc | 26 + > .../UnitTestFrameworkPkgHost.dsc.inc | 11 +- > .../UnitTestFrameworkPkgTarget.dsc.inc | 14 + > 35 files changed, 1977 insertions(+), 54 deletions(-) > create mode 100644 UnitTestFrameworkPkg/Library/UnitTestDebugAssertLib/= UnitTestDebugAssertLibHost.cpp > create mode 100644 UnitTestFrameworkPkg/Library/UnitTestDebugAssertLib/= UnitTestDebugAssertLibHost.inf > create mode 100644 UnitTestFrameworkPkg/Library/UnitTestDebugAssertLib/= UnitTestDebugAssertLibHost.uni > copy UnitTestFrameworkPkg/Test/GoogleTest/Sample/{SampleGoogleTest/Samp= leGoogleTest.cpp =3D> SampleGoogleTestExpectFail/SampleGoogleTestExpectFail= .cpp} (53%) > create mode 100644 UnitTestFrameworkPkg/Test/GoogleTest/Sample/SampleGo= ogleTestExpectFail/SampleGoogleTestHostExpectFail.inf > create mode 100644 UnitTestFrameworkPkg/Test/GoogleTest/Sample/SampleGo= ogleTestGenerateException/SampleGoogleTestGenerateException.cpp > create mode 100644 UnitTestFrameworkPkg/Test/GoogleTest/Sample/SampleGo= ogleTestGenerateException/SampleGoogleTestHostGenerateException.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestExpectFail/SampleUnitTestDxeExpectFail.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestExpectFail/SampleUnitTestExpectFail.c > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestExpectFail/SampleUnitTestHostExpectFail.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestExpectFail/SampleUnitTestPeiExpectFail.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestExpectFail/SampleUnitTestSmmExpectFail.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestExpectFail/SampleUnitTestUefiShellExpectFail.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestGenerateException/SampleUnitTestDxeGenerateException.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestGenerateException/SampleUnitTestGenerateException.c > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestGenerateException/SampleUnitTestHostGenerateException.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestGenerateException/SampleUnitTestPeiGenerateException.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestGenerateException/SampleUnitTestSmmGenerateException.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnit= TestGenerateException/SampleUnitTestUefiShellGenerateException.inf > create mode 100644 UnitTestFrameworkPkg/Test/UnitTestFrameworkPkgHostTe= stExpectFail.dsc >=20 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#115403): https://edk2.groups.io/g/devel/message/115403 Mute This Topic: https://groups.io/mt/104336757/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-