From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id E97C3740039 for ; Wed, 7 Feb 2024 03:41:20 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=jm+dL//yvrkxI9dyb4zt17r2vHg0mlMVDmKbxjIA8Ys=; c=relaxed/simple; d=groups.io; h=From:To:Subject:Date:Message-Id:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1707277279; v=1; b=ZwvFRzmPeFpZ4rcISrOHiQoHt7rM0XxoCQTeFYVMTUFjmJ61mBMHI60Eeipn9Ncf/TnwDz1P mB8Z9B7RhcIR2uTCXUUfSjSa3mf93hnCbdNes6X/dD6lEBdW+cL2RWrzKA1iF9M4U7J4+4G6eKB KxaoSnmxJN8SdMCSrQepXTm4= X-Received: by 127.0.0.2 with SMTP id y7XpYY7687511xUNBs0l7uGH; Tue, 06 Feb 2024 19:41:19 -0800 X-Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by mx.groups.io with SMTP id smtpd.web10.13160.1707277278891658966 for ; Tue, 06 Feb 2024 19:41:18 -0800 X-IronPort-AV: E=McAfee;i="6600,9927,10976"; a="12258657" X-IronPort-AV: E=Sophos;i="6.05,250,1701158400"; d="scan'208";a="12258657" X-Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2024 19:41:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,250,1701158400"; d="scan'208";a="1568981" X-Received: from mdkinney-mobl.amr.corp.intel.com ([10.209.93.91]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2024 19:41:18 -0800 From: "Michael D Kinney" To: devel@edk2.groups.io Subject: [edk2-devel] [edk2-stable202402][Patch 0/7] EDK II CI misses UnitTestFrameworkPkg failures Date: Tue, 6 Feb 2024 19:40:59 -0800 Message-Id: <20240207034106.1860-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: XgsbDGdeXTPDfDeCGI1fnTRQx7686176AA= Content-Transfer-Encoding: 8bit X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=ZwvFRzmP; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=intel.com (policy=none); spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io The main bug is that EDK II CI can miss a unit test failure when a unit test generates an exception. Addressing this one issue required the following chain of changes to the UnitTestFrameworkPkg that are included in this patch series. * For host-base VX20xx builds, use release builds by default to make sure unexpected exceptions are caught as a test failure. Using debug libraries may generate a popup window to prompt running a debugger, which is value for debug, but that mode does not generate an error in CI. * Update UNIT_TEST_DEBUG to enable use of VS20xx debug libraries to enable popup windows to run debugger. * Remove DebugLib.h internal macro name collision with windows.h that was introduced when release libraries are used by default. * Update MSFT and GCC flags to consistently enable structured exception handling so Google Test host-based unit tests that check for expected ASSERT() conditions can be properly detected. The additional benefit of this bug fix is that catching thrown exceptions is much faster than EXPECT_DEATH() tests. The gtest sample unit tests are updated to use ASSERT_ANY_THROW() instead of EXPECT_DEATH(). * In order to catch C++ exceptions for google test, the UnitTestDebugAssertLib has to be split into a target version and a host version and had to be able to detect the difference between cmock host-based tests and gtest host-based tests. This required minor changes to the UnitTestLib to be able to use the UnitTestLib GetFrameworkHandle() API to make that determination. It is NULL for gtest and non-NULL for cmocka. * In order to test the unit test failing conditions, unit tests the fail on purpose and unit tests that generate exceptions are added. These are not added to CI because that would cause CI to always fail. Instead, they can be run to make sure that reports generated provide the right level of detail for a developer to quickly identify the source of the unit test failure or the source of the exception. Cc: Michael Kubacki mikuback@linux.microsoft.com Cc: Sean Brogan sean.brogan@microsoft.com Cc: Liming Gao gaoliming@byosoft.com.cn Cc: Zhiguang Liu zhiguang.liu@intel.com Cc: Andrew Fish afish@apple.com Cc: Leif Lindholm quic_llindhol@quicinc.com Signed-off-by: Michael D Kinney michael.d.kinney@intel.com Michael D Kinney (7): MdePkg/Include: Rename _DEBUG() to address name collision UnitTestFrameworkPkg: MSFT CC_FLAGS add /MT to for host builds UnitTestFrameworkPkg: Expand host-based exception handling and gcov UnitTestFrameworkPkg/UnitTestLib: GetActiveFrameworkHandle() no ASSERT() UnitTestFrameworkPkg/UnitTestDebugAssertLib: Add GoogleTest support UnitTestFrameworkPkg/SampleGoogleTest: Use EXPECT_ANY_THROW() UnitTestFrameworkPkg: Add DSC and host tests that always fail MdePkg/Include/Library/DebugLib.h | 6 +- .../Library/GoogleTestLib/GoogleTestLib.inf | 6 +- .../UnitTestDebugAssertLibHost.cpp | 63 ++ .../UnitTestDebugAssertLibHost.inf | 36 + .../UnitTestDebugAssertLibHost.uni | 11 + .../Library/UnitTestLib/Assert.c | 4 + .../Library/UnitTestLib/Log.c | 4 + .../Library/UnitTestLib/RunTests.c | 1 - .../Library/UnitTestLib/RunTestsCmocka.c | 1 - .../Library/UnitTestLib/UnitTestLib.c | 4 + UnitTestFrameworkPkg/ReadMe.md | 2 +- .../SampleGoogleTest/SampleGoogleTest.cpp | 13 +- .../SampleGoogleTestExpectFail.cpp} | 78 +- .../SampleGoogleTestHostExpectFail.inf | 36 + .../SampleGoogleTestGenerateException.cpp | 54 ++ .../SampleGoogleTestHostGenerateException.inf | 39 + .../SampleUnitTestDxeExpectFail.inf | 41 + .../SampleUnitTestExpectFail.c | 861 ++++++++++++++++++ .../SampleUnitTestHostExpectFail.inf | 35 + .../SampleUnitTestPeiExpectFail.inf | 41 + .../SampleUnitTestSmmExpectFail.inf | 42 + .../SampleUnitTestUefiShellExpectFail.inf | 38 + .../SampleUnitTestDxeGenerateException.inf | 43 + .../SampleUnitTestGenerateException.c | 204 +++++ .../SampleUnitTestHostGenerateException.inf | 37 + .../SampleUnitTestPeiGenerateException.inf | 43 + .../SampleUnitTestSmmGenerateException.inf | 44 + ...mpleUnitTestUefiShellGenerateException.inf | 40 + .../Test/UnitTestFrameworkPkgHostTest.dsc | 1 + ...UnitTestFrameworkPkgHostTestExpectFail.dsc | 44 + .../UnitTestFrameworkPkg.ci.yaml | 18 +- UnitTestFrameworkPkg/UnitTestFrameworkPkg.dsc | 26 + .../UnitTestFrameworkPkgHost.dsc.inc | 11 +- .../UnitTestFrameworkPkgTarget.dsc.inc | 14 + 34 files changed, 1884 insertions(+), 57 deletions(-) create mode 100644 UnitTestFrameworkPkg/Library/UnitTestDebugAssertLib/UnitTestDebugAssertLibHost.cpp create mode 100644 UnitTestFrameworkPkg/Library/UnitTestDebugAssertLib/UnitTestDebugAssertLibHost.inf create mode 100644 UnitTestFrameworkPkg/Library/UnitTestDebugAssertLib/UnitTestDebugAssertLibHost.uni copy UnitTestFrameworkPkg/Test/GoogleTest/Sample/{SampleGoogleTest/SampleGoogleTest.cpp => SampleGoogleTestExpectFail/SampleGoogleTestExpectFail.cpp} (71%) create mode 100644 UnitTestFrameworkPkg/Test/GoogleTest/Sample/SampleGoogleTestExpectFail/SampleGoogleTestHostExpectFail.inf create mode 100644 UnitTestFrameworkPkg/Test/GoogleTest/Sample/SampleGoogleTestGenerateException/SampleGoogleTestGenerateException.cpp create mode 100644 UnitTestFrameworkPkg/Test/GoogleTest/Sample/SampleGoogleTestGenerateException/SampleGoogleTestHostGenerateException.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestExpectFail/SampleUnitTestDxeExpectFail.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestExpectFail/SampleUnitTestExpectFail.c create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestExpectFail/SampleUnitTestHostExpectFail.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestExpectFail/SampleUnitTestPeiExpectFail.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestExpectFail/SampleUnitTestSmmExpectFail.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestExpectFail/SampleUnitTestUefiShellExpectFail.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestGenerateException/SampleUnitTestDxeGenerateException.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestGenerateException/SampleUnitTestGenerateException.c create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestGenerateException/SampleUnitTestHostGenerateException.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestGenerateException/SampleUnitTestPeiGenerateException.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestGenerateException/SampleUnitTestSmmGenerateException.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTest/Sample/SampleUnitTestGenerateException/SampleUnitTestUefiShellGenerateException.inf create mode 100644 UnitTestFrameworkPkg/Test/UnitTestFrameworkPkgHostTestExpectFail.dsc -- 2.40.1.windows.1 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#115224): https://edk2.groups.io/g/devel/message/115224 Mute This Topic: https://groups.io/mt/104212827/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-