From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=40.107.0.42; helo=eur02-am5-obe.outbound.protection.outlook.com; envelope-from=supreeth.venkatesh@arm.com; receiver=edk2-devel@lists.01.org Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00042.outbound.protection.outlook.com [40.107.0.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 3CF61211A43CD for ; Wed, 16 Jan 2019 09:03:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector1-arm-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YN5vFoepEjxrl7MVC8bL5VksYp85lVEpHIYWbSiWYYQ=; b=mQqrVeAPuN6jHpXcHkUgHP+XSoFSdPCtE6SaclqSfj0u2W6DyGT5JhC1PHmg8vlO2ftnUrOYXxRd6iZdI3M3aTmboS+DvwNHHJ4mmm82ENoUXMDLtSMf+dBSJLq9wkEP3R8Rdad5H+KCs2b0FCJg0ARoNznU1zND+yLHSbruwd4= Received: from AM4PR08MB2788.eurprd08.prod.outlook.com (10.171.191.18) by AM4PR08MB0881.eurprd08.prod.outlook.com (10.164.83.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1537.26; Wed, 16 Jan 2019 17:03:10 +0000 Received: from AM4PR08MB2788.eurprd08.prod.outlook.com ([fe80::3807:aeb7:fde8:baff]) by AM4PR08MB2788.eurprd08.prod.outlook.com ([fe80::3807:aeb7:fde8:baff%3]) with mapi id 15.20.1516.019; Wed, 16 Jan 2019 17:03:10 +0000 From: Supreeth Venkatesh To: "Jin, Eric" , Sakar Arora CC: "edk2-devel@lists.01.org" Thread-Topic: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal Thread-Index: AdSgwgEf4U2uyT4uRYG+AR9zd3VV2QADkKdgAX1u4XAACrv4QACqU/agAIHQtBAAhiDTYA== Date: Wed, 16 Jan 2019 17:03:10 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Supreeth.Venkatesh@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM4PR08MB0881; 6:+rTX9rPNdtCsVI6ghpg0PIN9UoLaIEBe8W2D5r2izw9ZP+w2f6wBaDa3SAzXnNFKlouUJ50wOHHKGx56nYa2ySksscRqtPMwSN3xHfEl+Tb8k3qcMmK85700mxrXyeGwwvCTFyMNPllAuqW/XPhl0Xg3g6m1iCpmMqVqGCWumfvBZKvWStXZy3Z9zIP84gV73wfcMod/9ad+LbvaGylDIVtLhmY7jtjiRyCHyHACc0QXweeXl7R3T7nDPP5WnI6sTj2mRL3hIT0kXEnpj7MU78bSDMvYKWgnhRCofxCpaDBfpbG+ZzlxGD5pBnYRJ+prg4FknHp7G9s3YfFkORkVgcn/EK83O8eAH+nGhAevn8aYgPe2PPCpagLmPYT3xOBioo814O1bXz0Jw+PVAnficMNXeSxiBcEkE3j+wPQQMbgHVu3amM0jOfBpuNK5Os79KKdd4WHAAVdA45212PGGaA==; 5:87NOVhFKk+uqfMDcVAW8Ey8PqhyQdHCtMeop/HtEVSmb9arHrb/VrVWYYc/B6LbucNBZvBpL7Ycwf9JROZLgrQpNXKFSC2SkZ72CiVQnZWrQjc7EL6werybNeFUtwMYZznxM0zcfCzAjPRK5kwvZ5m1V/RLlxhs91TYV9StHBoxyg4mBJ/XiPBodRLoKHKK4PaPLGqr88IeClWOeJwCQsg==; 7:uH0WsO1723LIvLCQ6lZAwXIal81D2BhN40AIH3NtT6H0cGLICZFqfPRf3P4Mo8sgIRKZFlNhUOmYPiAehu9z42bj5DMGns84duFl5lVJ1V0lxgB5ecOzKEJ6F3OGhJzhMXZpes9hviObdYQgVs3faQ== x-ms-exchange-antispam-srfa-diagnostics: SOS;SOR; x-ms-office365-filtering-correlation-id: 06e3042f-32bc-42a4-6dc5-08d67bd47e9e x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM4PR08MB0881; x-ms-traffictypediagnostic: AM4PR08MB0881: x-microsoft-antispam-prvs: x-forefront-prvs: 091949432C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(39860400002)(376002)(346002)(136003)(396003)(40434004)(13464003)(53754006)(199004)(189003)(6636002)(8676002)(66066001)(25786009)(105586002)(6436002)(55016002)(8936002)(71190400001)(4326008)(97736004)(5660300001)(316002)(2906002)(110136005)(99286004)(81166006)(81156014)(71200400001)(6246003)(3846002)(486006)(6116002)(76176011)(102836004)(446003)(68736007)(7736002)(11346002)(476003)(5024004)(256004)(72206003)(966005)(86362001)(33656002)(229853002)(6506007)(9686003)(561944003)(26005)(7696005)(6306002)(305945005)(53546011)(186003)(106356001)(74316002)(14454004)(478600001)(53936002)(14444005)(93886005); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR08MB0881; H:AM4PR08MB2788.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: c+YYNbpMonyqy3JXlUepNB1Vew2nNGSybHATDVIWzGdkxsjsu5vvi1NzYig5O1UVXrrdzkJUscp+XaQeHihdrpDqG1hrlQs8tCOz+4Anj2TXlzuwGKT8e91C8GYnoSENpOpqgF2/L9B1sfjfX7zl7R/2QkxSdiLqNxdrLGJfoeZEadZfavp19TvnTCgW8+ol1hUgth3ACubZPH/1j30Mp6nIE+XyAmSQpIQbVsVFXo4fpK65b8tBMk+KkG+kFGXzTU7lpNbgfJjGDvJUttllj38ZzSAYN8bneEKUtDBCyT6XZ6kEiGyNzABnMeoWbee0pmcsL9HF3/jOPttsRiCGc0BGxHk/59GkQeAIA+LOTNNNZgR44jJSJIhJsc0RYiTn+iGYein5Ble2ru9AjjXZMAXidEqCvUeMEF8aJ115Pmo= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-Network-Message-Id: 06e3042f-32bc-42a4-6dc5-08d67bd47e9e X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jan 2019 17:03:10.6161 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR08MB0881 Subject: Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2019 17:03:14 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Eric, SBBR test cases does not affect the normal flow of UEFI-SCT, so I don't thi= nk it causes any issues to UEFI-SCT, but will instead have added value to U= EFI-SCT. However, Please carefully review the pre-processing workflow, when Sakar se= nds SBBR Patches. Sakar, I need more clarification on the multiple parameters you are proposing. >>From your statement in the email: "Tests with attribute SBBR are meant to run for both UEFI and SBBR complian= ce, while those with attribute SBBR_EXCL are only meant for SBBR compliance= ." Scenario 1: Top level Test suite (in this case - Yocto based ACS) will have to invoke U= EFI SCT once with SBBR to ensure both UEFI and SBBR compliance. What's the use of SBBR_EXCL in this case? Scenario 2: Top Level Test suite (Yocto based ACS) will invoke SBBR_EXCL to ensure SBBR= specific test cases (which are not pre-existing in UEFI-SCT, but new test = cases you are proposing to add). However, it does not completely satisfy SBBR specification as some of the S= BBR specification is realized by running pre-existing tests in UEFI-SCT. For this, Top level test suite will have to invoke UEFI_SCT **again** witho= ut any parameters. correct? This will create unnecessary issues for integrating into top level test sui= te. Please clarify on how you are going to use these parameters. (use case scen= ario.) If this gets clarified, then I have no objections with this proposal. Thanks, Supreeth -----Original Message----- From: Jin, Eric Sent: Tuesday, January 15, 2019 1:23 AM To: Sakar Arora ; Supreeth Venkatesh Cc: edk2-devel@lists.01.org; Jin, Eric Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A des= ign proposal Hi Sakar, Thank for the clarification. Now, I understand that the SBBR wants to add/modify in existing edk2-test/u= efi-sct. In your description, '-s' is not suitable for sbbr purpose. The fundamental point to be discussed is that edk2-test/uefi-sct is UEFI Se= lf Compliance Test and should keep the alignment to UEFI spec. To my understanding, the SBBR is published by arm and gives special require= ments on UEFI/ACPI/SMBIOS areas. I remember some new files/new directories with the prefix "sbbr" in file/di= rectory name are adding into the edk2-test/uefi-sct In previous push reques= t emails, it makes me confused. And it is the reason that I ask for the pos= sible standalone space. Could you please send the patches for review? We can discuss further based = on them and explore the correct direction. At least, the baseline is it doe= sn't impact existing test result (I want to hear the comments from Supreeth= on this baseline). Best Regards Eric -----Original Message----- From: Sakar Arora Sent: Friday, January 11, 2019 8:23 PM To: Jin, Eric ; Supreeth Venkatesh ; edk2-devel@lists.01.org Cc: Prasanth Pulla Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A des= ign proposal Hi Supreeth, Eric, My replies inlined. Thanks, Sakar -----Original Message----- From: Jin, Eric Sent: Tuesday, January 8, 2019 7:30 AM To: Supreeth Venkatesh ; Sakar Arora ; edk2-devel@lists.01.org Cc: Prasanth Pulla Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A des= ign proposal Hello Sakar, SBBR spec is not published by UEFI Forum and can be considered as a supplem= ent to ARM UEFI. The evolution of edk2-test/uefi-sct should be tagged to UE= FI Spec only. Can we land SBBR test on standalone space under edk2-test currently or in s= hort-term? I mean SBBR test can leverage edk2-test/uefi-sct (as sub-module= )or not to build test efi files itself and these test efi files can be inte= grated into UEFI SCT to execute with user usage. [Sakar] If I understand correctly, you want a separate directory in edk2-te= st for SBBR code, rather than having it in edk2-test/uefi-sct. The problem I see here is that apart from standalone SBBR test cases, there= are some changes to the existing test cases inside edk2-test/uefi-sct, tha= t we would need pushed for SBBR compliance (These changes will only take ef= fect if "-sbbr" parameter is passed while running SCT). So SBBR tests are n= ot exactly standalone. Keeping all SBBR specific stuff away, would mean mai= ntaining patches to SCT code, which will be difficult. I hope I am making s= ense. For the new attributes - SBBR and SBBR_EXCL, I agree with Supreeth that the= y are more like the additional parameters. So they are not the similar case= s to (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED). [Sakar] The reasoning for proposing these 2 attributes is this. There are some SBBR tests that are needed only for SBBR compliance. When SC= T app is run *without* the "-sbbr" cmdline parameter, it should run all tes= ts, excluding those specific to SBBR. So it needs to differentiate, at runt= ime, tests that are valid for both SBBR and UEFI compliance from the ones t= hat are exclusively for SBBR. Tests with attribute SBBR are meant to run fo= r both UEFI and SBBR compliance, while those with attribute SBBR_EXCL are o= nly meant for SBBR compliance. If the intention is to distinguish SBBR test from existing test, It can be = implemented with "-s" parameter plus the SBBR test sequence easily. [Sakar] Using the "-s" parameter with test sequence file is an option I exp= lored a bit. It doesn't seem to offer control over what sub-tests to run within a test s= cenario, which is something we want while adding SBBR support in SCT. For = example, there are some sub-tests that need to be removed from some main te= sts, for SBBR compliance. As far as I understand, sequence file does not pr= ovide that level of control. Please correct me if I am missing something. Above is my quick response and I will continue to consider it these days an= d look at SBBR spec. Any input is welcome. Best Regards Eric -----Original Message----- From: Supreeth Venkatesh Sent: Tuesday, January 8, 2019 4:14 AM To: Sakar Arora ; edk2-devel@lists.01.org; Jin, Eric <= eric.jin@intel.com> Cc: Prasanth Pulla Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A des= ign proposal Sakar, Looks good. However, Why do we need two additional parameters SBBR and SBBR_EXCL? Since SBBR is a subset/superset of UEFI specifications, would "sbbr_standal= one" or similar (just sbbr) which would run all the sbbr tests including on= es that is already part of UEFI-SCT Suffice? Lets wait for additional comments from Eric. Thanks, Supreeth -----Original Message----- From: Sakar Arora Sent: Monday, December 31, 2018 12:50 AM To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh Cc: Prasanth Pulla Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design = proposal Hi All, Introduction --> The intent is to run SBBR tests using SCT infrastructure. This is the SBBR= specification link for reference http://infocenter.arm.com/help/topic/com.= arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf Majority of SBBR requirements can be tested by current SCT code as is. A fe= w more SBBR specific tests are needed to cover complete spec. Proposal --> An ideal setup is new SBBR specific code residing in edk2-test master. SCT = test infrastructure will run tests for UEFI spec compliance or just SBBR sp= ec compliance based on command line arguments. This would require run time = classification of tests. - Details Each SCT test provides a list of sub-tests to verify various aspects of the= requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DEST= RUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructur= e to do pre and post processing for it. The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing = set. The test infrastructure will decide whether to run or skip a test base= d on command line parameters and these new attributes. - The new attributes SBBR : Tests with this attribute are required by both SBBR and UEFI. SBBR_EXCL : Tests with this attribute are required by SBBR exclusively. - Examples Shell> Sct.efi -a -sbbr Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_= EXCL. Shell> Sct.efi -a Would run tests required by UEFI, i.e., all tests excluding those with attr= ibute SBBR_EXCL. - SCT test pre-processing flow The following logic will be added to the pre-processing flow. If (SCT_FOR_SBBR) { If (!TEST_CASE_FOR_SBBR(TestCaseAttribute)) goto SkipTest; } else { If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute)) goto SkipTest; } Comments are welcome. Thanks, Sakar IMPORTANT NOTICE: The contents of this email and any attachments are confid= ential and may also be privileged. If you are not the intended recipient, p= lease notify the sender immediately and do not disclose the contents to any= other person, use it for any purpose, or store or copy the information in = any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confid= ential and may also be privileged. If you are not the intended recipient, p= lease notify the sender immediately and do not disclose the contents to any= other person, use it for any purpose, or store or copy the information in = any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confid= ential and may also be privileged. If you are not the intended recipient, p= lease notify the sender immediately and do not disclose the contents to any= other person, use it for any purpose, or store or copy the information in = any medium. Thank you.