From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=40.107.15.78; helo=eur01-db5-obe.outbound.protection.outlook.com; envelope-from=sakar.arora@arm.com; receiver=edk2-devel@lists.01.org Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150078.outbound.protection.outlook.com [40.107.15.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 186E0211B698F for ; Fri, 11 Jan 2019 04:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector1-arm-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F3PMxXIqq4CI02D7f8qDxzYvGH4UdaAsEx+/gu7EE3o=; b=coxi8pZ0El2Z9ZbWm0ZtKfQFmHxmvbl7cn5qq/PD3p48zdLofPEuWNesRGnqfQRDGCl+lsGifDfYh8fRGweVAf5JLkgjHv8meyoaUEaLTtgJNs+qb6llz7B6EJ+tLSA1/afkv242d09mAYMLBxYQdLz9TLiVaW0OdMsha1Mur6c= Received: from DB7PR08MB3209.eurprd08.prod.outlook.com (52.134.110.159) by DB7PR08MB3082.eurprd08.prod.outlook.com (52.134.110.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1516.18; Fri, 11 Jan 2019 12:22:44 +0000 Received: from DB7PR08MB3209.eurprd08.prod.outlook.com ([fe80::28c8:85c9:89bf:f21b]) by DB7PR08MB3209.eurprd08.prod.outlook.com ([fe80::28c8:85c9:89bf:f21b%2]) with mapi id 15.20.1495.011; Fri, 11 Jan 2019 12:22:44 +0000 From: Sakar Arora To: "Jin, Eric" , Supreeth Venkatesh , "edk2-devel@lists.01.org" Thread-Topic: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal Thread-Index: AdSgwgEf4U2uyT4uRYG+AR9zd3VV2QADkKdgAX1u4XAACrv4QACqU/ag Date: Fri, 11 Jan 2019 12:22:44 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Sakar.Arora@arm.com; x-originating-ip: [217.140.105.40] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DB7PR08MB3082; 6:mm5AfYt/3L8fDs3M2RfJSIU82T0lEJUQORjS6G64eRY7tKgt/kH2wePSFumC2xjXETuJMwfmCk71F+QxRmTOAYUvbTRmKAXwlhyeGhEmNJDP0Nf9k4V8terlgCqM5+akD4wqv8TplW41tLDQhzW4we95zxCEaVAxuLIv5aCg8JynVWnCCVbFUzsPtXAsvjNU/8D4z0eej81iJBIXRbxC6f9ElP0kFe+DbhGu7SSJgTsHax3IHfbCe+7I8AMC4JJeP6x2ChTSep6aPM/ImuEOU4sjXTa5Kn20jgAtX8SMAl1i5e50nnABbShBHkKXbmgHxi6/0iErF73+xCbB0ypoHYTVtQ6eN3NvqYApV4sLmc7l5mJv4SXjLp1OcAOxPbNZS+yemYqQLLhapI6x4K0gPg6iu5P6CwnAlrseiiczNmJorZ+WG8vdgPud9jyte+eeJF1DjvEOYOsAL7G9Rf6Gsw==; 5:aP3GyHv54SkogkagKx7yQRdvWQ279x6wyRO1Wwsf1tCxWegzFeaHqBpGi4Rv7GBsoXpx5JVXBujBP+Rn7HHYhbR/7rSxjV3ZLjpcc7quh90apV1mNM1KnrKzl1fEC1sttNdp6+u33utOuLKSon9FVd5XNuOc9IKbqc9I1Yp1LKxniUScuGk8MWZvcCbFcKeK/kIQGTElkEMvonbZq+5aqw==; 7:ohtg5ZFsSgIL0+IZB+sCjQvrCWsLDaolA3EYBGQ49Oeg6tZvTBRRm7AVb/dn/xrZl/LW6ggJOVhI/8HISLTYwGEKWeyt21lWWH133BHwSMQz0cL/x7Z7Jbj1sTMHkBZkOQ54FwbHbEGNAAjtMyMBzQ== x-ms-exchange-antispam-srfa-diagnostics: SOS;SOR; x-ms-office365-filtering-correlation-id: 7d7d34a3-f0df-4c56-bdd5-08d677bf7d5d x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:DB7PR08MB3082; x-ms-traffictypediagnostic: DB7PR08MB3082: x-microsoft-antispam-prvs: x-forefront-prvs: 09144DB0F7 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(136003)(376002)(396003)(39860400002)(53754006)(189003)(199004)(13464003)(40434004)(99286004)(229853002)(6116002)(3846002)(72206003)(966005)(6436002)(55016002)(86362001)(14444005)(256004)(5024004)(305945005)(68736007)(7736002)(2906002)(14454004)(8676002)(74316002)(81156014)(81166006)(8936002)(97736004)(316002)(11346002)(25786009)(76176011)(7696005)(102836004)(33656002)(5660300001)(6246003)(6506007)(446003)(53936002)(53546011)(105586002)(561944003)(26005)(4326008)(93886005)(66066001)(186003)(71190400001)(71200400001)(486006)(110136005)(476003)(6306002)(478600001)(2501003)(9686003)(106356001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR08MB3082; H:DB7PR08MB3209.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 3fMdqxRySd4wSiqkT2k+OlemsMDfApaaus1F92PRxJEI4sKigboUt7bKuT2OqenJIFdpsraT6Y+uOc//LCIQFxT3pbT/51CY3FC64tjcU6FNtu6+hI8/qomDHt2tn4X+rwBGIcy5i+kY5Cd63Vmvdqzvi43krVqpzClJIyHvzY3zgBR8Y8HvKbOUHdIx0b2fAQ0eNonbuv3DR257MaD77pSHhPKY0rXn2oIDSpwN3FvwJs38f7KwiRrU+svABxF7nPRHxDCBV+e3WeGQXG/b1h4EJPQeFLMfVBr6o/mD8ZqHJxTa18OvV/OR1NgN/AGpUEu/YG6YFumkMPO5YAqSE6sdMeGewXK4ZZ3ARoU3Mz+TY2vjkxUjpcpzHQYyLTmCF60ptj2RXS4I4kjQUZl5HwqPn+rdqP9mUF2K6bcRxI4= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7d7d34a3-f0df-4c56-bdd5-08d677bf7d5d X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jan 2019 12:22:44.2746 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR08MB3082 Subject: Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Jan 2019 12:22:48 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Supreeth, Eric, My replies inlined. Thanks, Sakar -----Original Message----- From: Jin, Eric Sent: Tuesday, January 8, 2019 7:30 AM To: Supreeth Venkatesh ; Sakar Arora ; edk2-devel@lists.01.org Cc: Prasanth Pulla Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A des= ign proposal Hello Sakar, SBBR spec is not published by UEFI Forum and can be considered as a supplem= ent to ARM UEFI. The evolution of edk2-test/uefi-sct should be tagged to UE= FI Spec only. Can we land SBBR test on standalone space under edk2-test currently or in s= hort-term? I mean SBBR test can leverage edk2-test/uefi-sct (as sub-module= )or not to build test efi files itself and these test efi files can be inte= grated into UEFI SCT to execute with user usage. [Sakar] If I understand correctly, you want a separate directory in edk2-te= st for SBBR code, rather than having it in edk2-test/uefi-sct. The problem I see here is that apart from standalone SBBR test cases, there= are some changes to the existing test cases inside edk2-test/uefi-sct, tha= t we would need pushed for SBBR compliance (These changes will only take ef= fect if "-sbbr" parameter is passed while running SCT). So SBBR tests are n= ot exactly standalone. Keeping all SBBR specific stuff away, would mean mai= ntaining patches to SCT code, which will be difficult. I hope I am making s= ense. For the new attributes - SBBR and SBBR_EXCL, I agree with Supreeth that the= y are more like the additional parameters. So they are not the similar case= s to (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED). [Sakar] The reasoning for proposing these 2 attributes is this. There are some SBBR tests that are needed only for SBBR compliance. When SC= T app is run *without* the "-sbbr" cmdline parameter, it should run all tes= ts, excluding those specific to SBBR. So it needs to differentiate, at runt= ime, tests that are valid for both SBBR and UEFI compliance from the ones t= hat are exclusively for SBBR. Tests with attribute SBBR are meant to run fo= r both UEFI and SBBR compliance, while those with attribute SBBR_EXCL are o= nly meant for SBBR compliance. If the intention is to distinguish SBBR test from existing test, It can be = implemented with "-s" parameter plus the SBBR test sequence easily. [Sakar] Using the "-s" parameter with test sequence file is an option I exp= lored a bit. It doesn't seem to offer control over what sub-tests to run within a test s= cenario, which is something we want while adding SBBR support in SCT. For = example, there are some sub-tests that need to be removed from some main te= sts, for SBBR compliance. As far as I understand, sequence file does not pr= ovide that level of control. Please correct me if I am missing something. Above is my quick response and I will continue to consider it these days an= d look at SBBR spec. Any input is welcome. Best Regards Eric -----Original Message----- From: Supreeth Venkatesh Sent: Tuesday, January 8, 2019 4:14 AM To: Sakar Arora ; edk2-devel@lists.01.org; Jin, Eric <= eric.jin@intel.com> Cc: Prasanth Pulla Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A des= ign proposal Sakar, Looks good. However, Why do we need two additional parameters SBBR and SBBR_EXCL? Since SBBR is a subset/superset of UEFI specifications, would "sbbr_standal= one" or similar (just sbbr) which would run all the sbbr tests including on= es that is already part of UEFI-SCT Suffice? Lets wait for additional comments from Eric. Thanks, Supreeth -----Original Message----- From: Sakar Arora Sent: Monday, December 31, 2018 12:50 AM To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh Cc: Prasanth Pulla Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design = proposal Hi All, Introduction --> The intent is to run SBBR tests using SCT infrastructure. This is the SBBR= specification link for reference http://infocenter.arm.com/help/topic/com.= arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf Majority of SBBR requirements can be tested by current SCT code as is. A fe= w more SBBR specific tests are needed to cover complete spec. Proposal --> An ideal setup is new SBBR specific code residing in edk2-test master. SCT = test infrastructure will run tests for UEFI spec compliance or just SBBR sp= ec compliance based on command line arguments. This would require run time = classification of tests. - Details Each SCT test provides a list of sub-tests to verify various aspects of the= requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DEST= RUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructur= e to do pre and post processing for it. The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing = set. The test infrastructure will decide whether to run or skip a test base= d on command line parameters and these new attributes. - The new attributes SBBR : Tests with this attribute are required by both SBBR and UEFI. SBBR_EXCL : Tests with this attribute are required by SBBR exclusively. - Examples Shell> Sct.efi -a -sbbr Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_= EXCL. Shell> Sct.efi -a Would run tests required by UEFI, i.e., all tests excluding those with attr= ibute SBBR_EXCL. - SCT test pre-processing flow The following logic will be added to the pre-processing flow. If (SCT_FOR_SBBR) { If (!TEST_CASE_FOR_SBBR(TestCaseAttribute)) goto SkipTest; } else { If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute)) goto SkipTest; } Comments are welcome. Thanks, Sakar IMPORTANT NOTICE: The contents of this email and any attachments are confid= ential and may also be privileged. If you are not the intended recipient, p= lease notify the sender immediately and do not disclose the contents to any= other person, use it for any purpose, or store or copy the information in = any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confid= ential and may also be privileged. If you are not the intended recipient, p= lease notify the sender immediately and do not disclose the contents to any= other person, use it for any purpose, or store or copy the information in = any medium. Thank you.