[edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[Sakar Arora]

Hi All,

Introduction -->

The intent is to run SBBR  tests using SCT infrastructure. This is the SBBR specification link for reference http://infocenter.arm.com/help/topic/com.arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf

Majority of SBBR requirements can be tested by current SCT code as is. A few more SBBR specific tests are needed to cover complete spec.

Proposal -->

An ideal setup is new SBBR specific code residing in edk2-test master. SCT test infrastructure will run tests for UEFI spec compliance or just SBBR spec compliance based on command line arguments. This would require run time classification of tests.

- Details

Each SCT test provides a list of sub-tests to verify various aspects of the requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructure to do pre and post processing for it.

The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing set. The test infrastructure will decide whether to run or skip a test based on command line parameters and these new attributes.

- The new attributes

SBBR : Tests with this attribute are required by both SBBR and UEFI.
SBBR_EXCL : Tests with this attribute are required by SBBR exclusively.

- Examples

Shell> Sct.efi -a -sbbr

Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_EXCL.

Shell> Sct.efi -a

Would run tests required by UEFI, i.e., all tests excluding those with attribute SBBR_EXCL.

-  SCT test pre-processing flow

The following logic will be added to the pre-processing flow.

If (SCT_FOR_SBBR) {
       If (!TEST_CASE_FOR_SBBR(TestCaseAttribute))
            goto SkipTest;
} else {
    If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute))
           goto SkipTest;
}

Comments are welcome.

Thanks,
Sakar
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[Supreeth Venkatesh]

Sakar,

Looks good.
However, Why do we need two  additional parameters SBBR and SBBR_EXCL?

Since SBBR is a subset/superset of UEFI specifications,
would "sbbr_standalone" or similar (just sbbr) which would run all the sbbr tests including ones that is already part of UEFI-SCT
Suffice?

Lets wait for additional comments from Eric.

Thanks,
Supreeth

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com>
Sent: Monday, December 31, 2018 12:50 AM
To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi All,

Introduction -->

The intent is to run SBBR  tests using SCT infrastructure. This is the SBBR specification link for reference http://infocenter.arm.com/help/topic/com.arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf

Majority of SBBR requirements can be tested by current SCT code as is. A few more SBBR specific tests are needed to cover complete spec.

Proposal -->

An ideal setup is new SBBR specific code residing in edk2-test master. SCT test infrastructure will run tests for UEFI spec compliance or just SBBR spec compliance based on command line arguments. This would require run time classification of tests.

- Details

Each SCT test provides a list of sub-tests to verify various aspects of the requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructure to do pre and post processing for it.

The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing set. The test infrastructure will decide whether to run or skip a test based on command line parameters and these new attributes.

- The new attributes

SBBR : Tests with this attribute are required by both SBBR and UEFI.
SBBR_EXCL : Tests with this attribute are required by SBBR exclusively.

- Examples

Shell> Sct.efi -a -sbbr

Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_EXCL.

Shell> Sct.efi -a

Would run tests required by UEFI, i.e., all tests excluding those with attribute SBBR_EXCL.

-  SCT test pre-processing flow

The following logic will be added to the pre-processing flow.

If (SCT_FOR_SBBR) {
       If (!TEST_CASE_FOR_SBBR(TestCaseAttribute))
            goto SkipTest;
} else {
    If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute))
           goto SkipTest;
}

Comments are welcome.

Thanks,
Sakar
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[Jin, Eric]

Hello Sakar,

SBBR spec is not published by UEFI Forum and can be considered as a supplement to ARM UEFI. The evolution of edk2-test/uefi-sct should be tagged to UEFI Spec only.  
Can we land SBBR test on standalone space under edk2-test currently or in short-term?  I mean SBBR test can leverage edk2-test/uefi-sct (as sub-module)or not to build test efi files itself and these test efi files can be integrated into UEFI SCT to execute with user usage.

For the new attributes - SBBR and SBBR_EXCL, I agree with Supreeth that they are more like the additional parameters. So they are not the similar cases to (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED).
If the intention is to distinguish SBBR test from existing test, It can be implemented with "-s" parameter plus the SBBR test sequence easily. 

Above is my quick response and I will continue to consider it these days and look at SBBR spec.  Any input is welcome. 

Best Regards
Eric

-----Original Message-----
From: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com> 
Sent: Tuesday, January 8, 2019 4:14 AM
To: Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org; Jin, Eric <eric.jin@intel.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Sakar,

Looks good.
However, Why do we need two  additional parameters SBBR and SBBR_EXCL?

Since SBBR is a subset/superset of UEFI specifications, would "sbbr_standalone" or similar (just sbbr) which would run all the sbbr tests including ones that is already part of UEFI-SCT Suffice?

Lets wait for additional comments from Eric.

Thanks,
Supreeth

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com>
Sent: Monday, December 31, 2018 12:50 AM
To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi All,

Introduction -->

The intent is to run SBBR  tests using SCT infrastructure. This is the SBBR specification link for reference http://infocenter.arm.com/help/topic/com.arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf

Majority of SBBR requirements can be tested by current SCT code as is. A few more SBBR specific tests are needed to cover complete spec.

Proposal -->

An ideal setup is new SBBR specific code residing in edk2-test master. SCT test infrastructure will run tests for UEFI spec compliance or just SBBR spec compliance based on command line arguments. This would require run time classification of tests.

- Details

Each SCT test provides a list of sub-tests to verify various aspects of the requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructure to do pre and post processing for it.

The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing set. The test infrastructure will decide whether to run or skip a test based on command line parameters and these new attributes.

- The new attributes

SBBR : Tests with this attribute are required by both SBBR and UEFI.
SBBR_EXCL : Tests with this attribute are required by SBBR exclusively.

- Examples

Shell> Sct.efi -a -sbbr

Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_EXCL.

Shell> Sct.efi -a

Would run tests required by UEFI, i.e., all tests excluding those with attribute SBBR_EXCL.

-  SCT test pre-processing flow

The following logic will be added to the pre-processing flow.

If (SCT_FOR_SBBR) {
       If (!TEST_CASE_FOR_SBBR(TestCaseAttribute))
            goto SkipTest;
} else {
    If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute))
           goto SkipTest;
}

Comments are welcome.

Thanks,
Sakar
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[Sakar Arora]

Hi Supreeth, Eric,

My replies inlined.

Thanks,
Sakar
-----Original Message-----
From: Jin, Eric <eric.jin@intel.com>
Sent: Tuesday, January 8, 2019 7:30 AM
To: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>; Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hello Sakar,

SBBR spec is not published by UEFI Forum and can be considered as a supplement to ARM UEFI. The evolution of edk2-test/uefi-sct should be tagged to UEFI Spec only.
Can we land SBBR test on standalone space under edk2-test currently or in short-term?  I mean SBBR test can leverage edk2-test/uefi-sct (as sub-module)or not to build test efi files itself and these test efi files can be integrated into UEFI SCT to execute with user usage.

[Sakar] If I understand correctly, you want a separate directory in edk2-test for SBBR code, rather than having it in edk2-test/uefi-sct.
The problem I see here is that apart from standalone SBBR test cases, there are some changes to the existing test cases inside edk2-test/uefi-sct, that we would need pushed for SBBR compliance (These changes will only take effect if "-sbbr" parameter is passed while running SCT). So SBBR tests are not exactly standalone. Keeping all SBBR specific stuff away, would mean maintaining patches to SCT code, which will be difficult. I hope I am making sense.

For the new attributes - SBBR and SBBR_EXCL, I agree with Supreeth that they are more like the additional parameters. So they are not the similar cases to (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED).

[Sakar] The reasoning for proposing these 2 attributes is this.
There are some SBBR tests that are needed only for SBBR compliance. When SCT app is run *without* the "-sbbr" cmdline parameter, it should run all tests, excluding those specific to SBBR. So it needs to differentiate, at runtime, tests that are valid for both SBBR and UEFI compliance from the ones that are exclusively for SBBR. Tests with attribute SBBR are meant to run for both UEFI and SBBR compliance, while those with attribute SBBR_EXCL are only meant for SBBR compliance.

If the intention is to distinguish SBBR test from existing test, It can be implemented with "-s" parameter plus the SBBR test sequence easily.

[Sakar] Using the "-s" parameter with test sequence file is an option I explored a bit.
It doesn't seem to offer control over what sub-tests to run within a test scenario,  which is something we want while adding SBBR support in SCT. For example, there are some sub-tests that need to be removed from some main tests, for SBBR compliance. As far as I understand, sequence file does not provide that level of control. Please correct me if I am missing something.

Above is my quick response and I will continue to consider it these days and look at SBBR spec.  Any input is welcome.

Best Regards
Eric
-----Original Message-----
From: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Sent: Tuesday, January 8, 2019 4:14 AM
To: Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org; Jin, Eric <eric.jin@intel.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Sakar,

Looks good.
However, Why do we need two  additional parameters SBBR and SBBR_EXCL?

Since SBBR is a subset/superset of UEFI specifications, would "sbbr_standalone" or similar (just sbbr) which would run all the sbbr tests including ones that is already part of UEFI-SCT Suffice?

Lets wait for additional comments from Eric.

Thanks,
Supreeth

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com>
Sent: Monday, December 31, 2018 12:50 AM
To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi All,

Introduction -->

The intent is to run SBBR  tests using SCT infrastructure. This is the SBBR specification link for reference http://infocenter.arm.com/help/topic/com.arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf

Majority of SBBR requirements can be tested by current SCT code as is. A few more SBBR specific tests are needed to cover complete spec.

Proposal -->

An ideal setup is new SBBR specific code residing in edk2-test master. SCT test infrastructure will run tests for UEFI spec compliance or just SBBR spec compliance based on command line arguments. This would require run time classification of tests.

- Details

Each SCT test provides a list of sub-tests to verify various aspects of the requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructure to do pre and post processing for it.

The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing set. The test infrastructure will decide whether to run or skip a test based on command line parameters and these new attributes.

- The new attributes

SBBR : Tests with this attribute are required by both SBBR and UEFI.
SBBR_EXCL : Tests with this attribute are required by SBBR exclusively.

- Examples

Shell> Sct.efi -a -sbbr

Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_EXCL.

Shell> Sct.efi -a

Would run tests required by UEFI, i.e., all tests excluding those with attribute SBBR_EXCL.

-  SCT test pre-processing flow

The following logic will be added to the pre-processing flow.

If (SCT_FOR_SBBR) {
       If (!TEST_CASE_FOR_SBBR(TestCaseAttribute))
            goto SkipTest;
} else {
    If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute))
           goto SkipTest;
}

Comments are welcome.

Thanks,
Sakar
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[Jin, Eric]

Hi Sakar,

Thank for the clarification.
Now, I understand that the SBBR wants to add/modify in existing edk2-test/uefi-sct. In your description, '-s' is not suitable for sbbr purpose. 

The fundamental point to be discussed is that edk2-test/uefi-sct is UEFI Self Compliance Test and should keep the alignment to UEFI spec.
To my understanding, the SBBR is published by arm and gives special requirements on UEFI/ACPI/SMBIOS areas. 
I remember some new files/new directories with the prefix "sbbr" in file/directory name are adding into the edk2-test/uefi-sct In previous push request emails, it makes me confused. And it is the reason that I ask for the possible standalone space.

Could you please send the patches for review? We can discuss further based on them and explore the correct direction. At least, the baseline is it doesn't impact existing test result (I want to hear the comments from Supreeth on this baseline). 

Best Regards
Eric

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com> 
Sent: Friday, January 11, 2019 8:23 PM
To: Jin, Eric <eric.jin@intel.com>; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>; edk2-devel@lists.01.org
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi Supreeth, Eric,

My replies inlined.

Thanks,
Sakar
-----Original Message-----
From: Jin, Eric <eric.jin@intel.com>
Sent: Tuesday, January 8, 2019 7:30 AM
To: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>; Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hello Sakar,

SBBR spec is not published by UEFI Forum and can be considered as a supplement to ARM UEFI. The evolution of edk2-test/uefi-sct should be tagged to UEFI Spec only.
Can we land SBBR test on standalone space under edk2-test currently or in short-term?  I mean SBBR test can leverage edk2-test/uefi-sct (as sub-module)or not to build test efi files itself and these test efi files can be integrated into UEFI SCT to execute with user usage.

[Sakar] If I understand correctly, you want a separate directory in edk2-test for SBBR code, rather than having it in edk2-test/uefi-sct.
The problem I see here is that apart from standalone SBBR test cases, there are some changes to the existing test cases inside edk2-test/uefi-sct, that we would need pushed for SBBR compliance (These changes will only take effect if "-sbbr" parameter is passed while running SCT). So SBBR tests are not exactly standalone. Keeping all SBBR specific stuff away, would mean maintaining patches to SCT code, which will be difficult. I hope I am making sense.

For the new attributes - SBBR and SBBR_EXCL, I agree with Supreeth that they are more like the additional parameters. So they are not the similar cases to (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED).

[Sakar] The reasoning for proposing these 2 attributes is this.
There are some SBBR tests that are needed only for SBBR compliance. When SCT app is run *without* the "-sbbr" cmdline parameter, it should run all tests, excluding those specific to SBBR. So it needs to differentiate, at runtime, tests that are valid for both SBBR and UEFI compliance from the ones that are exclusively for SBBR. Tests with attribute SBBR are meant to run for both UEFI and SBBR compliance, while those with attribute SBBR_EXCL are only meant for SBBR compliance.

If the intention is to distinguish SBBR test from existing test, It can be implemented with "-s" parameter plus the SBBR test sequence easily.

[Sakar] Using the "-s" parameter with test sequence file is an option I explored a bit.
It doesn't seem to offer control over what sub-tests to run within a test scenario,  which is something we want while adding SBBR support in SCT. For example, there are some sub-tests that need to be removed from some main tests, for SBBR compliance. As far as I understand, sequence file does not provide that level of control. Please correct me if I am missing something.

Above is my quick response and I will continue to consider it these days and look at SBBR spec.  Any input is welcome.

Best Regards
Eric
-----Original Message-----
From: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Sent: Tuesday, January 8, 2019 4:14 AM
To: Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org; Jin, Eric <eric.jin@intel.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Sakar,

Looks good.
However, Why do we need two  additional parameters SBBR and SBBR_EXCL?

Since SBBR is a subset/superset of UEFI specifications, would "sbbr_standalone" or similar (just sbbr) which would run all the sbbr tests including ones that is already part of UEFI-SCT Suffice?

Lets wait for additional comments from Eric.

Thanks,
Supreeth

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com>
Sent: Monday, December 31, 2018 12:50 AM
To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi All,

Introduction -->

The intent is to run SBBR  tests using SCT infrastructure. This is the SBBR specification link for reference http://infocenter.arm.com/help/topic/com.arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf

Majority of SBBR requirements can be tested by current SCT code as is. A few more SBBR specific tests are needed to cover complete spec.

Proposal -->

An ideal setup is new SBBR specific code residing in edk2-test master. SCT test infrastructure will run tests for UEFI spec compliance or just SBBR spec compliance based on command line arguments. This would require run time classification of tests.

- Details

Each SCT test provides a list of sub-tests to verify various aspects of the requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructure to do pre and post processing for it.

The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing set. The test infrastructure will decide whether to run or skip a test based on command line parameters and these new attributes.

- The new attributes

SBBR : Tests with this attribute are required by both SBBR and UEFI.
SBBR_EXCL : Tests with this attribute are required by SBBR exclusively.

- Examples

Shell> Sct.efi -a -sbbr

Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_EXCL.

Shell> Sct.efi -a

Would run tests required by UEFI, i.e., all tests excluding those with attribute SBBR_EXCL.

-  SCT test pre-processing flow

The following logic will be added to the pre-processing flow.

If (SCT_FOR_SBBR) {
       If (!TEST_CASE_FOR_SBBR(TestCaseAttribute))
            goto SkipTest;
} else {
    If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute))
           goto SkipTest;
}

Comments are welcome.

Thanks,
Sakar
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

[Supreeth Venkatesh]

Eric,

SBBR test cases does not affect the normal flow of UEFI-SCT, so I don't think it causes any issues to UEFI-SCT, but will instead have added value to UEFI-SCT.
However, Please carefully review the pre-processing workflow, when Sakar sends SBBR Patches.

Sakar,
I need more clarification on the multiple parameters you are proposing.
>From your statement in the email:
"Tests with attribute SBBR are meant to run for both UEFI and SBBR compliance, while those with attribute SBBR_EXCL are only meant for SBBR compliance."


Scenario 1:
Top level Test suite (in this case - Yocto based ACS) will have to invoke UEFI SCT once with SBBR to ensure both UEFI and SBBR compliance.
What's the use of SBBR_EXCL in this case?

Scenario 2:
Top Level Test suite (Yocto based ACS) will invoke SBBR_EXCL to ensure SBBR specific test cases (which are not pre-existing in UEFI-SCT, but new test cases you are proposing to add).
However, it does not completely satisfy SBBR specification as some of the SBBR specification is realized by running pre-existing tests in UEFI-SCT.
For this, Top level test suite will have to invoke UEFI_SCT **again** without any parameters. correct?
This will create unnecessary issues for integrating into top level test suite.

Please clarify on how you are going to use these parameters. (use case scenario.)
If this gets clarified, then I have no objections with this proposal.

Thanks,
Supreeth

-----Original Message-----
From: Jin, Eric <eric.jin@intel.com>
Sent: Tuesday, January 15, 2019 1:23 AM
To: Sakar Arora <Sakar.Arora@arm.com>; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Cc: edk2-devel@lists.01.org; Jin, Eric <eric.jin@intel.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi Sakar,

Thank for the clarification.
Now, I understand that the SBBR wants to add/modify in existing edk2-test/uefi-sct. In your description, '-s' is not suitable for sbbr purpose.

The fundamental point to be discussed is that edk2-test/uefi-sct is UEFI Self Compliance Test and should keep the alignment to UEFI spec.
To my understanding, the SBBR is published by arm and gives special requirements on UEFI/ACPI/SMBIOS areas.
I remember some new files/new directories with the prefix "sbbr" in file/directory name are adding into the edk2-test/uefi-sct In previous push request emails, it makes me confused. And it is the reason that I ask for the possible standalone space.

Could you please send the patches for review? We can discuss further based on them and explore the correct direction. At least, the baseline is it doesn't impact existing test result (I want to hear the comments from Supreeth on this baseline).

Best Regards
Eric

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com>
Sent: Friday, January 11, 2019 8:23 PM
To: Jin, Eric <eric.jin@intel.com>; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>; edk2-devel@lists.01.org
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi Supreeth, Eric,

My replies inlined.

Thanks,
Sakar
-----Original Message-----
From: Jin, Eric <eric.jin@intel.com>
Sent: Tuesday, January 8, 2019 7:30 AM
To: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>; Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hello Sakar,

SBBR spec is not published by UEFI Forum and can be considered as a supplement to ARM UEFI. The evolution of edk2-test/uefi-sct should be tagged to UEFI Spec only.
Can we land SBBR test on standalone space under edk2-test currently or in short-term?  I mean SBBR test can leverage edk2-test/uefi-sct (as sub-module)or not to build test efi files itself and these test efi files can be integrated into UEFI SCT to execute with user usage.

[Sakar] If I understand correctly, you want a separate directory in edk2-test for SBBR code, rather than having it in edk2-test/uefi-sct.
The problem I see here is that apart from standalone SBBR test cases, there are some changes to the existing test cases inside edk2-test/uefi-sct, that we would need pushed for SBBR compliance (These changes will only take effect if "-sbbr" parameter is passed while running SCT). So SBBR tests are not exactly standalone. Keeping all SBBR specific stuff away, would mean maintaining patches to SCT code, which will be difficult. I hope I am making sense.

For the new attributes - SBBR and SBBR_EXCL, I agree with Supreeth that they are more like the additional parameters. So they are not the similar cases to (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED).

[Sakar] The reasoning for proposing these 2 attributes is this.
There are some SBBR tests that are needed only for SBBR compliance. When SCT app is run *without* the "-sbbr" cmdline parameter, it should run all tests, excluding those specific to SBBR. So it needs to differentiate, at runtime, tests that are valid for both SBBR and UEFI compliance from the ones that are exclusively for SBBR. Tests with attribute SBBR are meant to run for both UEFI and SBBR compliance, while those with attribute SBBR_EXCL are only meant for SBBR compliance.

If the intention is to distinguish SBBR test from existing test, It can be implemented with "-s" parameter plus the SBBR test sequence easily.

[Sakar] Using the "-s" parameter with test sequence file is an option I explored a bit.
It doesn't seem to offer control over what sub-tests to run within a test scenario,  which is something we want while adding SBBR support in SCT. For example, there are some sub-tests that need to be removed from some main tests, for SBBR compliance. As far as I understand, sequence file does not provide that level of control. Please correct me if I am missing something.

Above is my quick response and I will continue to consider it these days and look at SBBR spec.  Any input is welcome.

Best Regards
Eric
-----Original Message-----
From: Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Sent: Tuesday, January 8, 2019 4:14 AM
To: Sakar Arora <Sakar.Arora@arm.com>; edk2-devel@lists.01.org; Jin, Eric <eric.jin@intel.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: RE: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Sakar,

Looks good.
However, Why do we need two  additional parameters SBBR and SBBR_EXCL?

Since SBBR is a subset/superset of UEFI specifications, would "sbbr_standalone" or similar (just sbbr) which would run all the sbbr tests including ones that is already part of UEFI-SCT Suffice?

Lets wait for additional comments from Eric.

Thanks,
Supreeth

-----Original Message-----
From: Sakar Arora <Sakar.Arora@arm.com>
Sent: Monday, December 31, 2018 12:50 AM
To: edk2-devel@lists.01.org; eric.jin@intel.com; Supreeth Venkatesh <Supreeth.Venkatesh@arm.com>
Cc: Prasanth Pulla <Prasanth.Pulla@arm.com>
Subject: [edk2][edk2-test][RFC] Integrating SBBR tests into SCT - A design proposal

Hi All,

Introduction -->

The intent is to run SBBR  tests using SCT infrastructure. This is the SBBR specification link for reference http://infocenter.arm.com/help/topic/com.arm.doc.den0044c/Server_Base_Boot_Requirements_v1_1_Arm_DEN_0044C.pdf

Majority of SBBR requirements can be tested by current SCT code as is. A few more SBBR specific tests are needed to cover complete spec.

Proposal -->

An ideal setup is new SBBR specific code residing in edk2-test master. SCT test infrastructure will run tests for UEFI spec compliance or just SBBR spec compliance based on command line arguments. This would require run time classification of tests.

- Details

Each SCT test provides a list of sub-tests to verify various aspects of the requirement. Each sub-test is assigned some attributes (AUTO, MANUAL, DESTRUCTIVE, or RESET_REQUIRED) which are referred to by the test infrastructure to do pre and post processing for it.

The proposal is to add 2 new attributes SBBR and SBBR_EXCL to the existing set. The test infrastructure will decide whether to run or skip a test based on command line parameters and these new attributes.

- The new attributes

SBBR : Tests with this attribute are required by both SBBR and UEFI.
SBBR_EXCL : Tests with this attribute are required by SBBR exclusively.

- Examples

Shell> Sct.efi -a -sbbr

Would run tests required by SBBR, i.e., tests with attributes SBBR or SBBR_EXCL.

Shell> Sct.efi -a

Would run tests required by UEFI, i.e., all tests excluding those with attribute SBBR_EXCL.

-  SCT test pre-processing flow

The following logic will be added to the pre-processing flow.

If (SCT_FOR_SBBR) {
       If (!TEST_CASE_FOR_SBBR(TestCaseAttribute))
            goto SkipTest;
} else {
    If (TEST_CASE_FOR_SBBR_EXCL(TestCaseAttribute))
           goto SkipTest;
}

Comments are welcome.

Thanks,
Sakar
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.