Re: [edk2-devel] running CI locally

["Sean" <spbrogan@outlook.com>]

Laszlo,



Idea being if the package supports host based unit tests it will scan 
package to make sure all host based unit tests and libraries are listed 
in the DSC.

As to why you see a different set of packages...that is interesting.  If 
you can include your full build logs i can look a little closer.  I know 
on the Azure CI servers we don't run the


On 8/21/2020 12:23 AM, Laszlo Ersek wrote:
> Hi Mike, Sean;
> 
> On 08/19/20 19:59, Laszlo Ersek wrote:
> 
>> I'll report back with more results.
> 
> I installed a Fedora 32 Server VM.
> 
> Installed the "mono-complete" package (+its deps).
> 
> Installed the "nodejs" package.
> 
> Installed "cspell" (+deps) with "npm" (with the latter coming from the
> nodejs package).
> 
> Edk2 checked out at current HEAD
> (5a6d764e1d073d28e8f398289ccb5592bf9a72ba).
> 
> 
> * Platform CI results:
> 
> - <https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI>:
> 
>    Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5).
> 
> - <https://github.com/tianocore/edk2/tree/master/ArmVirtPkg/PlatformCI>:
> 
>    Built successfully for ARM and AARCH64 (toolchain: GCC5).
> 
> 
> * Core CI results:
> 
> - <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>
> 
>    Complete run successful (toolchain: GCC5); no package / arch / target
>    / test restrictions.
> 
>    I got the following WARNINGs:
> 
>    - ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not
>      Core CI -- these are expected:
> 
>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> 
>> PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>> PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> 
>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> 
>    - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
>      NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are
>      not covered by host-based unit tests -- I think also expected:
> 
>> PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>    - I'm not really sure about the following warnings. They were emitted
>      for a subset of the above packages. I read the docs at
>      <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck>
>      but I still don't understand :)

Each plugin/test has a readme as well.  Not sure if this helps explain 
the HostUnitTestDscCompleteCheck more.

https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck

As to why you don't see it skipped for ArmVirtPkg or a few others: I 
think this is a bug.

In Azure CI i see it as test passed.

PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check Test 
NO-TARGET --
PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test 
NO-TARGET

When i look at the code
https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59

I see that how those packages *.ci.yaml have it configured is different 
than those that show skipped below.  It avoids the skip conditions but 
since those packages don't have any host unit test libraries or modules 
there is no failure.

This should probably be a bugzilla on the 
HostUnitTestDscCompleteCheck.py as the reporting is not correct.  Test 
should show up as skipped.


You can see them here showing up as success
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=11309&view=logs&j=780eaa17-fc2d-5131-b9d2-21b19bc9bec5&t=0791c90f-e7d8-5372-c9c5-edbe30bf6aeb&l=32

> 
>> PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>    - Still related to host-based unit tests, I believe the following
>      warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and
>      UnitTestFrameworkPkg, report that running the tests on the build
>      host (understandably) prevents the tests from covering all edk2
>      architectures:

Yes this message is because we override the arch for NOOPT to match that 
of the build system since we are going to run these executables in the 
host environment.

> 
>> PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>      - The below seems to be related to the "spell checking in audit
>        mode" known issues at
>        <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.
> 
>        What does "audit mode" mean?

see readme here.
https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration

Basically runs the tests and reports all the errors as warnings and then 
reports the test as skipped.
It was hopefully to encourage package maintainers to fix their spelling 
issues but not breaking the build on day 1.


> 
>        FWIW, the packages listed below (EmulatorPkg, MdeModulePkg,
>        MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper
>        subset of the packges noted in ".pytool/Readme.md" -- the latter
>        mentions CryptoPkg and SecurityPkg in addition.
> 
>> PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>    - Not sure about the last one:
> 
>> PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Library Class Check Test NO-TARGET
> 
> 
> * One suggestion (in addition to the others made in this thread):
> 
> - Please add "BaseToolsBuild" to "BaseTools/.gitignore".

There is a bugzilla for this already.
https://bugzilla.tianocore.org/show_bug.cgi?id=2581


> 
> 
> * A question:
> 
> - On github, there is some logic that restricts the full Core CI build
>    to packages that may be affected by a patch series. Is this available
>    locally?
> 
>    According to the Core CI documentation, the "stuart_ci_build" command
>    can be limited with "-p" / "-a" / "-t" flags; given a commit range,
>    how do we calculate the "tightest" flag values?

A couple things here.
You can always look at the azurepipelines files to get a better 
understanding of the CI process.  Hopefully there is no magic (besides 
being able to read yaml).  You can see here: 
https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35

It is actually another stuart tool.
stuart_pr_eval.
We have bug to document it.
https://github.com/tianocore/edk2-pytool-extensions/issues/89
But the python file is pretty well documented. There are 4 policies that 
can trigger the need for a change to build a package.
See here
https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117



> 
> Thank you very much!
> Laszlo
>