Re: [edk2-devel] running CI locally

["Sean" <spbrogan@outlook.com>]

sorry.  In my last email I started typing at top and then moved inline 
and didn't delete my initial comments.  Please ignore the comments on 
top and look inline for details. :)

Thanks
Sean




On 8/21/2020 2:36 PM, Sean wrote:
> Laszlo,
> 
> 
> 
> Idea being if the package supports host based unit tests it will scan 
> package to make sure all host based unit tests and libraries are listed 
> in the DSC.
> 
> As to why you see a different set of packages...that is interesting.  If 
> you can include your full build logs i can look a little closer.  I know 
> on the Azure CI servers we don't run the
> 
> 
> On 8/21/2020 12:23 AM, Laszlo Ersek wrote:
>> Hi Mike, Sean;
>>
>> On 08/19/20 19:59, Laszlo Ersek wrote:
>>
>>> I'll report back with more results.
>>
>> I installed a Fedora 32 Server VM.
>>
>> Installed the "mono-complete" package (+its deps).
>>
>> Installed the "nodejs" package.
>>
>> Installed "cspell" (+deps) with "npm" (with the latter coming from the
>> nodejs package).
>>
>> Edk2 checked out at current HEAD
>> (5a6d764e1d073d28e8f398289ccb5592bf9a72ba).
>>
>>
>> * Platform CI results:
>>
>> - <https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI>:
>>
>>    Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5).
>>
>> - <https://github.com/tianocore/edk2/tree/master/ArmVirtPkg/PlatformCI>:
>>
>>    Built successfully for ARM and AARCH64 (toolchain: GCC5).
>>
>>
>> * Core CI results:
>>
>> - <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>
>>
>>    Complete run successful (toolchain: GCC5); no package / arch / target
>>    / test restrictions.
>>
>>    I got the following WARNINGs:
>>
>>    - ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not
>>      Core CI -- these are expected:
>>
>>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>
>>> PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>
>>    - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
>>      NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are
>>      not covered by host-based unit tests -- I think also expected:
>>
>>> PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>    - I'm not really sure about the following warnings. They were emitted
>>      for a subset of the above packages. I read the docs at
>>
>> <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck> 
>>
>>      but I still don't understand :)
> 
> Each plugin/test has a readme as well.  Not sure if this helps explain 
> the HostUnitTestDscCompleteCheck more.
> 
> https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck 
> 
> 
> As to why you don't see it skipped for ArmVirtPkg or a few others: I 
> think this is a bug.
> 
> In Azure CI i see it as test passed.
> 
> PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check Test 
> NO-TARGET --
> PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test 
> NO-TARGET
> 
> When i look at the code
> https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59 
> 
> 
> I see that how those packages *.ci.yaml have it configured is different 
> than those that show skipped below.  It avoids the skip conditions but 
> since those packages don't have any host unit test libraries or modules 
> there is no failure.
> 
> This should probably be a bugzilla on the 
> HostUnitTestDscCompleteCheck.py as the reporting is not correct.  Test 
> should show up as skipped.
> 
> 
> You can see them here showing up as success
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=11309&view=logs&j=780eaa17-fc2d-5131-b9d2-21b19bc9bec5&t=0791c90f-e7d8-5372-c9c5-edbe30bf6aeb&l=32 
> 
> 
>>
>>> PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check 
>>> Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test 
>>> NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check 
>>> Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check 
>>> Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test 
>>> NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>    - Still related to host-based unit tests, I believe the following
>>      warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and
>>      UnitTestFrameworkPkg, report that running the tests on the build
>>      host (understandably) prevents the tests from covering all edk2
>>      architectures:
> 
> Yes this message is because we override the arch for NOOPT to match that 
> of the build system since we are going to run these executables in the 
> host environment.
> 
>>
>>> PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler 
>>> Plugin NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>      - The below seems to be related to the "spell checking in audit
>>        mode" known issues at
>>        <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.
>>
>>        What does "audit mode" mean?
> 
> see readme here.
> https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration 
> 
> 
> Basically runs the tests and reports all the errors as warnings and then 
> reports the test as skipped.
> It was hopefully to encourage package maintainers to fix their spelling 
> issues but not breaking the build on day 1.
> 
> 
>>
>>        FWIW, the packages listed below (EmulatorPkg, MdeModulePkg,
>>        MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper
>>        subset of the packges noted in ".pytool/Readme.md" -- the latter
>>        mentions CryptoPkg and SecurityPkg in addition.
>>
>>> PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>    - Not sure about the last one:
>>
>>> PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Library Class Check Test 
>>> NO-TARGET
>>
>>
>> * One suggestion (in addition to the others made in this thread):
>>
>> - Please add "BaseToolsBuild" to "BaseTools/.gitignore".
> 
> There is a bugzilla for this already.
> https://bugzilla.tianocore.org/show_bug.cgi?id=2581
> 
> 
>>
>>
>> * A question:
>>
>> - On github, there is some logic that restricts the full Core CI build
>>    to packages that may be affected by a patch series. Is this available
>>    locally?
>>
>>    According to the Core CI documentation, the "stuart_ci_build" command
>>    can be limited with "-p" / "-a" / "-t" flags; given a commit range,
>>    how do we calculate the "tightest" flag values?
> 
> A couple things here.
> You can always look at the azurepipelines files to get a better 
> understanding of the CI process.  Hopefully there is no magic (besides 
> being able to read yaml).  You can see here: 
> https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35 
> 
> 
> It is actually another stuart tool.
> stuart_pr_eval.
> We have bug to document it.
> https://github.com/tianocore/edk2-pytool-extensions/issues/89
> But the python file is pretty well documented. There are 4 policies that 
> can trigger the need for a change to build a package.
> See here
> https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117 
> 
> 
> 
> 
>>
>> Thank you very much!
>> Laszlo
>>
> 
> 
>