running CI locally

running CI locally

[Laszlo Ersek]

Hi!

I'd like to test CI locally. I'm going through ".pytool/Readme.md" with
the tree checked out at 7e6f150b6902 (= current HEAD). I'm doing this in
a RHEL8 VM, with a python3 virtual environment set up / entered.


* My first note is that the command

  pip install --upgrade pip-requirements.txt

under "Prerequisets", has a small typo; it should be

  pip install --upgrade -r pip-requirements.txt

(the "-r" option is missing).

(

After adding "-r", the following components are now installed in my
virtual env:

- edk2-pytool-library: 0.10.12
- edk2-pytool-extensions: 0.13.9
- antlr4-python3-runtime: 4.7.1
- pyyaml: 5.3.1

Stating this because it might matter for the rest of my email.

)


* Second, when I run the following command:

  stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

I get the following warnings:

> WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl

(repeated one more time:)

> WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl

and then finally:

> ERROR - We were unable to successfully update 2 dependencies in environment
> ERROR - Error

The virtual machine has NASM installed (2.13.03-2.el8) and IASL too
(acpica-tools-20180629-3.el8).

Where do the NASM and IASL version requirements (2.14.02 and
20190215.0.0, respectively) come from?

Hm... After a git-grep for those version numbers, I find:

- BaseTools/Bin/nasm_ext_dep.yaml
- BaseTools/Bin/iasl_ext_dep.yaml

I was about to say that these version requirements are too strict: for
example, "BaseTools/Conf/tools_def.template" requires "NASM 2.10 or
later for use with the GCC toolchain family". What I have installed
satisfies that, and so CI shouldn't require anything more recent.
*However*, both of the above YAML files have very helpful comments, so I
understand these high versions are downloaded afresh, and only for the
CI run.

And so my question becomes: why do the "nuget" downloads fail for me
(because, presumably, they work fine in the central CI env on github /
Azure); and how can I fix the issue if it pops up again?

I've checked "nuget.org" in my browser, and it has:

- https://www.nuget.org/packages/mu_nasm/ -->  2.14.2
- https://www.nuget.org/packages/iasl/ -->  20190215.0.0

... On a hunch, I've attempted adding the "--verbose" option to the
"stuart_update" command; this is the output (excerpt):

> DEBUG - Verify 'mu_nasm' returning 'False'.
> DEBUG - Verify 'iasl' returning 'False'.
> DEBUG - Creating 4 threads for the SDE update
> UpdatingDEBUG - Verify 'mu_nasm' returning 'False'.
> DEBUG - Verify 'gcc_aarch64_linux' returning 'True'.
> DEBUG - Cleaning dependency directory for 'mu_nasm'...
> DEBUG - Verify 'gcc_arm_linux' returning 'True'.
> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
> DEBUG - Verify 'iasl' returning 'False'.
> INFO - ------------------------------------------------
> DEBUG - Cleaning dependency directory for 'iasl'...
> DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.
> INFO - --------------Cmd Output Starting---------------
> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
> INFO - ------------------------------------------------
> INFO - ------------------------------------------------
> INFO - --------------Cmd Output Starting---------------
> INFO - ------------------------------------------------
> INFO - /bin/sh: mono: command not found
> INFO - /bin/sh: mono: command not found
> INFO - ------------------------------------------------
> INFO - --------------Cmd Output Finished---------------
> INFO - --------- Running Time (mm:ss): 00:00 ----------
> INFO - ----------- Return Code: 0x0000007f ------------
> INFO - ------------------------------------------------
> INFO - Nuget was unable to provide global packages cache location.
> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.14.02 -Verbosity detailed -OutputDirectory "/root/src/rhel8/edk2/BaseTools/Bin/mu_nasm_extdep_temp"
> INFO - ------------------------------------------------
> INFO - --------------Cmd Output Starting---------------
> INFO - ------------------------------------------------
> INFO - ------------------------------------------------
> INFO - --------------Cmd Output Finished---------------
> INFO - --------- Running Time (mm:ss): 00:00 ----------
> INFO - ----------- Return Code: 0x0000007f ------------
> INFO - ------------------------------------------------
> INFO - Nuget was unable to provide global packages cache location.
> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install iasl -Source https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 20190215.0.0 -Verbosity detailed -OutputDirectory "/root/src/rhel8/edk2/BaseTools/Bin/iasl_extdep_temp"
> INFO - ------------------------------------------------
> INFO - --------------Cmd Output Starting---------------
> INFO - /bin/sh: mono: command not found
> INFO - ------------------------------------------------
> INFO - ------------------------------------------------
> INFO - --------------Cmd Output Finished---------------
> INFO - --------- Running Time (mm:ss): 00:00 ----------
> INFO - ----------- Return Code: 0x0000007f ------------
> INFO - ------------------------------------------------
> WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm

Aha! So "mono" is another requirement.

After installing the "mono-complete" package from EPEL8 (together with
its dependencies), the "stuart_update" command completes fine for me.

Can we spell out the "mono" dependency in ".pytool/Readme.md" somewhere,
or can we perhaps check for mono automatically?


* Third, attempting

  stuart_ci_build -c .pytool/CISettings.py -p OvmfPkg -a IA32,X64 -t DEBUG TOOL_CHAIN_TAG=GCC5

doesn't seem to do anything:

> SECTION - Init SDE
> SECTION - Loading Plugins
> SECTION - Start Invocable Tool
> SECTION - Getting Environment
> SECTION - Loading plugins
> SECTION - Building OvmfPkg Package
> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> PROGRESS - Overall Build Status: Success
> SECTION - Summary
> PROGRESS - Success

If I remove the "-t DEBUG" option, a bit more happens (but it also
fails):

> SECTION - Init SDE
> SECTION - Loading Plugins
> SECTION - Start Invocable Tool
> SECTION - Getting Environment
> SECTION - Loading plugins
> SECTION - Building OvmfPkg Package
> PROGRESS - --Running OvmfPkg: Char Encoding Check Test NO-TARGET --
> PROGRESS - --->Test Success: Char Encoding Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> PROGRESS - --Running OvmfPkg: Dependency Check Test NO-TARGET --
> PROGRESS - --->Test Success: Dependency Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: Dsc Complete Check Test NO-TARGET --
> PROGRESS - --->Test Success: Dsc Complete Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: Guid Check Test NO-TARGET --
> PROGRESS - --->Test Success: Guid Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: Library Class Check Test NO-TARGET --
> PROGRESS - --->Test Success: Library Class Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
> WARNING - NodeJs not installed. Test can't run
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: EccCheck Test NO-TARGET --
> PROGRESS - --->Test Success: EccCheck Test NO-TARGET
> PROGRESS - --Running OvmfPkg: License Check Test NO-TARGET --
> PROGRESS - --->Test Success: License Check Test NO-TARGET
> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> PROGRESS - --Running OvmfPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET
> PROGRESS - Overall Build Status: Success
> SECTION - Summary
> PROGRESS - Success

I'm not sure about the "NodeJs not installed. Test can't run" message --
I guess it only relates to the "Spell Check Test".

But what's missing for the actual build? I've noticed the following
warning:

> Test Skipped: in plugin! Compiler Plugin DEBUG

But even if I add "--verbose", the only lines around that warning are:

> INFO - Running on Package: OvmfPkg
> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> PROGRESS - Overall Build Status: Success

Why is the build skipped?

--*--

Summary:

- the typo in "pip install --upgrade -r pip-requirements.txt" should be fixed

- "mono" should be documented, or auto-installed, as a dependency

- I don't know why "stuart_ci_build" skips the actual build.

Thanks!
Laszlo

Re: [edk2-devel] running CI locally

[Michael D Kinney]

Hi Laszlo,

Thank you for the feedback.  I agree there are some documentation updates required.

The spell check requirements are documented here:

https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#spell-checking---cspell

You do need to install nodejs and cspell.

* Install nodejs from https://nodejs.org/en/
* Install cspell
  1. Open cmd prompt with access to node and npm
  2. Run `npm install -g cspell`

I agree that the verbosity of the output is very brief.  There is a more complete
log that is always produced in the build output directory in Build/CI_BUILDLOG.txt.
Can you look at that file and see if it provide a more detailed reason for the
failure?

Thanks,

Mike


> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
> Sent: Wednesday, August 19, 2020 5:27 AM
> To: edk2-devel-groups-io <devel@edk2.groups.io>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew
> <Bret.Barkelew@microsoft.com>
> Cc: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>; Rebecca Cran <rebecca@bsdio.com>; Tom Lendacky
> <thomas.lendacky@amd.com>
> Subject: [edk2-devel] running CI locally
> 
> Hi!
> 
> I'd like to test CI locally. I'm going through ".pytool/Readme.md" with
> the tree checked out at 7e6f150b6902 (= current HEAD). I'm doing this in
> a RHEL8 VM, with a python3 virtual environment set up / entered.
> 
> 
> * My first note is that the command
> 
>   pip install --upgrade pip-requirements.txt
> 
> under "Prerequisets", has a small typo; it should be
> 
>   pip install --upgrade -r pip-requirements.txt
> 
> (the "-r" option is missing).
> 
> (
> 
> After adding "-r", the following components are now installed in my
> virtual env:
> 
> - edk2-pytool-library: 0.10.12
> - edk2-pytool-extensions: 0.13.9
> - antlr4-python3-runtime: 4.7.1
> - pyyaml: 5.3.1
> 
> Stating this because it might matter for the rest of my email.
> 
> )
> 
> 
> * Second, when I run the following command:
> 
>   stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5
> 
> I get the following warnings:
> 
> > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> > WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl
> 
> (repeated one more time:)
> 
> > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> > WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl
> 
> and then finally:
> 
> > ERROR - We were unable to successfully update 2 dependencies in environment
> > ERROR - Error
> 
> The virtual machine has NASM installed (2.13.03-2.el8) and IASL too
> (acpica-tools-20180629-3.el8).
> 
> Where do the NASM and IASL version requirements (2.14.02 and
> 20190215.0.0, respectively) come from?
> 
> Hm... After a git-grep for those version numbers, I find:
> 
> - BaseTools/Bin/nasm_ext_dep.yaml
> - BaseTools/Bin/iasl_ext_dep.yaml
> 
> I was about to say that these version requirements are too strict: for
> example, "BaseTools/Conf/tools_def.template" requires "NASM 2.10 or
> later for use with the GCC toolchain family". What I have installed
> satisfies that, and so CI shouldn't require anything more recent.
> *However*, both of the above YAML files have very helpful comments, so I
> understand these high versions are downloaded afresh, and only for the
> CI run.
> 
> And so my question becomes: why do the "nuget" downloads fail for me
> (because, presumably, they work fine in the central CI env on github /
> Azure); and how can I fix the issue if it pops up again?
> 
> I've checked "nuget.org" in my browser, and it has:
> 
> - https://www.nuget.org/packages/mu_nasm/ -->  2.14.2
> - https://www.nuget.org/packages/iasl/ -->  20190215.0.0
> 
> ... On a hunch, I've attempted adding the "--verbose" option to the
> "stuart_update" command; this is the output (excerpt):
> 
> > DEBUG - Verify 'mu_nasm' returning 'False'.
> > DEBUG - Verify 'iasl' returning 'False'.
> > DEBUG - Creating 4 threads for the SDE update
> > UpdatingDEBUG - Verify 'mu_nasm' returning 'False'.
> > DEBUG - Verify 'gcc_aarch64_linux' returning 'True'.
> > DEBUG - Cleaning dependency directory for 'mu_nasm'...
> > DEBUG - Verify 'gcc_arm_linux' returning 'True'.
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
> > DEBUG - Verify 'iasl' returning 'False'.
> > INFO - ------------------------------------------------
> > DEBUG - Cleaning dependency directory for 'iasl'...
> > DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.
> > INFO - --------------Cmd Output Starting---------------
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
> > INFO - ------------------------------------------------
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Starting---------------
> > INFO - ------------------------------------------------
> > INFO - /bin/sh: mono: command not found
> > INFO - /bin/sh: mono: command not found
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Finished---------------
> > INFO - --------- Running Time (mm:ss): 00:00 ----------
> > INFO - ----------- Return Code: 0x0000007f ------------
> > INFO - ------------------------------------------------
> > INFO - Nuget was unable to provide global packages cache location.
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source
> https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.14.02 -Verbosity detailed -OutputDirectory
> "/root/src/rhel8/edk2/BaseTools/Bin/mu_nasm_extdep_temp"
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Starting---------------
> > INFO - ------------------------------------------------
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Finished---------------
> > INFO - --------- Running Time (mm:ss): 00:00 ----------
> > INFO - ----------- Return Code: 0x0000007f ------------
> > INFO - ------------------------------------------------
> > INFO - Nuget was unable to provide global packages cache location.
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install iasl -Source
> https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 20190215.0.0 -Verbosity detailed -OutputDirectory
> "/root/src/rhel8/edk2/BaseTools/Bin/iasl_extdep_temp"
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Starting---------------
> > INFO - /bin/sh: mono: command not found
> > INFO - ------------------------------------------------
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Finished---------------
> > INFO - --------- Running Time (mm:ss): 00:00 ----------
> > INFO - ----------- Return Code: 0x0000007f ------------
> > INFO - ------------------------------------------------
> > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> 
> Aha! So "mono" is another requirement.
> 
> After installing the "mono-complete" package from EPEL8 (together with
> its dependencies), the "stuart_update" command completes fine for me.
> 
> Can we spell out the "mono" dependency in ".pytool/Readme.md" somewhere,
> or can we perhaps check for mono automatically?
> 
> 
> * Third, attempting
> 
>   stuart_ci_build -c .pytool/CISettings.py -p OvmfPkg -a IA32,X64 -t DEBUG TOOL_CHAIN_TAG=GCC5
> 
> doesn't seem to do anything:
> 
> > SECTION - Init SDE
> > SECTION - Loading Plugins
> > SECTION - Start Invocable Tool
> > SECTION - Getting Environment
> > SECTION - Loading plugins
> > SECTION - Building OvmfPkg Package
> > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> > PROGRESS - Overall Build Status: Success
> > SECTION - Summary
> > PROGRESS - Success
> 
> If I remove the "-t DEBUG" option, a bit more happens (but it also
> fails):
> 
> > SECTION - Init SDE
> > SECTION - Loading Plugins
> > SECTION - Start Invocable Tool
> > SECTION - Getting Environment
> > SECTION - Loading plugins
> > SECTION - Building OvmfPkg Package
> > PROGRESS - --Running OvmfPkg: Char Encoding Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Char Encoding Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> > PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> > PROGRESS - --Running OvmfPkg: Dependency Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Dependency Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Dsc Complete Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Dsc Complete Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Guid Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Guid Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Library Class Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Library Class Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
> > WARNING - NodeJs not installed. Test can't run
> > WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: EccCheck Test NO-TARGET --
> > PROGRESS - --->Test Success: EccCheck Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: License Check Test NO-TARGET --
> > PROGRESS - --->Test Success: License Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
> > WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> > PROGRESS - --Running OvmfPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET
> > PROGRESS - Overall Build Status: Success
> > SECTION - Summary
> > PROGRESS - Success
> 
> I'm not sure about the "NodeJs not installed. Test can't run" message --
> I guess it only relates to the "Spell Check Test".
> 
> But what's missing for the actual build? I've noticed the following
> warning:
> 
> > Test Skipped: in plugin! Compiler Plugin DEBUG
> 
> But even if I add "--verbose", the only lines around that warning are:
> 
> > INFO - Running on Package: OvmfPkg
> > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> > PROGRESS - Overall Build Status: Success
> 
> Why is the build skipped?
> 
> --*--
> 
> Summary:
> 
> - the typo in "pip install --upgrade -r pip-requirements.txt" should be fixed
> 
> - "mono" should be documented, or auto-installed, as a dependency
> 
> - I don't know why "stuart_ci_build" skips the actual build.
> 
> Thanks!
> Laszlo
> 
> 
> 

Re: [edk2-devel] running CI locally

[Sean]

Laszlo/Mike,

This is the joy of distributed repositories.
Remember edk2 ci is actually using edk2-pytool-extensions and 
edk2-pytool-library.  Documentation is in those projects.

https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs
https://github.com/tianocore/edk2-pytool-library/tree/master/docs

I won't say they are great and I hope someday relatively soon we can 
talk about an edk2 static site generator that can include docs from 
multiple repositories (much like https://microsoft.github.io/mu/) as I 
think documentation on edk2 is a weak spot.

Regarding Mono and nuget.  It is one of those things we wish was 
different as we have found it very inconsistent across different 
distributions of linux.  But we do have docs here.
https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_linux.md

And more specifically here:
https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_extdep.md#a-note-on-nuget-on-linux


Finally getting to why you don't run the compile.
Core CI (stuart_ci_build)  is a plugin runner.  Compile test is just one 
of those tests.  OvmfPkg is a platform and thus didn't opt into core ci 
compile testing.  In the table here i tried to document that OvmfPkg 
didn't compile using core ci and that a user should look at the readme.
https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#basic-status 


So for OvmfPkg we enabled what we call platform ci (stuart_build).  I 
think the write up here is pretty complete (although i see it has no 
mention of mono either). 
https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI

One final note.  Yes the logging to console is by default very brief. 
This is by design as it is easier to quickly look and see what test 
failed and then use the log file (as mike mentioned) to find the root 
cause.

Hope that helps and it is great to see people using it.  Feedback is 
much appreciated.

Thanks
Sean

On 8/19/2020 8:29 AM, Michael D Kinney wrote:
> Hi Laszlo,
> 
> Thank you for the feedback.  I agree there are some documentation updates required.
> 
> The spell check requirements are documented here:
> 
> https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#spell-checking---cspell
> 
> You do need to install nodejs and cspell.
> 
> * Install nodejs from https://nodejs.org/en/
> * Install cspell
>    1. Open cmd prompt with access to node and npm
>    2. Run `npm install -g cspell`
> 
> I agree that the verbosity of the output is very brief.  There is a more complete
> log that is always produced in the build output directory in Build/CI_BUILDLOG.txt.
> Can you look at that file and see if it provide a more detailed reason for the
> failure?
> 
> Thanks,
> 
> Mike
> 
> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
>> Sent: Wednesday, August 19, 2020 5:27 AM
>> To: edk2-devel-groups-io <devel@edk2.groups.io>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew
>> <Bret.Barkelew@microsoft.com>
>> Cc: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>; Rebecca Cran <rebecca@bsdio.com>; Tom Lendacky
>> <thomas.lendacky@amd.com>
>> Subject: [edk2-devel] running CI locally
>>
>> Hi!
>>
>> I'd like to test CI locally. I'm going through ".pytool/Readme.md" with
>> the tree checked out at 7e6f150b6902 (= current HEAD). I'm doing this in
>> a RHEL8 VM, with a python3 virtual environment set up / entered.
>>
>>
>> * My first note is that the command
>>
>>    pip install --upgrade pip-requirements.txt
>>
>> under "Prerequisets", has a small typo; it should be
>>
>>    pip install --upgrade -r pip-requirements.txt
>>
>> (the "-r" option is missing).
>>
>> (
>>
>> After adding "-r", the following components are now installed in my
>> virtual env:
>>
>> - edk2-pytool-library: 0.10.12
>> - edk2-pytool-extensions: 0.13.9
>> - antlr4-python3-runtime: 4.7.1
>> - pyyaml: 5.3.1
>>
>> Stating this because it might matter for the rest of my email.
>>
>> )
>>
>>
>> * Second, when I run the following command:
>>
>>    stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5
>>
>> I get the following warnings:
>>
>>> WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
>>> WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl
>>
>> (repeated one more time:)
>>
>>> WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
>>> WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl
>>
>> and then finally:
>>
>>> ERROR - We were unable to successfully update 2 dependencies in environment
>>> ERROR - Error
>>
>> The virtual machine has NASM installed (2.13.03-2.el8) and IASL too
>> (acpica-tools-20180629-3.el8).
>>
>> Where do the NASM and IASL version requirements (2.14.02 and
>> 20190215.0.0, respectively) come from?
>>
>> Hm... After a git-grep for those version numbers, I find:
>>
>> - BaseTools/Bin/nasm_ext_dep.yaml
>> - BaseTools/Bin/iasl_ext_dep.yaml
>>
>> I was about to say that these version requirements are too strict: for
>> example, "BaseTools/Conf/tools_def.template" requires "NASM 2.10 or
>> later for use with the GCC toolchain family". What I have installed
>> satisfies that, and so CI shouldn't require anything more recent.
>> *However*, both of the above YAML files have very helpful comments, so I
>> understand these high versions are downloaded afresh, and only for the
>> CI run.
>>
>> And so my question becomes: why do the "nuget" downloads fail for me
>> (because, presumably, they work fine in the central CI env on github /
>> Azure); and how can I fix the issue if it pops up again?
>>
>> I've checked "nuget.org" in my browser, and it has:
>>
>> - https://www.nuget.org/packages/mu_nasm/ -->  2.14.2
>> - https://www.nuget.org/packages/iasl/ -->  20190215.0.0
>>
>> ... On a hunch, I've attempted adding the "--verbose" option to the
>> "stuart_update" command; this is the output (excerpt):
>>
>>> DEBUG - Verify 'mu_nasm' returning 'False'.
>>> DEBUG - Verify 'iasl' returning 'False'.
>>> DEBUG - Creating 4 threads for the SDE update
>>> UpdatingDEBUG - Verify 'mu_nasm' returning 'False'.
>>> DEBUG - Verify 'gcc_aarch64_linux' returning 'True'.
>>> DEBUG - Cleaning dependency directory for 'mu_nasm'...
>>> DEBUG - Verify 'gcc_arm_linux' returning 'True'.
>>> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
>>> DEBUG - Verify 'iasl' returning 'False'.
>>> INFO - ------------------------------------------------
>>> DEBUG - Cleaning dependency directory for 'iasl'...
>>> DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.
>>> INFO - --------------Cmd Output Starting---------------
>>> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
>>> INFO - ------------------------------------------------
>>> INFO - ------------------------------------------------
>>> INFO - --------------Cmd Output Starting---------------
>>> INFO - ------------------------------------------------
>>> INFO - /bin/sh: mono: command not found
>>> INFO - /bin/sh: mono: command not found
>>> INFO - ------------------------------------------------
>>> INFO - --------------Cmd Output Finished---------------
>>> INFO - --------- Running Time (mm:ss): 00:00 ----------
>>> INFO - ----------- Return Code: 0x0000007f ------------
>>> INFO - ------------------------------------------------
>>> INFO - Nuget was unable to provide global packages cache location.
>>> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source
>> https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.14.02 -Verbosity detailed -OutputDirectory
>> "/root/src/rhel8/edk2/BaseTools/Bin/mu_nasm_extdep_temp"
>>> INFO - ------------------------------------------------
>>> INFO - --------------Cmd Output Starting---------------
>>> INFO - ------------------------------------------------
>>> INFO - ------------------------------------------------
>>> INFO - --------------Cmd Output Finished---------------
>>> INFO - --------- Running Time (mm:ss): 00:00 ----------
>>> INFO - ----------- Return Code: 0x0000007f ------------
>>> INFO - ------------------------------------------------
>>> INFO - Nuget was unable to provide global packages cache location.
>>> INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install iasl -Source
>> https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 20190215.0.0 -Verbosity detailed -OutputDirectory
>> "/root/src/rhel8/edk2/BaseTools/Bin/iasl_extdep_temp"
>>> INFO - ------------------------------------------------
>>> INFO - --------------Cmd Output Starting---------------
>>> INFO - /bin/sh: mono: command not found
>>> INFO - ------------------------------------------------
>>> INFO - ------------------------------------------------
>>> INFO - --------------Cmd Output Finished---------------
>>> INFO - --------- Running Time (mm:ss): 00:00 ----------
>>> INFO - ----------- Return Code: 0x0000007f ------------
>>> INFO - ------------------------------------------------
>>> WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
>>
>> Aha! So "mono" is another requirement.
>>
>> After installing the "mono-complete" package from EPEL8 (together with
>> its dependencies), the "stuart_update" command completes fine for me.
>>
>> Can we spell out the "mono" dependency in ".pytool/Readme.md" somewhere,
>> or can we perhaps check for mono automatically?
>>
>>
>> * Third, attempting
>>
>>    stuart_ci_build -c .pytool/CISettings.py -p OvmfPkg -a IA32,X64 -t DEBUG TOOL_CHAIN_TAG=GCC5
>>
>> doesn't seem to do anything:
>>
>>> SECTION - Init SDE
>>> SECTION - Loading Plugins
>>> SECTION - Start Invocable Tool
>>> SECTION - Getting Environment
>>> SECTION - Loading plugins
>>> SECTION - Building OvmfPkg Package
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - Overall Build Status: Success
>>> SECTION - Summary
>>> PROGRESS - Success
>>
>> If I remove the "-t DEBUG" option, a bit more happens (but it also
>> fails):
>>
>>> SECTION - Init SDE
>>> SECTION - Loading Plugins
>>> SECTION - Start Invocable Tool
>>> SECTION - Getting Environment
>>> SECTION - Loading plugins
>>> SECTION - Building OvmfPkg Package
>>> PROGRESS - --Running OvmfPkg: Char Encoding Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: Char Encoding Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>> PROGRESS - --Running OvmfPkg: Dependency Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: Dependency Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: Dsc Complete Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: Dsc Complete Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: Guid Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: Guid Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: Library Class Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: Library Class Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
>>> WARNING - NodeJs not installed. Test can't run
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: EccCheck Test NO-TARGET --
>>> PROGRESS - --->Test Success: EccCheck Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: License Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: License Check Test NO-TARGET
>>> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
>>> PROGRESS - --Running OvmfPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>>> PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET
>>> PROGRESS - Overall Build Status: Success
>>> SECTION - Summary
>>> PROGRESS - Success
>>
>> I'm not sure about the "NodeJs not installed. Test can't run" message --
>> I guess it only relates to the "Spell Check Test".
>>
>> But what's missing for the actual build? I've noticed the following
>> warning:
>>
>>> Test Skipped: in plugin! Compiler Plugin DEBUG
>>
>> But even if I add "--verbose", the only lines around that warning are:
>>
>>> INFO - Running on Package: OvmfPkg
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - Overall Build Status: Success
>>
>> Why is the build skipped?
>>
>> --*--
>>
>> Summary:
>>
>> - the typo in "pip install --upgrade -r pip-requirements.txt" should be fixed
>>
>> - "mono" should be documented, or auto-installed, as a dependency
>>
>> - I don't know why "stuart_ci_build" skips the actual build.
>>
>> Thanks!
>> Laszlo
>>
>>
>>
> 
> 
> 
> 

Re: [edk2-devel] running CI locally

[Laszlo Ersek]

Hi Mike,

On 08/19/20 17:29, Kinney, Michael D wrote:
> Hi Laszlo,
> 
> Thank you for the feedback.  I agree there are some documentation updates required.
> 
> The spell check requirements are documented here:
> 
> https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#spell-checking---cspell
> 
> You do need to install nodejs and cspell.
> 
> * Install nodejs from https://nodejs.org/en/
> * Install cspell
>   1. Open cmd prompt with access to node and npm
>   2. Run `npm install -g cspell`

npm is a security hole one can drive a truck through :)

Additionally, I just realized that "nuget" downloads native iasl and
nasm binaries ("BaseTools/Bin/iasl_extdep/Linux-x86/iasl",
"BaseTools/Bin/mu_nasm_extdep/Linux-x86-64/nasm").

This basically requires me to trust two package repositories (npmjs.org,
nuget.org) that, well, I don't trust.

This is *not* criticism of the CI system; it only means that running it
locally on my laptop needs more work from me. In particular, it requires
spinning up a VM that I don't use for anything else.

Given the typing / clicking / scripting necessary for this, and the
network-originated updates for the "in VM" CI system, I wonder if
running CI locally actually saves me any time.

One thing it would certainly give me is "confidentiality", as I wouldn't
have to push my branch to github (for a personal CI build PR) before I
were ready to submit the set for real.

I think I'll set up a new libvirt/QEMU/KVM domain for this, and report
back with more results once I have them.

> I agree that the verbosity of the output is very brief.  There is a more complete
> log that is always produced in the build output directory in Build/CI_BUILDLOG.txt.
> Can you look at that file and see if it provide a more detailed reason for the
> failure?

Yes, I'll check that.

Thanks!
Laszlo

Re: [edk2-devel] running CI locally

[Laszlo Ersek]

On 08/19/20 18:22, Sean Brogan wrote:
> Laszlo/Mike,
> 
> This is the joy of distributed repositories.
> Remember edk2 ci is actually using edk2-pytool-extensions and
> edk2-pytool-library.  Documentation is in those projects.
> 
> https://github.com/tianocore/edk2-pytool-extensions/tree/master/docs
> https://github.com/tianocore/edk2-pytool-library/tree/master/docs
> 
> I won't say they are great and I hope someday relatively soon we can
> talk about an edk2 static site generator that can include docs from
> multiple repositories (much like https://microsoft.github.io/mu/) as I
> think documentation on edk2 is a weak spot.
> 
> Regarding Mono and nuget.  It is one of those things we wish was
> different as we have found it very inconsistent across different
> distributions of linux.  But we do have docs here.
> https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_linux.md
> 
> 
> And more specifically here:
> https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_extdep.md#a-note-on-nuget-on-linux
> 
> 
> 
> Finally getting to why you don't run the compile.
> Core CI (stuart_ci_build)  is a plugin runner.  Compile test is just one
> of those tests.  OvmfPkg is a platform and thus didn't opt into core ci
> compile testing.  In the table here i tried to document that OvmfPkg
> didn't compile using core ci and that a user should look at the readme.
> https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#basic-status
> 
> 
> So for OvmfPkg we enabled what we call platform ci (stuart_build).  I
> think the write up here is pretty complete (although i see it has no
> mention of mono either).
> https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI
> 
> One final note.  Yes the logging to console is by default very brief.
> This is by design as it is easier to quickly look and see what test
> failed and then use the log file (as mike mentioned) to find the root
> cause.
> 
> Hope that helps and it is great to see people using it.  Feedback is
> much appreciated.

The documentation looks *awesome*, and it's entirely my fault that I
couldn't find it. I apologize.

I'll report back with more results.

Thanks!
Laszlo

Re: [edk2-devel] running CI locally

[Laszlo Ersek]

Hi Mike, Sean;

On 08/19/20 19:59, Laszlo Ersek wrote:

> I'll report back with more results.

I installed a Fedora 32 Server VM.

Installed the "mono-complete" package (+its deps).

Installed the "nodejs" package.

Installed "cspell" (+deps) with "npm" (with the latter coming from the
nodejs package).

Edk2 checked out at current HEAD
(5a6d764e1d073d28e8f398289ccb5592bf9a72ba).


* Platform CI results:

- <https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI>:

  Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5).

- <https://github.com/tianocore/edk2/tree/master/ArmVirtPkg/PlatformCI>:

  Built successfully for ARM and AARCH64 (toolchain: GCC5).


* Core CI results:

- <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>

  Complete run successful (toolchain: GCC5); no package / arch / target
  / test restrictions.

  I got the following WARNINGs:

  - ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not
    Core CI -- these are expected:

> PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE

> PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE

> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE

  - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
    NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are
    not covered by host-based unit tests -- I think also expected:

> PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

> PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT

  - I'm not really sure about the following warnings. They were emitted
    for a subset of the above packages. I read the docs at
    <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck>
    but I still don't understand :)

> PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET

> PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET

> PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET

> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET

> PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET

> PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET

  - Still related to host-based unit tests, I believe the following
    warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and
    UnitTestFrameworkPkg, report that running the tests on the build
    host (understandably) prevents the tests from covering all edk2
    architectures:

> PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - Allowing Override for key TARGET_ARCH

> PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - Allowing Override for key TARGET_ARCH

> PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - Allowing Override for key TARGET_ARCH

> PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - Allowing Override for key TARGET_ARCH

> PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler Plugin NOOPT --
> WARNING - Allowing Override for key TARGET_ARCH

    - The below seems to be related to the "spell checking in audit
      mode" known issues at
      <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.

      What does "audit mode" mean?

      FWIW, the packages listed below (EmulatorPkg, MdeModulePkg,
      MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper
      subset of the packges noted in ".pytool/Readme.md" -- the latter
      mentions CryptoPkg and SecurityPkg in addition.

> PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

> PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

> PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

> PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

> PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

> PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET

  - Not sure about the last one:

> PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET --
> WARNING - --->Test Skipped: in plugin! Library Class Check Test NO-TARGET


* One suggestion (in addition to the others made in this thread):

- Please add "BaseToolsBuild" to "BaseTools/.gitignore".


* A question:

- On github, there is some logic that restricts the full Core CI build
  to packages that may be affected by a patch series. Is this available
  locally?

  According to the Core CI documentation, the "stuart_ci_build" command
  can be limited with "-p" / "-a" / "-t" flags; given a commit range,
  how do we calculate the "tightest" flag values?

Thank you very much!
Laszlo

Re: [edk2-devel] running CI locally

[Sean]

Laszlo,



Idea being if the package supports host based unit tests it will scan 
package to make sure all host based unit tests and libraries are listed 
in the DSC.

As to why you see a different set of packages...that is interesting.  If 
you can include your full build logs i can look a little closer.  I know 
on the Azure CI servers we don't run the


On 8/21/2020 12:23 AM, Laszlo Ersek wrote:
> Hi Mike, Sean;
> 
> On 08/19/20 19:59, Laszlo Ersek wrote:
> 
>> I'll report back with more results.
> 
> I installed a Fedora 32 Server VM.
> 
> Installed the "mono-complete" package (+its deps).
> 
> Installed the "nodejs" package.
> 
> Installed "cspell" (+deps) with "npm" (with the latter coming from the
> nodejs package).
> 
> Edk2 checked out at current HEAD
> (5a6d764e1d073d28e8f398289ccb5592bf9a72ba).
> 
> 
> * Platform CI results:
> 
> - <https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI>:
> 
>    Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5).
> 
> - <https://github.com/tianocore/edk2/tree/master/ArmVirtPkg/PlatformCI>:
> 
>    Built successfully for ARM and AARCH64 (toolchain: GCC5).
> 
> 
> * Core CI results:
> 
> - <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>
> 
>    Complete run successful (toolchain: GCC5); no package / arch / target
>    / test restrictions.
> 
>    I got the following WARNINGs:
> 
>    - ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not
>      Core CI -- these are expected:
> 
>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> 
>> PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>> PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> 
>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> 
>    - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
>      NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are
>      not covered by host-based unit tests -- I think also expected:
> 
>> PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>> PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> 
>    - I'm not really sure about the following warnings. They were emitted
>      for a subset of the above packages. I read the docs at
>      <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck>
>      but I still don't understand :)

Each plugin/test has a readme as well.  Not sure if this helps explain 
the HostUnitTestDscCompleteCheck more.

https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck

As to why you don't see it skipped for ArmVirtPkg or a few others: I 
think this is a bug.

In Azure CI i see it as test passed.

PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check Test 
NO-TARGET --
PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test 
NO-TARGET

When i look at the code
https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59

I see that how those packages *.ci.yaml have it configured is different 
than those that show skipped below.  It avoids the skip conditions but 
since those packages don't have any host unit test libraries or modules 
there is no failure.

This should probably be a bugzilla on the 
HostUnitTestDscCompleteCheck.py as the reporting is not correct.  Test 
should show up as skipped.


You can see them here showing up as success
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=11309&view=logs&j=780eaa17-fc2d-5131-b9d2-21b19bc9bec5&t=0791c90f-e7d8-5372-c9c5-edbe30bf6aeb&l=32

> 
>> PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>> PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete Check Test NO-TARGET
> 
>    - Still related to host-based unit tests, I believe the following
>      warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and
>      UnitTestFrameworkPkg, report that running the tests on the build
>      host (understandably) prevents the tests from covering all edk2
>      architectures:

Yes this message is because we override the arch for NOOPT to match that 
of the build system since we are going to run these executables in the 
host environment.

> 
>> PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>> PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler Plugin NOOPT --
>> WARNING - Allowing Override for key TARGET_ARCH
> 
>      - The below seems to be related to the "spell checking in audit
>        mode" known issues at
>        <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.
> 
>        What does "audit mode" mean?

see readme here.
https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration

Basically runs the tests and reports all the errors as warnings and then 
reports the test as skipped.
It was hopefully to encourage package maintainers to fix their spelling 
issues but not breaking the build on day 1.


> 
>        FWIW, the packages listed below (EmulatorPkg, MdeModulePkg,
>        MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper
>        subset of the packges noted in ".pytool/Readme.md" -- the latter
>        mentions CryptoPkg and SecurityPkg in addition.
> 
>> PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>> PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> 
>    - Not sure about the last one:
> 
>> PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET --
>> WARNING - --->Test Skipped: in plugin! Library Class Check Test NO-TARGET
> 
> 
> * One suggestion (in addition to the others made in this thread):
> 
> - Please add "BaseToolsBuild" to "BaseTools/.gitignore".

There is a bugzilla for this already.
https://bugzilla.tianocore.org/show_bug.cgi?id=2581


> 
> 
> * A question:
> 
> - On github, there is some logic that restricts the full Core CI build
>    to packages that may be affected by a patch series. Is this available
>    locally?
> 
>    According to the Core CI documentation, the "stuart_ci_build" command
>    can be limited with "-p" / "-a" / "-t" flags; given a commit range,
>    how do we calculate the "tightest" flag values?

A couple things here.
You can always look at the azurepipelines files to get a better 
understanding of the CI process.  Hopefully there is no magic (besides 
being able to read yaml).  You can see here: 
https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35

It is actually another stuart tool.
stuart_pr_eval.
We have bug to document it.
https://github.com/tianocore/edk2-pytool-extensions/issues/89
But the python file is pretty well documented. There are 4 policies that 
can trigger the need for a change to build a package.
See here
https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117



> 
> Thank you very much!
> Laszlo
> 

Re: [edk2-devel] running CI locally

[Sean]

sorry.  In my last email I started typing at top and then moved inline 
and didn't delete my initial comments.  Please ignore the comments on 
top and look inline for details. :)

Thanks
Sean




On 8/21/2020 2:36 PM, Sean wrote:
> Laszlo,
> 
> 
> 
> Idea being if the package supports host based unit tests it will scan 
> package to make sure all host based unit tests and libraries are listed 
> in the DSC.
> 
> As to why you see a different set of packages...that is interesting.  If 
> you can include your full build logs i can look a little closer.  I know 
> on the Azure CI servers we don't run the
> 
> 
> On 8/21/2020 12:23 AM, Laszlo Ersek wrote:
>> Hi Mike, Sean;
>>
>> On 08/19/20 19:59, Laszlo Ersek wrote:
>>
>>> I'll report back with more results.
>>
>> I installed a Fedora 32 Server VM.
>>
>> Installed the "mono-complete" package (+its deps).
>>
>> Installed the "nodejs" package.
>>
>> Installed "cspell" (+deps) with "npm" (with the latter coming from the
>> nodejs package).
>>
>> Edk2 checked out at current HEAD
>> (5a6d764e1d073d28e8f398289ccb5592bf9a72ba).
>>
>>
>> * Platform CI results:
>>
>> - <https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI>:
>>
>>    Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5).
>>
>> - <https://github.com/tianocore/edk2/tree/master/ArmVirtPkg/PlatformCI>:
>>
>>    Built successfully for ARM and AARCH64 (toolchain: GCC5).
>>
>>
>> * Core CI results:
>>
>> - <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>
>>
>>    Complete run successful (toolchain: GCC5); no package / arch / target
>>    / test restrictions.
>>
>>    I got the following WARNINGs:
>>
>>    - ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not
>>      Core CI -- these are expected:
>>
>>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>
>>> PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
>>> PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
>>> WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
>>
>>    - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
>>      NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are
>>      not covered by host-based unit tests -- I think also expected:
>>
>>> PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>> PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin 
>>> NOOPT
>>
>>    - I'm not really sure about the following warnings. They were emitted
>>      for a subset of the above packages. I read the docs at
>>
>> <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck> 
>>
>>      but I still don't understand :)
> 
> Each plugin/test has a readme as well.  Not sure if this helps explain 
> the HostUnitTestDscCompleteCheck more.
> 
> https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck 
> 
> 
> As to why you don't see it skipped for ArmVirtPkg or a few others: I 
> think this is a bug.
> 
> In Azure CI i see it as test passed.
> 
> PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check Test 
> NO-TARGET --
> PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test 
> NO-TARGET
> 
> When i look at the code
> https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59 
> 
> 
> I see that how those packages *.ci.yaml have it configured is different 
> than those that show skipped below.  It avoids the skip conditions but 
> since those packages don't have any host unit test libraries or modules 
> there is no failure.
> 
> This should probably be a bugzilla on the 
> HostUnitTestDscCompleteCheck.py as the reporting is not correct.  Test 
> should show up as skipped.
> 
> 
> You can see them here showing up as success
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=11309&view=logs&j=780eaa17-fc2d-5131-b9d2-21b19bc9bec5&t=0791c90f-e7d8-5372-c9c5-edbe30bf6aeb&l=32 
> 
> 
>>
>>> PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check 
>>> Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test 
>>> NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check 
>>> Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check 
>>> Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>> PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test 
>>> NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete 
>>> Check Test NO-TARGET
>>
>>    - Still related to host-based unit tests, I believe the following
>>      warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and
>>      UnitTestFrameworkPkg, report that running the tests on the build
>>      host (understandably) prevents the tests from covering all edk2
>>      architectures:
> 
> Yes this message is because we override the arch for NOOPT to match that 
> of the build system since we are going to run these executables in the 
> host environment.
> 
>>
>>> PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin 
>>> NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>> PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler 
>>> Plugin NOOPT --
>>> WARNING - Allowing Override for key TARGET_ARCH
>>
>>      - The below seems to be related to the "spell checking in audit
>>        mode" known issues at
>>        <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.
>>
>>        What does "audit mode" mean?
> 
> see readme here.
> https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration 
> 
> 
> Basically runs the tests and reports all the errors as warnings and then 
> reports the test as skipped.
> It was hopefully to encourage package maintainers to fix their spelling 
> issues but not breaking the build on day 1.
> 
> 
>>
>>        FWIW, the packages listed below (EmulatorPkg, MdeModulePkg,
>>        MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper
>>        subset of the packges noted in ".pytool/Readme.md" -- the latter
>>        mentions CryptoPkg and SecurityPkg in addition.
>>
>>> PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>> PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
>>
>>    - Not sure about the last one:
>>
>>> PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET --
>>> WARNING - --->Test Skipped: in plugin! Library Class Check Test 
>>> NO-TARGET
>>
>>
>> * One suggestion (in addition to the others made in this thread):
>>
>> - Please add "BaseToolsBuild" to "BaseTools/.gitignore".
> 
> There is a bugzilla for this already.
> https://bugzilla.tianocore.org/show_bug.cgi?id=2581
> 
> 
>>
>>
>> * A question:
>>
>> - On github, there is some logic that restricts the full Core CI build
>>    to packages that may be affected by a patch series. Is this available
>>    locally?
>>
>>    According to the Core CI documentation, the "stuart_ci_build" command
>>    can be limited with "-p" / "-a" / "-t" flags; given a commit range,
>>    how do we calculate the "tightest" flag values?
> 
> A couple things here.
> You can always look at the azurepipelines files to get a better 
> understanding of the CI process.  Hopefully there is no magic (besides 
> being able to read yaml).  You can see here: 
> https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35 
> 
> 
> It is actually another stuart tool.
> stuart_pr_eval.
> We have bug to document it.
> https://github.com/tianocore/edk2-pytool-extensions/issues/89
> But the python file is pretty well documented. There are 4 policies that 
> can trigger the need for a change to build a package.
> See here
> https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117 
> 
> 
> 
> 
>>
>> Thank you very much!
>> Laszlo
>>
> 
> 
> 

Re: [edk2-devel] running CI locally

[Laszlo Ersek]

On 08/21/20 23:36, Sean Brogan wrote:
> On 8/21/2020 12:23 AM, Laszlo Ersek wrote:

>>    - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
>>      NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg
>>      are not covered by host-based unit tests -- I think also
>>      expected:
>>
>>> [...]
>>
>>    - I'm not really sure about the following warnings. They were
>>      emitted for a subset of the above packages. I read the docs at
>>
>> <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck>
>>
>>      but I still don't understand :)
>
> Each plugin/test has a readme as well.  Not sure if this helps explain
> the HostUnitTestDscCompleteCheck more.
>
> https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck

I've carefully re-read the explanation now, and (apparently) I do
understand it now :/

Before, I didn't realize that the expression

  "those related to host based unit tests"

stood for the set of INF files where each INF was
MODULE_TYPE=HOST_APPLICATION or
LIBRARY_CLASS=FoobarLib|HOST_APPLICATION.

> As to why you don't see it skipped for ArmVirtPkg or a few others: I
> think this is a bug.
>
> In Azure CI i see it as test passed.
>
> PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check
> Test NO-TARGET --
> PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test
> NO-TARGET
>
> When i look at the code
> https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59
>
>
> I see that how those packages *.ci.yaml have it configured is
> different than those that show skipped below.  It avoids the skip
> conditions but since those packages don't have any host unit test
> libraries or modules there is no failure.

"OvmfPkg/OvmfPkg.ci.yaml" (as an example) sets both
"HostUnitTestCompilerPlugin.DscPath" and
"HostUnitTestDscCompleteCheck.DscPath" to the empty string.

"ShellPkg/ShellPkg.ci.yaml" (an an example) sets neither attribute at
all.

Both HostUnitTestCompilerPlugin and HostUnitTestDscCompleteCheck check
for the existence of the DscPath attribute, as first step, so both
plugins warn about, and skip, ShellPkg.

OvmfPkg passes the same, first -- i.e., DscPath attribute existence --
check in both plugins. This suggests OvmfPkg is ruled out by *further*
checks in one plugin, and not ruled out by any further checks in the
other plugin.

The next (2nd) check in both plugins seems to be, whether we can
actually find the DSC file pointed-to by the DscPath attribute.
Furthermore, in HostUnitTestDscCompleteCheck, this check is the last one
(no other checks).

I think that this 2nd check *too* should fail for OvmfPkg in *both*
plugins. Because the DscPath attribute, while it exists, is empty.

I don't see why the "DSC path lookup" would fail in
HostUnitTestCompilerPlugin (and so we'd get a warning about OvmfPkg),
but not in HostUnitTestDscCompleteCheck (and so we'd get no warning
about OvmfPkg).

... Is it possible that the 2nd check actually fires in
HostUnitTestDscCompleteCheck too, but we suppress the warning by
returning 0, and not -1?

Compare the return values after the tc.SetSkipped() calls:

[https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestCompilerPlugin/HostUnitTestCompilerPlugin.py#L103]

100         if AP is None or AP_Path is None or not os.path.isfile(APDSC):
101             tc.SetSkipped()
102             tc.LogStdError("Package HostBasedUnitTest Dsc not found.")
103             return -1
                       ^^

versus

[https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L73]

70         if abs_dsc_path is None or wsr_dsc_path == "" or not os.path.isfile(abs_dsc_path):
71             tc.SetSkipped()
72             tc.LogStdError("Package Host Unit Test Dsc not found")
73             return 0
                      ^

I guess I could verify if the branch was taken at all, from the standard
error...

Ah, "Build/TestSuites.xml" seems relevant.

... Yes, I'm fairly sure the zero value after the "return" statement, on
line 73, is a typo. Because this is what I have in
"Build/TestSuites.xml" (excerpt re-wrapped here for readability):

> <testsuites>
>     <testsuite
>      id="14"
>      name="OvmfPkg"
>      package="Edk2CiBuild.Edk2.OvmfPkg"
>      errors="0"
>      tests="12"
>      failures="0"
>      skipped="6">
>
>         <testcase
>          classname="Edk2CiBuild.Edk2.OvmfPkg.HostUnitTestCompiler.X64"
>          name="Compile and Run Host-Based UnitTests for Edk2CiBuild.Edk2.OvmfPkg on arch X64"
>          time="0.0002715587615966797">
>             <skipped type="skipped"/>
>             <system-out/>
>             <system-err>Package HostBasedUnitTest Dsc not found.</system-err>
>         </testcase>
>
>         <testcase
>          classname="Edk2CiBuild.Edk2.OvmfPkg.HostUnitTestDscCompleteCheck"
>          name="Check the Edk2CiBuild.Edk2.OvmfPkg Host Unit Test DSC for a being complete"
>          time="0.00020742416381835938">
>             <skipped type="skipped"/>
>             <system-out/>
>             <system-err>Package Host Unit Test Dsc not found</system-err>
>         </testcase>
>
>     </testsuite>
> </testsuites>

Note that both quoted test cases (both plugins) report having been
skipped.

And from the <system-err> element, we can confirm that both plugins
catch OvmfPkg's DscPath attribute being the empty string. It's just that
"HostUnitTestDscCompleteCheck" returns zero on that branch.

> This should probably be a bugzilla on the
> HostUnitTestDscCompleteCheck.py as the reporting is not correct.  Test
> should show up as skipped.

I've now filed:

  https://bugzilla.tianocore.org/show_bug.cgi?id=2924

For now I've assigned it to you, and set BaseTools as the Package
needing the fix. Please update the metadata as necessary :)

>>      - The below seems to be related to the "spell checking in audit
>>        mode" known issues at
>>        <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.
>>
>>        What does "audit mode" mean?
>
> see readme here.
> https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration
>
>
> Basically runs the tests and reports all the errors as warnings and
> then reports the test as skipped. It was hopefully to encourage
> package maintainers to fix their spelling issues but not breaking the
> build on day 1.

Oh. In OvmfPkg, the CI YAML file says "Fails right now with over 270
errors" :/

>> * One suggestion (in addition to the others made in this thread):
>>
>> - Please add "BaseToolsBuild" to "BaseTools/.gitignore".
>
> There is a bugzilla for this already.
> https://bugzilla.tianocore.org/show_bug.cgi?id=2581

Thanks!

>> * A question:
>>
>> - On github, there is some logic that restricts the full Core CI
>>   build to packages that may be affected by a patch series. Is this
>>   available locally?
>>
>>   According to the Core CI documentation, the "stuart_ci_build"
>>   command can be limited with "-p" / "-a" / "-t" flags; given a
>>   commit range, how do we calculate the "tightest" flag values?
>
> A couple things here.
> You can always look at the azurepipelines files to get a better
> understanding of the CI process.  Hopefully there is no magic (besides
> being able to read yaml).  You can see here:
> https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35
>
>
> It is actually another stuart tool.
> stuart_pr_eval.
> We have bug to document it.
> https://github.com/tianocore/edk2-pytool-extensions/issues/89
> But the python file is pretty well documented. There are 4 policies
> that can trigger the need for a change to build a package.
> See here
> https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117

Cool, so I can run something like

  stuart_pr_eval -c .pytool/CISettings.py -p CANDIDATE-PACKAGE-LIST --pr-target TOPIC-BRANCH \
      [output formatting options]

I'll try that next time!

Thank you!
Laszlo