From: "Michael D Kinney" <michael.d.kinney@intel.com>
To: "devel@edk2.groups.io" <devel@edk2.groups.io>,
"lersek@redhat.com" <lersek@redhat.com>,
Sean Brogan <sean.brogan@microsoft.com>,
Bret Barkelew <Bret.Barkelew@microsoft.com>,
"Kinney, Michael D" <michael.d.kinney@intel.com>
Cc: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>,
Rebecca Cran <rebecca@bsdio.com>,
Tom Lendacky <thomas.lendacky@amd.com>
Subject: Re: [edk2-devel] running CI locally
Date: Wed, 19 Aug 2020 15:29:17 +0000 [thread overview]
Message-ID: <MN2PR11MB4461B620EC8521E3462D9BECD25D0@MN2PR11MB4461.namprd11.prod.outlook.com> (raw)
In-Reply-To: <d4c167bb-d050-e184-2e19-fc2f92c048ec@redhat.com>
Hi Laszlo,
Thank you for the feedback. I agree there are some documentation updates required.
The spell check requirements are documented here:
https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#spell-checking---cspell
You do need to install nodejs and cspell.
* Install nodejs from https://nodejs.org/en/
* Install cspell
1. Open cmd prompt with access to node and npm
2. Run `npm install -g cspell`
I agree that the verbosity of the output is very brief. There is a more complete
log that is always produced in the build output directory in Build/CI_BUILDLOG.txt.
Can you look at that file and see if it provide a more detailed reason for the
failure?
Thanks,
Mike
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
> Sent: Wednesday, August 19, 2020 5:27 AM
> To: edk2-devel-groups-io <devel@edk2.groups.io>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew
> <Bret.Barkelew@microsoft.com>
> Cc: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>; Rebecca Cran <rebecca@bsdio.com>; Tom Lendacky
> <thomas.lendacky@amd.com>
> Subject: [edk2-devel] running CI locally
>
> Hi!
>
> I'd like to test CI locally. I'm going through ".pytool/Readme.md" with
> the tree checked out at 7e6f150b6902 (= current HEAD). I'm doing this in
> a RHEL8 VM, with a python3 virtual environment set up / entered.
>
>
> * My first note is that the command
>
> pip install --upgrade pip-requirements.txt
>
> under "Prerequisets", has a small typo; it should be
>
> pip install --upgrade -r pip-requirements.txt
>
> (the "-r" option is missing).
>
> (
>
> After adding "-r", the following components are now installed in my
> virtual env:
>
> - edk2-pytool-library: 0.10.12
> - edk2-pytool-extensions: 0.13.9
> - antlr4-python3-runtime: 4.7.1
> - pyyaml: 5.3.1
>
> Stating this because it might matter for the rest of my email.
>
> )
>
>
> * Second, when I run the following command:
>
> stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5
>
> I get the following warnings:
>
> > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> > WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl
>
> (repeated one more time:)
>
> > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
> > WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl
>
> and then finally:
>
> > ERROR - We were unable to successfully update 2 dependencies in environment
> > ERROR - Error
>
> The virtual machine has NASM installed (2.13.03-2.el8) and IASL too
> (acpica-tools-20180629-3.el8).
>
> Where do the NASM and IASL version requirements (2.14.02 and
> 20190215.0.0, respectively) come from?
>
> Hm... After a git-grep for those version numbers, I find:
>
> - BaseTools/Bin/nasm_ext_dep.yaml
> - BaseTools/Bin/iasl_ext_dep.yaml
>
> I was about to say that these version requirements are too strict: for
> example, "BaseTools/Conf/tools_def.template" requires "NASM 2.10 or
> later for use with the GCC toolchain family". What I have installed
> satisfies that, and so CI shouldn't require anything more recent.
> *However*, both of the above YAML files have very helpful comments, so I
> understand these high versions are downloaded afresh, and only for the
> CI run.
>
> And so my question becomes: why do the "nuget" downloads fail for me
> (because, presumably, they work fine in the central CI env on github /
> Azure); and how can I fix the issue if it pops up again?
>
> I've checked "nuget.org" in my browser, and it has:
>
> - https://www.nuget.org/packages/mu_nasm/ --> 2.14.2
> - https://www.nuget.org/packages/iasl/ --> 20190215.0.0
>
> ... On a hunch, I've attempted adding the "--verbose" option to the
> "stuart_update" command; this is the output (excerpt):
>
> > DEBUG - Verify 'mu_nasm' returning 'False'.
> > DEBUG - Verify 'iasl' returning 'False'.
> > DEBUG - Creating 4 threads for the SDE update
> > UpdatingDEBUG - Verify 'mu_nasm' returning 'False'.
> > DEBUG - Verify 'gcc_aarch64_linux' returning 'True'.
> > DEBUG - Cleaning dependency directory for 'mu_nasm'...
> > DEBUG - Verify 'gcc_arm_linux' returning 'True'.
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
> > DEBUG - Verify 'iasl' returning 'False'.
> > INFO - ------------------------------------------------
> > DEBUG - Cleaning dependency directory for 'iasl'...
> > DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.
> > INFO - --------------Cmd Output Starting---------------
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list
> > INFO - ------------------------------------------------
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Starting---------------
> > INFO - ------------------------------------------------
> > INFO - /bin/sh: mono: command not found
> > INFO - /bin/sh: mono: command not found
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Finished---------------
> > INFO - --------- Running Time (mm:ss): 00:00 ----------
> > INFO - ----------- Return Code: 0x0000007f ------------
> > INFO - ------------------------------------------------
> > INFO - Nuget was unable to provide global packages cache location.
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source
> https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.14.02 -Verbosity detailed -OutputDirectory
> "/root/src/rhel8/edk2/BaseTools/Bin/mu_nasm_extdep_temp"
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Starting---------------
> > INFO - ------------------------------------------------
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Finished---------------
> > INFO - --------- Running Time (mm:ss): 00:00 ----------
> > INFO - ----------- Return Code: 0x0000007f ------------
> > INFO - ------------------------------------------------
> > INFO - Nuget was unable to provide global packages cache location.
> > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install iasl -Source
> https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 20190215.0.0 -Verbosity detailed -OutputDirectory
> "/root/src/rhel8/edk2/BaseTools/Bin/iasl_extdep_temp"
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Starting---------------
> > INFO - /bin/sh: mono: command not found
> > INFO - ------------------------------------------------
> > INFO - ------------------------------------------------
> > INFO - --------------Cmd Output Finished---------------
> > INFO - --------- Running Time (mm:ss): 00:00 ----------
> > INFO - ----------- Return Code: 0x0000007f ------------
> > INFO - ------------------------------------------------
> > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm
>
> Aha! So "mono" is another requirement.
>
> After installing the "mono-complete" package from EPEL8 (together with
> its dependencies), the "stuart_update" command completes fine for me.
>
> Can we spell out the "mono" dependency in ".pytool/Readme.md" somewhere,
> or can we perhaps check for mono automatically?
>
>
> * Third, attempting
>
> stuart_ci_build -c .pytool/CISettings.py -p OvmfPkg -a IA32,X64 -t DEBUG TOOL_CHAIN_TAG=GCC5
>
> doesn't seem to do anything:
>
> > SECTION - Init SDE
> > SECTION - Loading Plugins
> > SECTION - Start Invocable Tool
> > SECTION - Getting Environment
> > SECTION - Loading plugins
> > SECTION - Building OvmfPkg Package
> > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> > PROGRESS - Overall Build Status: Success
> > SECTION - Summary
> > PROGRESS - Success
>
> If I remove the "-t DEBUG" option, a bit more happens (but it also
> fails):
>
> > SECTION - Init SDE
> > SECTION - Loading Plugins
> > SECTION - Start Invocable Tool
> > SECTION - Getting Environment
> > SECTION - Loading plugins
> > SECTION - Building OvmfPkg Package
> > PROGRESS - --Running OvmfPkg: Char Encoding Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Char Encoding Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> > PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
> > PROGRESS - --Running OvmfPkg: Dependency Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Dependency Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Dsc Complete Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Dsc Complete Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Guid Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Guid Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Library Class Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Library Class Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
> > WARNING - NodeJs not installed. Test can't run
> > WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: EccCheck Test NO-TARGET --
> > PROGRESS - --->Test Success: EccCheck Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: License Check Test NO-TARGET --
> > PROGRESS - --->Test Success: License Check Test NO-TARGET
> > PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
> > WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
> > PROGRESS - --Running OvmfPkg: Host Unit Test Dsc Complete Check Test NO-TARGET --
> > PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET
> > PROGRESS - Overall Build Status: Success
> > SECTION - Summary
> > PROGRESS - Success
>
> I'm not sure about the "NodeJs not installed. Test can't run" message --
> I guess it only relates to the "Spell Check Test".
>
> But what's missing for the actual build? I've noticed the following
> warning:
>
> > Test Skipped: in plugin! Compiler Plugin DEBUG
>
> But even if I add "--verbose", the only lines around that warning are:
>
> > INFO - Running on Package: OvmfPkg
> > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
> > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
> > PROGRESS - Overall Build Status: Success
>
> Why is the build skipped?
>
> --*--
>
> Summary:
>
> - the typo in "pip install --upgrade -r pip-requirements.txt" should be fixed
>
> - "mono" should be documented, or auto-installed, as a dependency
>
> - I don't know why "stuart_ci_build" skips the actual build.
>
> Thanks!
> Laszlo
>
>
>
next prev parent reply other threads:[~2020-08-19 15:29 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-19 12:27 running CI locally Laszlo Ersek
2020-08-19 15:29 ` Michael D Kinney [this message]
2020-08-19 16:22 ` [edk2-devel] " Sean
2020-08-19 17:59 ` Laszlo Ersek
2020-08-21 7:23 ` Laszlo Ersek
2020-08-21 21:36 ` Sean
2020-08-24 12:35 ` Laszlo Ersek
[not found] ` <162D662A3371B4DC.7353@groups.io>
2020-08-21 21:50 ` Sean
2020-08-19 17:56 ` Laszlo Ersek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=MN2PR11MB4461B620EC8521E3462D9BECD25D0@MN2PR11MB4461.namprd11.prod.outlook.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox