From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.81]) by mx.groups.io with SMTP id smtpd.web11.87012.1597840047688592853 for ; Wed, 19 Aug 2020 05:27:27 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Yf7Nxx2+; spf=pass (domain: redhat.com, ip: 207.211.31.81, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1597840046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=oUmA7/b1QG9aueoLDAGCGsArcyhVHpjwH9zWq7ru5GQ=; b=Yf7Nxx2+ka5EQ08iBcZCJV3gFTGvgbzOSNC33lcPZs7ALbUVUbAdOqeQa2sQbkXep+YatP vIuHDrbD8Qhje6hE3Oib/Vg6gzZ3Uc8tcaL9lQAX+h2GaMM7P1lJ8e89AxeOs7wPG76ayh jki5Rq0RpynZaSQiBH39bcuNZsMoPgU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-579-eNgZAyAYMOqJ9ei-qhZCUA-1; Wed, 19 Aug 2020 08:27:11 -0400 X-MC-Unique: eNgZAyAYMOqJ9ei-qhZCUA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 212D52FD00; Wed, 19 Aug 2020 12:27:10 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-114-57.ams2.redhat.com [10.36.114.57]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8FBC17BE92; Wed, 19 Aug 2020 12:27:08 +0000 (UTC) To: edk2-devel-groups-io , Sean Brogan , Bret Barkelew Cc: Vladimir Olovyannikov , Rebecca Cran , Tom Lendacky From: "Laszlo Ersek" Subject: running CI locally Message-ID: Date: Wed, 19 Aug 2020 14:27:07 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Thunderbird/52.9.1 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=lersek@redhat.com X-Mimecast-Spam-Score: 0.001 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi! I'd like to test CI locally. I'm going through ".pytool/Readme.md" with the tree checked out at 7e6f150b6902 (= current HEAD). I'm doing this in a RHEL8 VM, with a python3 virtual environment set up / entered. * My first note is that the command pip install --upgrade pip-requirements.txt under "Prerequisets", has a small typo; it should be pip install --upgrade -r pip-requirements.txt (the "-r" option is missing). ( After adding "-r", the following components are now installed in my virtual env: - edk2-pytool-library: 0.10.12 - edk2-pytool-extensions: 0.13.9 - antlr4-python3-runtime: 4.7.1 - pyyaml: 5.3.1 Stating this because it might matter for the rest of my email. ) * Second, when I run the following command: stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 I get the following warnings: > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm > WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl (repeated one more time:) > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm > WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We failed to install this version 20190215.0.0 of iasl and then finally: > ERROR - We were unable to successfully update 2 dependencies in environment > ERROR - Error The virtual machine has NASM installed (2.13.03-2.el8) and IASL too (acpica-tools-20180629-3.el8). Where do the NASM and IASL version requirements (2.14.02 and 20190215.0.0, respectively) come from? Hm... After a git-grep for those version numbers, I find: - BaseTools/Bin/nasm_ext_dep.yaml - BaseTools/Bin/iasl_ext_dep.yaml I was about to say that these version requirements are too strict: for example, "BaseTools/Conf/tools_def.template" requires "NASM 2.10 or later for use with the GCC toolchain family". What I have installed satisfies that, and so CI shouldn't require anything more recent. *However*, both of the above YAML files have very helpful comments, so I understand these high versions are downloaded afresh, and only for the CI run. And so my question becomes: why do the "nuget" downloads fail for me (because, presumably, they work fine in the central CI env on github / Azure); and how can I fix the issue if it pops up again? I've checked "nuget.org" in my browser, and it has: - https://www.nuget.org/packages/mu_nasm/ --> 2.14.2 - https://www.nuget.org/packages/iasl/ --> 20190215.0.0 ... On a hunch, I've attempted adding the "--verbose" option to the "stuart_update" command; this is the output (excerpt): > DEBUG - Verify 'mu_nasm' returning 'False'. > DEBUG - Verify 'iasl' returning 'False'. > DEBUG - Creating 4 threads for the SDE update > UpdatingDEBUG - Verify 'mu_nasm' returning 'False'. > DEBUG - Verify 'gcc_aarch64_linux' returning 'True'. > DEBUG - Cleaning dependency directory for 'mu_nasm'... > DEBUG - Verify 'gcc_arm_linux' returning 'True'. > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list > DEBUG - Verify 'iasl' returning 'False'. > INFO - ------------------------------------------------ > DEBUG - Cleaning dependency directory for 'iasl'... > DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'. > INFO - --------------Cmd Output Starting--------------- > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list > INFO - ------------------------------------------------ > INFO - ------------------------------------------------ > INFO - --------------Cmd Output Starting--------------- > INFO - ------------------------------------------------ > INFO - /bin/sh: mono: command not found > INFO - /bin/sh: mono: command not found > INFO - ------------------------------------------------ > INFO - --------------Cmd Output Finished--------------- > INFO - --------- Running Time (mm:ss): 00:00 ---------- > INFO - ----------- Return Code: 0x0000007f ------------ > INFO - ------------------------------------------------ > INFO - Nuget was unable to provide global packages cache location. > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.14.02 -Verbosity detailed -OutputDirectory "/root/src/rhel8/edk2/BaseTools/Bin/mu_nasm_extdep_temp" > INFO - ------------------------------------------------ > INFO - --------------Cmd Output Starting--------------- > INFO - ------------------------------------------------ > INFO - ------------------------------------------------ > INFO - --------------Cmd Output Finished--------------- > INFO - --------- Running Time (mm:ss): 00:00 ---------- > INFO - ----------- Return Code: 0x0000007f ------------ > INFO - ------------------------------------------------ > INFO - Nuget was unable to provide global packages cache location. > INFO - Cmd to run is: mono /root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install iasl -Source https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 20190215.0.0 -Verbosity detailed -OutputDirectory "/root/src/rhel8/edk2/BaseTools/Bin/iasl_extdep_temp" > INFO - ------------------------------------------------ > INFO - --------------Cmd Output Starting--------------- > INFO - /bin/sh: mono: command not found > INFO - ------------------------------------------------ > INFO - ------------------------------------------------ > INFO - --------------Cmd Output Finished--------------- > INFO - --------- Running Time (mm:ss): 00:00 ---------- > INFO - ----------- Return Code: 0x0000007f ------------ > INFO - ------------------------------------------------ > WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We failed to install this version 2.14.02 of mu_nasm Aha! So "mono" is another requirement. After installing the "mono-complete" package from EPEL8 (together with its dependencies), the "stuart_update" command completes fine for me. Can we spell out the "mono" dependency in ".pytool/Readme.md" somewhere, or can we perhaps check for mono automatically? * Third, attempting stuart_ci_build -c .pytool/CISettings.py -p OvmfPkg -a IA32,X64 -t DEBUG TOOL_CHAIN_TAG=GCC5 doesn't seem to do anything: > SECTION - Init SDE > SECTION - Loading Plugins > SECTION - Start Invocable Tool > SECTION - Getting Environment > SECTION - Loading plugins > SECTION - Building OvmfPkg Package > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG -- > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG > PROGRESS - Overall Build Status: Success > SECTION - Summary > PROGRESS - Success If I remove the "-t DEBUG" option, a bit more happens (but it also fails): > SECTION - Init SDE > SECTION - Loading Plugins > SECTION - Start Invocable Tool > SECTION - Getting Environment > SECTION - Loading plugins > SECTION - Building OvmfPkg Package > PROGRESS - --Running OvmfPkg: Char Encoding Check Test NO-TARGET -- > PROGRESS - --->Test Success: Char Encoding Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG -- > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG > PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE -- > WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE > PROGRESS - --Running OvmfPkg: Dependency Check Test NO-TARGET -- > PROGRESS - --->Test Success: Dependency Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: Dsc Complete Check Test NO-TARGET -- > PROGRESS - --->Test Success: Dsc Complete Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: Guid Check Test NO-TARGET -- > PROGRESS - --->Test Success: Guid Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: Library Class Check Test NO-TARGET -- > PROGRESS - --->Test Success: Library Class Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET -- > WARNING - NodeJs not installed. Test can't run > WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: EccCheck Test NO-TARGET -- > PROGRESS - --->Test Success: EccCheck Test NO-TARGET > PROGRESS - --Running OvmfPkg: License Check Test NO-TARGET -- > PROGRESS - --->Test Success: License Check Test NO-TARGET > PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT -- > WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT > PROGRESS - --Running OvmfPkg: Host Unit Test Dsc Complete Check Test NO-TARGET -- > PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET > PROGRESS - Overall Build Status: Success > SECTION - Summary > PROGRESS - Success I'm not sure about the "NodeJs not installed. Test can't run" message -- I guess it only relates to the "Spell Check Test". But what's missing for the actual build? I've noticed the following warning: > Test Skipped: in plugin! Compiler Plugin DEBUG But even if I add "--verbose", the only lines around that warning are: > INFO - Running on Package: OvmfPkg > PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG -- > WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG > PROGRESS - Overall Build Status: Success Why is the build skipped? --*-- Summary: - the typo in "pip install --upgrade -r pip-requirements.txt" should be fixed - "mono" should be documented, or auto-installed, as a dependency - I don't know why "stuart_ci_build" skips the actual build. Thanks! Laszlo