From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) by mx.groups.io with SMTP id smtpd.web12.193349.1598272528278968808 for ; Mon, 24 Aug 2020 05:35:28 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=egFl4/pO; spf=pass (domain: redhat.com, ip: 205.139.110.61, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1598272527; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W6dzN4MT85FU8Bv/MzWybeikMVLvCpxMG5TPOKMWwO0=; b=egFl4/pOEAmibhOl3UVgcxz8GGLfc+is8Oj/grIpZV4wyRHWYF3dkgpq1JrrH7DshUPAWL coFPL2hFg3FVYc2w6rbT9RX92yiPIQM+ohF09z3stSmXDULN1yTgMphXgZ9Z7UyqAfky2m NqUqAh7T03W2HqV2iy6d7elWYCc6cdo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-20-QRSUObicPIGQxGdexKUAMA-1; Mon, 24 Aug 2020 08:35:16 -0400 X-MC-Unique: QRSUObicPIGQxGdexKUAMA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AA75F100746A; Mon, 24 Aug 2020 12:35:14 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-112-186.ams2.redhat.com [10.36.112.186]) by smtp.corp.redhat.com (Postfix) with ESMTP id 79946197E5; Mon, 24 Aug 2020 12:35:12 +0000 (UTC) Subject: Re: [edk2-devel] running CI locally To: Sean Brogan , devel@edk2.groups.io, michael.d.kinney@intel.com, Sean Brogan , Bret Barkelew Cc: Vladimir Olovyannikov , Rebecca Cran , Tom Lendacky References: <2d9572c6-7549-d0ae-3ec9-95158cfef484@redhat.com> <5ca9187c-96f3-53f1-0ac0-809bc1156ece@redhat.com> From: "Laszlo Ersek" Message-ID: <04b0f327-1e31-a4b4-943d-df5bd4caff28@redhat.com> Date: Mon, 24 Aug 2020 14:35:11 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=lersek@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit On 08/21/20 23:36, Sean Brogan wrote: > On 8/21/2020 12:23 AM, Laszlo Ersek wrote: >> - ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg, >> NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg >> are not covered by host-based unit tests -- I think also >> expected: >> >>> [...] >> >> - I'm not really sure about the following warnings. They were >> emitted for a subset of the above packages. I read the docs at >> >> >> >> but I still don't understand :) > > Each plugin/test has a readme as well. Not sure if this helps explain > the HostUnitTestDscCompleteCheck more. > > https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck I've carefully re-read the explanation now, and (apparently) I do understand it now :/ Before, I didn't realize that the expression "those related to host based unit tests" stood for the set of INF files where each INF was MODULE_TYPE=HOST_APPLICATION or LIBRARY_CLASS=FoobarLib|HOST_APPLICATION. > As to why you don't see it skipped for ArmVirtPkg or a few others: I > think this is a bug. > > In Azure CI i see it as test passed. > > PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check > Test NO-TARGET -- > PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test > NO-TARGET > > When i look at the code > https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59 > > > I see that how those packages *.ci.yaml have it configured is > different than those that show skipped below. It avoids the skip > conditions but since those packages don't have any host unit test > libraries or modules there is no failure. "OvmfPkg/OvmfPkg.ci.yaml" (as an example) sets both "HostUnitTestCompilerPlugin.DscPath" and "HostUnitTestDscCompleteCheck.DscPath" to the empty string. "ShellPkg/ShellPkg.ci.yaml" (an an example) sets neither attribute at all. Both HostUnitTestCompilerPlugin and HostUnitTestDscCompleteCheck check for the existence of the DscPath attribute, as first step, so both plugins warn about, and skip, ShellPkg. OvmfPkg passes the same, first -- i.e., DscPath attribute existence -- check in both plugins. This suggests OvmfPkg is ruled out by *further* checks in one plugin, and not ruled out by any further checks in the other plugin. The next (2nd) check in both plugins seems to be, whether we can actually find the DSC file pointed-to by the DscPath attribute. Furthermore, in HostUnitTestDscCompleteCheck, this check is the last one (no other checks). I think that this 2nd check *too* should fail for OvmfPkg in *both* plugins. Because the DscPath attribute, while it exists, is empty. I don't see why the "DSC path lookup" would fail in HostUnitTestCompilerPlugin (and so we'd get a warning about OvmfPkg), but not in HostUnitTestDscCompleteCheck (and so we'd get no warning about OvmfPkg). ... Is it possible that the 2nd check actually fires in HostUnitTestDscCompleteCheck too, but we suppress the warning by returning 0, and not -1? Compare the return values after the tc.SetSkipped() calls: [https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestCompilerPlugin/HostUnitTestCompilerPlugin.py#L103] 100 if AP is None or AP_Path is None or not os.path.isfile(APDSC): 101 tc.SetSkipped() 102 tc.LogStdError("Package HostBasedUnitTest Dsc not found.") 103 return -1 ^^ versus [https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L73] 70 if abs_dsc_path is None or wsr_dsc_path == "" or not os.path.isfile(abs_dsc_path): 71 tc.SetSkipped() 72 tc.LogStdError("Package Host Unit Test Dsc not found") 73 return 0 ^ I guess I could verify if the branch was taken at all, from the standard error... Ah, "Build/TestSuites.xml" seems relevant. ... Yes, I'm fairly sure the zero value after the "return" statement, on line 73, is a typo. Because this is what I have in "Build/TestSuites.xml" (excerpt re-wrapped here for readability): > > id="14" > name="OvmfPkg" > package="Edk2CiBuild.Edk2.OvmfPkg" > errors="0" > tests="12" > failures="0" > skipped="6"> > > classname="Edk2CiBuild.Edk2.OvmfPkg.HostUnitTestCompiler.X64" > name="Compile and Run Host-Based UnitTests for Edk2CiBuild.Edk2.OvmfPkg on arch X64" > time="0.0002715587615966797"> > > > Package HostBasedUnitTest Dsc not found. > > > classname="Edk2CiBuild.Edk2.OvmfPkg.HostUnitTestDscCompleteCheck" > name="Check the Edk2CiBuild.Edk2.OvmfPkg Host Unit Test DSC for a being complete" > time="0.00020742416381835938"> > > > Package Host Unit Test Dsc not found > > > > Note that both quoted test cases (both plugins) report having been skipped. And from the element, we can confirm that both plugins catch OvmfPkg's DscPath attribute being the empty string. It's just that "HostUnitTestDscCompleteCheck" returns zero on that branch. > This should probably be a bugzilla on the > HostUnitTestDscCompleteCheck.py as the reporting is not correct. Test > should show up as skipped. I've now filed: https://bugzilla.tianocore.org/show_bug.cgi?id=2924 For now I've assigned it to you, and set BaseTools as the Package needing the fix. Please update the metadata as necessary :) >> - The below seems to be related to the "spell checking in audit >> mode" known issues at >> . >> >> What does "audit mode" mean? > > see readme here. > https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration > > > Basically runs the tests and reports all the errors as warnings and > then reports the test as skipped. It was hopefully to encourage > package maintainers to fix their spelling issues but not breaking the > build on day 1. Oh. In OvmfPkg, the CI YAML file says "Fails right now with over 270 errors" :/ >> * One suggestion (in addition to the others made in this thread): >> >> - Please add "BaseToolsBuild" to "BaseTools/.gitignore". > > There is a bugzilla for this already. > https://bugzilla.tianocore.org/show_bug.cgi?id=2581 Thanks! >> * A question: >> >> - On github, there is some logic that restricts the full Core CI >> build to packages that may be affected by a patch series. Is this >> available locally? >> >> According to the Core CI documentation, the "stuart_ci_build" >> command can be limited with "-p" / "-a" / "-t" flags; given a >> commit range, how do we calculate the "tightest" flag values? > > A couple things here. > You can always look at the azurepipelines files to get a better > understanding of the CI process. Hopefully there is no magic (besides > being able to read yaml). You can see here: > https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35 > > > It is actually another stuart tool. > stuart_pr_eval. > We have bug to document it. > https://github.com/tianocore/edk2-pytool-extensions/issues/89 > But the python file is pretty well documented. There are 4 policies > that can trigger the need for a change to build a package. > See here > https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117 Cool, so I can run something like stuart_pr_eval -c .pytool/CISettings.py -p CANDIDATE-PACKAGE-LIST --pr-target TOPIC-BRANCH \ [output formatting options] I'll try that next time! Thank you! Laszlo