OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Rebecca Cran]

I noticed OvmfPkg/PlatformCI/iasl_ext_dep.yaml specifies iasl version 
20190215.0.0 while BaseTools/Bin/iasl_ext_dep.yaml has the newer 
20200717.0.0, and "mono .../edk2toolext/bin/NuGet.exe list -Source 
https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json" 
shows there's version 20210105.0.6 available.


Though OvmfPkg is using source https://api.nuget.org/v3/index.json while 
BaseTools uses 
https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json 
- I don't know why they're different.


I was wondering if iasl_ext_dep.yaml should be updated?


Also, the example in 
https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart 
of using stuart_build to build OVMF seems to be missing a step: running 
"stuart_update -c PlatformBuild.py TOOL_CHAIN_TAG=GCC5 -a X64" appears 
to be required otherwise stuart_build will complain that the iasl 
dependency hasn't been met.


-- 
Rebecca Cran

Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Sean]

[-- Attachment #1: Type: text/plain, Size: 3109 bytes --]

Rebecca,

I agree the iasl dependency for CI has not been managed consistently.   
When all of the CI was setup we decided that iasl should be controlled 
by the platform and thus EmulatorPkg, ArmVirt, and OVMF have their own 
extdep.  This gives those platforms control to rev their version as 
necessary for their platform.   We have found it very common in platform 
development for platforms to have different required versions of iasl.exe.

For Core CI (meaning just package builds) we managed iasl in Basetools.  
We decided that this would only need to be updated when newer features 
were used by core components (not very often given the very little ASL 
in edk2 tree).


As for the feed.  Yes they are inconsistent.   We were moving away from 
a global nuget.org feed as it just didn't seem necessary to push to 
nuget.org.  But now we are evaluating ways to move entirely away from 
nuget.  Nuget.exe worked pretty well for Windows development and our 
initial use cases but has definitely created a headache on Linux, MacOS 
and other.  There really isn't a generic package management solution 
that is supported cross platform that has free/high quality/secure 
hosting.  If anyone has ideas please share.


So my suggestion is to hold off for a couple of weeks (unless something 
is broken) and lets see if we can use web downloads from github 
releases.  This should still allow consistency with tools, work cross 
platform, and give the flexibility needed per platform.


Regarding the steps in that document.  In that example it doesn't call 
out all the steps needed as that would just rehash the section before.  
Instead it relies on a user having followed the generic steps in the 
section above (How to Build With Stuart · tianocore/tianocore.github.io 
Wiki 
<https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart#how-to-build-in-edk2-with-stuart>). 
For example the user would need to have also done:  setup python virtual 
env, install pypi dependencies, and clone source + submodules.


Thanks

Sean




On 5/3/2023 12:34 PM, Rebecca Cran wrote:
> I noticed OvmfPkg/PlatformCI/iasl_ext_dep.yaml specifies iasl version 
> 20190215.0.0 while BaseTools/Bin/iasl_ext_dep.yaml has the newer 
> 20200717.0.0, and "mono .../edk2toolext/bin/NuGet.exe list -Source 
> https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json" 
> shows there's version 20210105.0.6 available.
>
>
> Though OvmfPkg is using source https://api.nuget.org/v3/index.json 
> while BaseTools uses 
> https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json 
> - I don't know why they're different.
>
>
> I was wondering if iasl_ext_dep.yaml should be updated?
>
>
> Also, the example in 
> https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart 
> of using stuart_build to build OVMF seems to be missing a step: 
> running "stuart_update -c PlatformBuild.py TOOL_CHAIN_TAG=GCC5 -a X64" 
> appears to be required otherwise stuart_build will complain that the 
> iasl dependency hasn't been met.
>
>

[-- Attachment #2: Type: text/html, Size: 4549 bytes --]

Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Rebecca Cran]

On 5/3/23 13:50, Sean wrote:

> As for the feed.  Yes they are inconsistent.   We were moving away 
> from a global nuget.org feed as it just didn't seem necessary to push 
> to nuget.org.  But now we are evaluating ways to move entirely away 
> from nuget.  Nuget.exe worked pretty well for Windows development and 
> our initial use cases but has definitely created a headache on Linux, 
> MacOS and other.  There really isn't a generic package management 
> solution that is supported cross platform that has free/high 
> quality/secure hosting.  If anyone has ideas please share.
>
>
> So my suggestion is to hold off for a couple of weeks (unless 
> something is broken) and lets see if we can use web downloads from 
> github releases.  This should still allow consistency with tools, work 
> cross platform, and give the flexibility needed per platform.
>
That's great news! That would remove a major impediment to using the 
stuart tools for me on FreeBSD, and a hassle getting setup on Linux.

> Regarding the steps in that document.  In that example it doesn't call 
> out all the steps needed as that would just rehash the section 
> before.  Instead it relies on a user having followed the generic steps 
> in the section above (How to Build With Stuart · 
> tianocore/tianocore.github.io Wiki 
> <https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart#how-to-build-in-edk2-with-stuart>).  
> For example the user would need to have also done:  setup python 
> virtual env, install pypi dependencies, and clone source + submodules.
>
I hadn't realized that .pytool/CISettings.py was for CI and I should 
swap out the platform's UefiBuilder script when running 
stuart_setup/stuart_update.


-- 
Rebecca Cran

Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Gerd Hoffmann]

  Hi,

> I agree the iasl dependency for CI has not been managed consistently.   When
> all of the CI was setup we decided that iasl should be controlled by the
> platform and thus EmulatorPkg, ArmVirt, and OVMF have their own extdep. 
> This gives those platforms control to rev their version as necessary for
> their platform.   We have found it very common in platform development for
> platforms to have different required versions of iasl.exe.

Are there cases where /old/ iasl versions are required?

> As for the feed.  Yes they are inconsistent.   We were moving away from a
> global nuget.org feed as it just didn't seem necessary to push to
> nuget.org.  But now we are evaluating ways to move entirely away from
> nuget.  Nuget.exe worked pretty well for Windows development and our initial
> use cases but has definitely created a headache on Linux, MacOS and other. 
> There really isn't a generic package management solution that is supported
> cross platform that has free/high quality/secure hosting.  If anyone has
> ideas please share.

On linux / macos / *bsd there is usually no need to create your own
package management.  Standard stuff like iasl / nasm is available as
linux distro package / bsd ports collection package.

Usually you can't pick specific versions.  Usually this isn't a big
issue though, unless you are using an older distro (such as ubuntu 18.04
we used for CI before switching to containers) and need a recent version.

take care,
  Gerd

Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 3765 bytes --]

Sean,

Can a GitHub release area for each dependency be used as an alternative to nuget?

Mike

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean
Sent: Wednesday, May 3, 2023 12:51 PM
To: devel@edk2.groups.io; rebecca@bsdio.com; Ard Biesheuvel <ardb+tianocore@kernel.org>; Yao, Jiewen <jiewen.yao@intel.com>; Justen, Jordan L <jordan.l.justen@intel.com>; Gerd Hoffmann <kraxel@redhat.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>; Michael Kubacki <mikuback@linux.microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>
Subject: Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?


Rebecca,

I agree the iasl dependency for CI has not been managed consistently.   When all of the CI was setup we decided that iasl should be controlled by the platform and thus EmulatorPkg, ArmVirt, and OVMF have their own extdep.  This gives those platforms control to rev their version as necessary for their platform.   We have found it very common in platform development for platforms to have different required versions of iasl.exe.

For Core CI (meaning just package builds) we managed iasl in Basetools.  We decided that this would only need to be updated when newer features were used by core components (not very often given the very little ASL in edk2 tree).



As for the feed.  Yes they are inconsistent.   We were moving away from a global nuget.org feed as it just didn't seem necessary to push to nuget.org.  But now we are evaluating ways to move entirely away from nuget.  Nuget.exe worked pretty well for Windows development and our initial use cases but has definitely created a headache on Linux, MacOS and other.  There really isn't a generic package management solution that is supported cross platform that has free/high quality/secure hosting.  If anyone has ideas please share.



So my suggestion is to hold off for a couple of weeks (unless something is broken) and lets see if we can use web downloads from github releases.  This should still allow consistency with tools, work cross platform, and give the flexibility needed per platform.



Regarding the steps in that document.  In that example it doesn't call out all the steps needed as that would just rehash the section before.  Instead it relies on a user having followed the generic steps in the section above (How to Build With Stuart · tianocore/tianocore.github.io Wiki<https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart#how-to-build-in-edk2-with-stuart>).  For example the user would need to have also done:  setup python virtual env, install pypi dependencies, and clone source + submodules.



Thanks

Sean






On 5/3/2023 12:34 PM, Rebecca Cran wrote:
I noticed OvmfPkg/PlatformCI/iasl_ext_dep.yaml specifies iasl version 20190215.0.0 while BaseTools/Bin/iasl_ext_dep.yaml has the newer 20200717.0.0, and "mono .../edk2toolext/bin/NuGet.exe list -Source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json" shows there's version 20210105.0.6 available.


Though OvmfPkg is using source https://api.nuget.org/v3/index.json while BaseTools uses https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json - I don't know why they're different.


I was wondering if iasl_ext_dep.yaml should be updated?


Also, the example in https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart of using stuart_build to build OVMF seems to be missing a step: running "stuart_update -c PlatformBuild.py TOOL_CHAIN_TAG=GCC5 -a X64" appears to be required otherwise stuart_build will complain that the iasl dependency hasn't been met.



[-- Attachment #2: Type: text/html, Size: 7058 bytes --]

Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Michael Kubacki]

Hi Mike,

If I understand correctly, yes. For example, that's how we're pulling 
the CodeQL CLI from the codeql-cli-binaries repo here:

https://github.com/microsoft/mu_basecore/blob/release/202208/.pytool/Plugin/CodeQL/codeqlcli_ext_dep.yaml

If a repo (like a fork) needs to build and publish a tagged release, the 
files can be attached to the release from either a pipeline (using 
something like the GitHub REST API) or a GitHub workflow (using the 
GitHub CLI pre-installed on agents) with a command like gh release 
upload - https://cli.github.com/manual/gh_release_upload.

For example, we have pipelines that apply semantic versioning and upload 
release artifacts as zip files to the GitHub release:

https://github.com/microsoft/mu_tiano_platforms/releases

- Michael

On 5/4/2023 1:21 PM, Michael D Kinney wrote:
> Sean,
> 
> Can a GitHub release area for each dependency be used as an alternative 
> to nuget?
> 
> Mike
> 
> *From:* devel@edk2.groups.io <devel@edk2.groups.io> *On Behalf Of *Sean
> *Sent:* Wednesday, May 3, 2023 12:51 PM
> *To:* devel@edk2.groups.io; rebecca@bsdio.com; Ard Biesheuvel 
> <ardb+tianocore@kernel.org>; Yao, Jiewen <jiewen.yao@intel.com>; Justen, 
> Jordan L <jordan.l.justen@intel.com>; Gerd Hoffmann <kraxel@redhat.com>
> *Cc:* Sean Brogan <sean.brogan@microsoft.com>; Michael Kubacki 
> <mikuback@linux.microsoft.com>; Kinney, Michael D 
> <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>
> *Subject:* Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency 
> be updated from 20190215.0.0 ?
> 
> Rebecca,
> 
> I agree the iasl dependency for CI has not been managed consistently.   
> When all of the CI was setup we decided that iasl should be controlled 
> by the platform and thus EmulatorPkg, ArmVirt, and OVMF have their own 
> extdep.  This gives those platforms control to rev their version as 
> necessary for their platform.   We have found it very common in platform 
> development for platforms to have different required versions of iasl.exe.
> 
> For Core CI (meaning just package builds) we managed iasl in Basetools.  
> We decided that this would only need to be updated when newer features 
> were used by core components (not very often given the very little ASL 
> in edk2 tree).
> 
> As for the feed.  Yes they are inconsistent.   We were moving away from 
> a global nuget.org feed as it just didn't seem necessary to push to 
> nuget.org.  But now we are evaluating ways to move entirely away from 
> nuget.  Nuget.exe worked pretty well for Windows development and our 
> initial use cases but has definitely created a headache on Linux, MacOS 
> and other.  There really isn't a generic package management solution 
> that is supported cross platform that has free/high quality/secure 
> hosting.  If anyone has ideas please share.
> 
> So my suggestion is to hold off for a couple of weeks (unless something 
> is broken) and lets see if we can use web downloads from github 
> releases.  This should still allow consistency with tools, work cross 
> platform, and give the flexibility needed per platform.
> 
> Regarding the steps in that document.  In that example it doesn't call 
> out all the steps needed as that would just rehash the section before.  
> Instead it relies on a user having followed the generic steps in the 
> section above (How to Build With Stuart · tianocore/tianocore.github.io 
> Wiki 
> <https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart#how-to-build-in-edk2-with-stuart>).  For example the user would need to have also done:  setup python virtual env, install pypi dependencies, and clone source + submodules.
> 
> Thanks
> 
> Sean
> 
> On 5/3/2023 12:34 PM, Rebecca Cran wrote:
> 
>     I noticed OvmfPkg/PlatformCI/iasl_ext_dep.yaml specifies iasl
>     version 20190215.0.0 while BaseTools/Bin/iasl_ext_dep.yaml has the
>     newer 20200717.0.0, and "mono .../edk2toolext/bin/NuGet.exe list
>     -Source
>     https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json <https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json>" shows there's version 20210105.0.6 available.
> 
> 
>     Though OvmfPkg is using source https://api.nuget.org/v3/index.json
>     <https://api.nuget.org/v3/index.json> while BaseTools uses
>     https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json <https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json> - I don't know why they're different.
> 
> 
>     I was wondering if iasl_ext_dep.yaml should be updated?
> 
> 
>     Also, the example in
>     https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart <https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart> of using stuart_build to build OVMF seems to be missing a step: running "stuart_update -c PlatformBuild.py TOOL_CHAIN_TAG=GCC5 -a X64" appears to be required otherwise stuart_build will complain that the iasl dependency hasn't been met.
> 
> 

Re: [edk2-devel] OvmfPkg PlatformCI: Should iasl dependency be updated from 20190215.0.0 ?

[Sean]

[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]


On 5/3/2023 10:56 PM, Gerd Hoffmann wrote:
>    Hi,
>
>> I agree the iasl dependency for CI has not been managed consistently.   When
>> all of the CI was setup we decided that iasl should be controlled by the
>> platform and thus EmulatorPkg, ArmVirt, and OVMF have their own extdep.
>> This gives those platforms control to rev their version as necessary for
>> their platform.   We have found it very common in platform development for
>> platforms to have different required versions of iasl.exe.
> Are there cases where /old/ iasl versions are required?

Yes.  On the commercial PC side we see this a lot.  Given how much ASL 
is in modern PCs it is very common for a platform's ASL code to not 
compile with the latest version.  That product and by extension, the ASL 
code, is often supported for 5+ years and then if you add on that the 
ASL compiler was probably already a year old at initial release that 
means the compiler can often be pretty old.  Additionally for those 
building brand new products that are using new capabilities in ACPI, 
they often need the newest version and can't wait for the distro package 
repository to pick up.  As a outsider/consumer of Linux, we have 
definitely had pain caused by the distro's default/available version.  
For the Edk2 repo I can remember a lot of challenges around python 2 and 
3 and versions and the various distro's package management.

Additionally, for CI we want to drive consistency between Windows and 
Linux.  We want visibility into versions used and reproducibility.   
Depending on distro package management make this significantly harder.

EmulatorPkg, ArmVirtPkg, and OVMF could all opt into using the "scope" 
(edk2/Readme.md at master · tianocore/edk2 · GitHub 
<https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#pytool-scopes>) 
that would leverage the Core CI's version of iasl so that they don't 
need to manage this but my initial hope was to showcase this capability 
and allow those "platforms" some of their own autonomy.  Obviously they 
haven't needed or used that yet so as we update Core CI we could also 
update those platforms to follow with the same extdep and version.

>
>> As for the feed.  Yes they are inconsistent.   We were moving away from a
>> global nuget.org feed as it just didn't seem necessary to push to
>> nuget.org.  But now we are evaluating ways to move entirely away from
>> nuget.  Nuget.exe worked pretty well for Windows development and our initial
>> use cases but has definitely created a headache on Linux, MacOS and other.
>> There really isn't a generic package management solution that is supported
>> cross platform that has free/high quality/secure hosting.  If anyone has
>> ideas please share.
> On linux / macos / *bsd there is usually no need to create your own
> package management.  Standard stuff like iasl / nasm is available as
> linux distro package / bsd ports collection package.
>
> Usually you can't pick specific versions.  Usually this isn't a big
> issue though, unless you are using an older distro (such as ubuntu 18.04
> we used for CI before switching to containers) and need a recent version.
>
> take care,
>    Gerd
>

[-- Attachment #2: Type: text/html, Size: 4270 bytes --]