From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from rn-mailsvcp-ppex-lapp24.apple.com (rn-mailsvcp-ppex-lapp24.apple.com [17.179.253.38]) by mx.groups.io with SMTP id smtpd.web12.360.1630007466122679913 for ; Thu, 26 Aug 2021 12:51:06 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=uAtOAv3q; spf=pass (domain: apple.com, ip: 17.179.253.38, mailfrom: afish@apple.com) Received: from pps.filterd (rn-mailsvcp-ppex-lapp24.rno.apple.com [127.0.0.1]) by rn-mailsvcp-ppex-lapp24.rno.apple.com (8.16.1.2/8.16.1.2) with SMTP id 17QJlmI3028475; Thu, 26 Aug 2021 12:51:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=/BIXrjzieKLwK9rG4mp7ZEaVJuqcmUgft18/7SzyS2Y=; b=uAtOAv3q86KRPeCKTvpJeFVGd3/qbDnR1hE9rJIo+IwAQAwKLDvtzLF9b3+quiUlJEOw WOWpheYKMZ72Yhl/If+SCJznlMGXPLM52WPYAl3hYPqopiXXm/hqFVmCxNoBWOS83nxI sa+I6vcLKF5/6EVVw1mOIyf5TwPDKl12BwyuP0tIUJiHsPe/8zrPf/fItZHkkV5zNerC 8l7Z/9ULJ+ebATLpTGnpMfvFArjLmenPLH+RPE02sg+O+cBb5S+qa18y8dWDQfSqYdoL kDx3K1DhJKvnAPPQZRI2+iHoHqPCh/Qk/Pl7uqqlc+Ley75Wif1P64uObNw7w7FzMTWR YQ== Received: from rn-mailsvcp-mta-lapp04.rno.apple.com (rn-mailsvcp-mta-lapp04.rno.apple.com [10.225.203.152]) by rn-mailsvcp-ppex-lapp24.rno.apple.com with ESMTP id 3ajy39yv6t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 26 Aug 2021 12:51:03 -0700 Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QYG005I4PT37780@rn-mailsvcp-mta-lapp04.rno.apple.com>; Thu, 26 Aug 2021 12:51:03 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QYG00800P50IG00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 26 Aug 2021 12:51:03 -0700 (PDT) X-Va-A: X-Va-T-CD: da3a4df698400084da27c6ab403bcb35 X-Va-E-CD: dc42d2e6004558bfddd6d70f081a7746 X-Va-R-CD: cc7dcf987882c206d52f8e141a493435 X-Va-CD: 0 X-Va-ID: 86aa07cc-b901-4bf4-84f8-838a87d21848 X-V-A: X-V-T-CD: da3a4df698400084da27c6ab403bcb35 X-V-E-CD: dc42d2e6004558bfddd6d70f081a7746 X-V-R-CD: cc7dcf987882c206d52f8e141a493435 X-V-CD: 0 X-V-ID: 5a83b2c9-e2ce-4d4c-8682-12162a5e18d9 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-08-26_05:2021-08-26,2021-08-26 signatures=0 Received: from [17.235.42.52] (unknown [17.235.42.52]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QYG00WAFPSZFF00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 26 Aug 2021 12:51:01 -0700 (PDT) From: "Andrew Fish" Message-id: MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.1\)) Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables Date: Thu, 26 Aug 2021 12:50:59 -0700 In-reply-to: Cc: =?utf-8?Q?Marvin_H=C3=A4user?= , "tim.lewis@insyde.com" , "devel@edk2.groups.io" , "leif@nuviainc.com" , "Ni, Ray" , "Gao, Zhichao" , "Wang, Jian J" , "Wu, Hao A" , "Bi, Dandan" , "Dong, Eric" , Bret Barkelew , Vitaly Cheptsov To: Mike Kinney References: <1e7201d79a93$b5386140$1fa923c0$@insyde.com> <8ea3fa57-3091-b2c5-dba2-c3eca153b697@posteo.de> X-Mailer: Apple Mail (2.3654.20.0.2.1) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-08-26_05:2021-08-26,2021-08-26 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_8007B5E6-1026-4680-81BA-0298483AF551" --Apple-Mail=_8007B5E6-1026-4680-81BA-0298483AF551 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Mike, That reminds me that my patch to fix this for Xcode is still in limbo.=20 Thanks, Andrew Fish > On Aug 26, 2021, at 9:44 AM, Kinney, Michael D wrote: >=20 > Marvin, >=20 > The main reason most components in the EDK II repos continue to use the v= ariables is=20 > because there are incomplete tools to generate PE/COFF resource sections = for all > the OS/compiler combinations that EDK II supports. >=20 > The preference would be to use PE/COFF resource sections and we would hav= e converted > all components to that style long ago if the tools existed to be aligned = with the > UEFI Specification instead of an EDK II specific implementation feature. >=20 > Also, it is not a good idea to only look at the open source EDK II repos = to > understand how a feature is used. There are many downstream consumers of= the > EDK II repos. >=20 > The UEFI Driver Writer's Guide *only* documents the PE/COFF resource sect= ion > method. >=20 > https://tianocore-docs.github.io/edk2-UefiDriverWritersGuide/draft/7_driv= er_entry_point/74_adding_hii_packages_feature.html#74-adding-hii-packages-f= eature >=20 > Best regards, >=20 > Mike >=20 >> -----Original Message----- >> From: Marvin H=C3=A4user = > >> Sent: Thursday, August 26, 2021 9:07 AM >> To: tim.lewis@insyde.com ; devel@edk2.group= s.io ; Kinney, Michael D > >> Cc: 'Andrew Fish' >; leif@nuvia= inc.com ; Ni, Ray >; Gao, Zhichao >; >> Wang, Jian J >; Wu,= Hao A >; Bi, Dandan >; Dong, Eric >> >; 'Bret Barkelew' >; 'Vitaly Che= ptsov' > >> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables >>=20 >> Hey Tim, >>=20 >> Interesting, do you happen to have some tool or code that performs such >> edits at hand to take a look at? Seems like most modules already use >> variables and thus cannot be modified in such a way even right now? >>=20 >> That's the kind of information I am looking for, thanks a lot! >>=20 >> Best regards, >> Marvin >>=20 >> On 26/08/2021 18:01, tim.lewis@insyde.com wrote: >>> Hi Marvin -- >>>=20 >>> I would like to add some historical perspective on this. One of the des= ign requirements back when HII was first >> introduced into the UEFI specification after Intel's initial contributio= n was that of binary editability. In order to be >> able to reliably find, extract and then re-insert the HII data into the = binary, it needed to be discoverable and not >> affect the offsets of the code and data in the binary. >>>=20 >>> While it was possible to put some sort of signature in the read-only da= ta sections of the binary and leave padding for >> possible future edits, it was felt that using a PE/COFF section similar = to the resource sections was better. Resource >> sections are commonly in use for PE/COFF files (in Windows) and the simi= lar idea existed in the then-extant Mac binary >> format (resource fork?). >>>=20 >>> Thanks, >>>=20 >>> Tim Lewis >>> CTO, Insyde Software >>> www.insyde.com >>>=20 >>> -----Original Message----- >>> From: devel@edk2.groups.io On Behalf Of Michael = D Kinney >>> Sent: Thursday, August 26, 2021 7:37 AM >>> To: devel@edk2.groups.io; mhaeuser@posteo.de; Kinney, Michael D >>> Cc: Andrew Fish ; leif@nuviainc.com; Ni, Ray ; Gao, Zhichao ; >> Wang, Jian J ; Wu, Hao A ; Bi= , Dandan ; Dong, Eric >> ; Bret Barkelew ; Vita= ly Cheptsov >>> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables >>>=20 >>> Marvin, >>>=20 >>> One constraint in this discussion is that the HII content in a PE/COFF = resource section is defined in the UEFI >> Specification, Which means UEFI Apps and UEFI Drivers from media and opt= ion ROMs that are not part of the system FW image >> are allowed to use this feature, This means the system FW PE/COFF loade= r must support loading HII content from this >> PE/COFF resource section to be UEFI conformant. So we cannot remove thi= s feature from the PE/COFF loader without changes >> to the UEFI Specification. Even if changes to the UEFI Specification we= made, we would have to continue to support this >> feature for backward compatibility with existing UEFI Apps/Drivers that = may be using this feature. >>>=20 >>> Thanks, >>>=20 >>> Mike >>>=20 >>>> -----Original Message----- >>>> From: devel@edk2.groups.io On Behalf Of Marvin >>>> H=C3=A4user >>>> Sent: Thursday, August 26, 2021 1:51 AM >>>> To: devel@edk2.groups.io; Kinney, Michael D >>>> >>>> Cc: devel@edk2.groups.io; Andrew Fish ; >>>> leif@nuviainc.com; Ni, Ray ; Gao, Zhichao >>>> ; Wang, Jian J ; Wu, Hao >>>> A ; Bi, Dandan ; Dong, Eric >>>> ; Bret Barkelew ; >>>> Vitaly Cheptsov >>>> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C >>>> variables >>>>=20 >>>> Hey Mike, >>>>=20 >>>> Thanks for your reply! >>>>=20 >>>> Well, this switch is not well-documented. Looking now at both the >>>> generation code and the emitted code, it does not generate a package >>>> list like my code does, but separate data variables (strings and >>>> images) that cannot easily be passed to HiiDatabase as-is. However >>>> apparently there are drivers that use this functionality successfully >>>> by composing the package list at runtime [1]. >>>>=20 >>>> Looking with this information now at the pattern of using HII C >>>> variables (which I did not know existed before) vs the PE/COFF HII >>>> section, most that use latter are Shell applications, which I guess >>>> means the section has actually been introduced to resolve D.? There >>>> are exceptions such as LogoDxe [2], which use the PE/COFF section whil= e D. >>>> is not a problem, hence I got confused, sorry. I think these modules >>>> should be updated in any case. Do you agree? >>>>=20 >>>> So, for modules that use C variables already, my patch would save some >>>> runtime generation code and dynamic memory allocation for the HII >>>> package list. This was not my goal (as I said, I didn't realise HII C >>>> variables already were a thing in the first place), but the changes >>>> are small enough that it might be worth considering anyway, in my opin= ion. >>>> If a HII package list is generated for both Shell and non-Shell apps, >>>> this also means code paths can be unified. For example, there could be >>>> a library class with constructor and destructor to add/remove packages >>>> from the HII database for all modules that use such, Shell or not. For >>>> BaseTools it means that there is no real need for separate Python and >>>> C paths as ideally they just generate the exact same data. >>>>=20 >>>> Now to D., the only usage for this seems to be that Shell can locate >>>> the help text in the executable without executing it, yet it is fully >>>> loaded anyway [3]. To be honest, I find it hard to justify loading an >>>> executable (PE/COFF loading, memory permission application, the full >>>> process) to retrieve a help text and then unloading it again, >>>> especially with the HII code being on a core dispatcher level. 1. to >>>> 7. still hold true in my opinion. Was there any discussion I could >>>> read through why Shell apps cannot simply support a "--help" or "-?" >>>> command and output the string themselves? Pushing the burden to the >>>> Shell apps does preserve the "drawback" that a full loading is >>>> required (which honestly does not matter for a debugging application >>>> like Shell), however it does relieve the burden of PE/COFF HII parsing >>>> from the core dispatcher (which matters a lot in my opinion to keep >>>> the core simple). It would simply be a normal Shell app execution as >>>> any other however. If someone wants to avoid the PE/COFF burden >>>> altogether, they can still provide a .man file. >>>>=20 >>>> As for my points 6. and 7., maybe I should provide some context. Due >>>> to many issues with TE files, platforms started abandoning them and >>>> returned to PE/COFF Images. I think a big reason for this is that TE >>>> is not really a sound and complete format, but a stripped version of >>>> PE/COFF with none of the necessary fixups applied. I'm currently >>>> sketching a possible alternative [4], and I would really like to not >>>> having to specify a HII section type, while still preserving >>>> compatibility with all of the UEFI Image types and use-cases [4]. >>>>=20 >>>> Thanks again! >>>>=20 >>>> Best regards, >>>> Marvin >>>>=20 >>>>=20 >>>> [1] >>>> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0ea >>>> b4ac688d5/MdeModulePkg/Application/BootManagerMenuAp >>>> p/BootManagerMenu.c#L929-L934 >>>>=20 >>>> [2] >>>> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0ea >>>> b4ac688d5/MdeModulePkg/Logo/LogoDxe.inf#L23 >>>>=20 >>>> [3] >>>>=20 >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0eab4= ac688d5/ShellPkg/Application/Shell/ShellManParser. >>>> c#L646-L671 >>>>=20 >>>> [4] >>>> https://github.com/mhaeuser/edk2/blob/ue_poc/MdePkg/Include/IndustrySt >>>> andard/UeImage.h >>>>=20 >>>> 26.08.2021 00:34:12 Michael D Kinney : >>>>=20 >>>>> Hi Marvin, >>>>>=20 >>>>> I think this feature is already there and supported. >>>>>=20 >>>>> HII can either be in a global variable or in a PE/COFF resource secti= on. >>>>> The default is a global variable because HII was implemented before >>>>> the PE/COFF resource section feature was added to the UEFI Specificat= ion. >>>>>=20 >>>>> There is an INF [Defines] section statement to enable the PE/COFF >>>>> section. See UefiHiiResource in the following link. >>>>>=20 >>>>> https://tianocore-docs.github.io/edk2-InfSpecification/draft/3_edk_i >>>>> i_inf_file_format/34_[defines]_section.html#34- >>>> defines-section >>>>> How is your proposal different than this existing capability? >>>>>=20 >>>>> Thanks, >>>>>=20 >>>>> Mike >>>>>=20 >>>>>> -----Original Message----- >>>>>> From: devel@edk2.groups.io On Behalf Of >>>>>> Marvin H=C3=A4user >>>>>> Sent: Wednesday, August 25, 2021 2:21 PM >>>>>> To: devel@edk2.groups.io >>>>>> Cc: Andrew Fish ; leif@nuviainc.com; Kinney, >>>>>> Michael D ; Ni, Ray ; >>>>>> Gao, Zhichao ; Wang, Jian J >>>>>> ; Wu, Hao A ; Bi, Dandan >>>>>> ; Dong, Eric ; Bret >>>>>> Barkelew ; Vitaly Cheptsov >>>>>> >>>>>> Subject: [edk2-devel] [RFC] Expose HII package list via C variables >>>>>>=20 >>>>>> Good day everyone, >>>>>>=20 >>>>>> Currently, the HII package list is stored in a PE/COFF resource >>>>>> section [1]. I propose to store it in a C variable (byte array with >>>>>> a pointer to it and its size exposed) instead. DxeCore would have a >>>>>> guard to toggle the deprecated support for the automatic protocol >>>>>> installation. This has the following advantages: >>>>>>=20 >>>>>> 1. Fixes BZ (incl. future toolchains): >>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=3D557 >>>>>> 2. Universal method across all toolchains and output file formats >>>>>> 3. Saves error-prone parsing work 4. Saves protocol install/locate >>>>>> work, the data is available right away 5. The omission of a >>>>>> dedicated section can save space 6. Terse file formats can support >>>>>> this and remain terse :) 7. Removes a dependency on the PE/COFF >>>>>> format specifically >>>>>>=20 >>>>>> A *very rough* PoC diff can be found here: >>>>>> https://github.com/mhaeuser/edk2/compare/master...wip_hii_cvar >>>>>> If the feedback is positive, I will clean it up of course. OVMF >>>>>> boots with everything working fine. >>>>>>=20 >>>>>> I'd explicitly like feedback on the following: >>>>>> A. Is this an acceptable solution to BZ 557 (Andrew?)? >>>>>> B. Is this an acceptable solution for the "HII workflow" (MdeModule >>>>>> maintainers?)? >>>>>> C. Is it acceptable to make support UEFI-side support for the old >>>>>> mechanism optional (Stewards?)? >>>>>> D. Can an acceptable alternative be found for the removed ShellPkg >>>>>> code (Shell maintainers?)? >>>>>>=20 >>>>>> As you can see the BaseTools part also is rough, but that is more a >>>>>> question of "how" rather than "whether", so I'll postpone asking abo= ut it. >>>>>>=20 >>>>>> Thanks for your time and feedback! >>>>>>=20 >>>>>> Best regards, >>>>>> Marvin >>>>>>=20 >>>>>>=20 >>>>>> [1] "Once the image is loaded, LoadImage() installs >>>>>> EFI_HII_PACKAGE_LIST_PROTOCOL on the handle if the image contains a >>>>>> custom PE/COFF resource with the type 'HII'." >>>>>> - UEFI 2.9, 7.4, "EFI_BOOT_SERVICES.LoadImage()" >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>=20 >>>=20 >>>=20 --Apple-Mail=_8007B5E6-1026-4680-81BA-0298483AF551 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Mike,

That reminds me that my patch to fix this f= or Xcode is still in limbo. 

Thanks,

Andrew Fish

On Aug 26, 2021, at 9:44 AM, Kinney, Michae= l D <michael.d.= kinney@intel.com> wrote:

Marvin,<= br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 1= 2px; font-style: normal; font-variant-caps: normal; font-weight: normal; le= tter-spacing: normal; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0p= x; text-decoration: none;" class=3D"">
The main reason most components in the EDK II repos continue to us= e the variables is 
because there are incomplete= tools to generate PE/COFF resource sections for all
the OS/compiler combinations that EDK II supp= orts.

The preference would be to use PE/COFF resource secti= ons and we would have converted
all components to that style long ago if the tools existed to be a= ligned with the
UEFI Sp= ecification instead of an EDK II specific implementation feature.
= Also, it is not a good idea to only look at the open source EDK II r= epos to
understand how = a feature is used.  There are many downstream consumers of the<= br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 1= 2px; font-style: normal; font-variant-caps: normal; font-weight: normal; le= tter-spacing: normal; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0p= x; text-decoration: none;" class=3D"">EDK II repos.

The UEFI Driver Writer's Guide *only* documents the PE/COFF resource secti= on
method.

https://tianoco= re-docs.github.io/edk2-UefiDriverWritersGuide/draft/7_driver_entry_point/74= _adding_hii_packages_feature.html#74-adding-hii-packages-feature

Best regards,

Mike

-----Ori= ginal Message-----
From: Marvin H=C3=A4user <mhaeuser@posteo.de>
Sent: Thursday, August 26, 2021 9:07 AM
To: tim.lewis@insyde.com;&= nbsp;devel@edk2.g= roups.io; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: 'And= rew Fish' <afish@apple.com= >; leif@nuviainc.com; Ni, Ray <ray.ni@intel.com>; Gao, Zhic= hao <zhichao.gao@int= el.com>;
Wang, Jian J <jian.j.wang@intel.com>; Wu, Hao A <hao.a.wu@intel.com>; Bi, = Dandan <dandan.bi@inte= l.com>; Dong, Eric
<eric.dong@intel.com>; 'Bret Barkelew' <Bret.Barkelew@microsoft.= com>; 'Vitaly Cheptsov' <vit9696@protonmail.com>
Subject: Re: [edk= 2-devel] [RFC] Expose HII package list via C variables

Hey Tim,

Interesting, do you happen t= o have some tool or code that performs such
edits at hand to = take a look at? Seems like most modules already use
variables= and thus cannot be modified in such a way even right now?
That's the kind of information I am looking for, thanks a lot!=

Best regards,
Marvin

On 26/08/2021 18:01, tim.lewis@insyde.com wrote:
Hi Marvin --

I would= like to add some historical perspective on this. One of the design require= ments back when HII was first
introduced into th= e UEFI specification after Intel's initial contribution was that of binary = editability. In order to be
able to reliably find, extract an= d then re-insert the HII data into the binary, it needed to be discoverable= and not
affect the offsets of the code and data in the binar= y.

While = it was possible to put some sort of signature in the read-only data section= s of the binary and leave padding for
possible f= uture edits, it was felt that using a PE/COFF section similar to the resour= ce sections was better. Resource
sections are commonly in use= for PE/COFF files (in Windows) and the similar idea existed in the then-ex= tant Mac binary
format (resource fork?).

Thanks,

Tim Lewis
CTO, Insyde Software
www.insyde.com

-----Original Message-----
From: devel@edk2.groups= .io <devel@edk2.groups.io> On Behalf Of Michael D Kinney
Sent: Thursday, August 26, 2021 7:37 AM
To: devel@edk2.grou= ps.io; mhaeuser@posteo.de; Kinney, Michael D <michael.d.kinney@intel.com= >
Cc: Andrew Fish <afish@apple.com>; leif@nuviainc.c= om; Ni, Ray <ray.ni@intel.com>; Gao, Zhichao <zhichao.gao@intel.co= m>;
Wang, Jian J <jian.j.wang@intel.com>; Wu, Hao A <hao.a.wu@intel.com>; = Bi, Dandan <dandan.bi@= intel.com>; Dong, Eric
<eric.dong@intel.com>; Bret Barkelew <Bret.Barkelew@microsof= t.com>; Vitaly Cheptsov <vit9696@protonmail.com>
Subject: Re: [edk2-devel] [RFC] Expose HII package lis= t via C variables

Marvin,

One constraint in this discussion is that the HII content in a PE/= COFF resource section is defined in the UEFI
Spe= cification, Which means UEFI Apps and UEFI Drivers from media and option RO= Ms that are not part of the system FW image
are allowed to us= e this feature,  This means the system FW PE/COFF loader must support = loading HII content from this
PE/COFF resource section to be = UEFI conformant.  So we cannot remove this feature from the PE/COFF lo= ader without changes
to the UEFI Specification.  Even if= changes to the UEFI Specification we made, we would have to continue to su= pport this
feature for backward compatibility with existing U= EFI Apps/Drivers that may be using this feature.

Thanks,

Mike

--= ---Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin<= br class=3D"">H=C3=A4user
Sent: Thursday, August 26, 2021 1:5= 1 AM
To: d= evel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com&g= t;
Cc: dev= el@edk2.groups.io; Andrew Fish <afish@apple.com>;
leif@nuviainc.com; Ni, Ray <ray.ni@intel.com>; Gao, Zhichao
<zhichao.gao@= intel.com>; Wang, Jian J <jian.j.wang@intel.com>; Wu, Hao
A <hao.a.wu@intel.com>; B= i, Dandan <dandan.bi@i= ntel.com>; Dong, Eric
<eric.dong@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft= .com>;
Vitaly Cheptsov <vit9696@protonmail.com>
Subj= ect: Re: [edk2-devel] [RFC] Expose HII package list via C
var= iables

Hey Mike,

= Thanks for your reply!

Well, this switch is no= t well-documented. Looking now at both the
generation code an= d the emitted code, it does not generate a package
list like = my code does, but separate data variables (strings and
images= ) that cannot easily be passed to HiiDatabase as-is. However
= apparently there are drivers that use this functionality successfully
by composing the package list at runtime [1].

Looking with this information now at the pattern of using HII Cvariables (which I did not know existed before) vs the PE/COFF = HII
section, most that use latter are Shell applications, whi= ch I guess
means the section has actually been introduced to = resolve D.? There
are exceptions such as LogoDxe [2], which u= se the PE/COFF section while D.
is not a problem, hence I got= confused, sorry. I think these modules
should be updated in = any case. Do you agree?

So, for modules that u= se C variables already, my patch would save some
runtime gene= ration code and dynamic memory allocation for the HII
package= list. This was not my goal (as I said, I didn't realise HII C
variables already were a thing in the first place), but the changes
are small enough that it might be worth considering anyway, in my = opinion.
If a HII package list is generated for both Shell an= d non-Shell apps,
this also means code paths can be unified. = For example, there could be
a library class with constructor = and destructor to add/remove packages
from the HII database f= or all modules that use such, Shell or not. For
BaseTools it = means that there is no real need for separate Python and
C pa= ths as ideally they just generate the exact same data.

Now to D., the only usage for this seems to be that Shell can loca= te
the help text in the executable without executing it, yet = it is fully
loaded anyway [3]. To be honest, I find it hard t= o justify loading an
executable (PE/COFF loading, memory perm= ission application, the full
process) to retrieve a help text= and then unloading it again,
especially with the HII code be= ing on a core dispatcher level. 1. to
7. still hold true in m= y opinion. Was there any discussion I could
read through why = Shell apps cannot simply support a "--help" or "-?"
command a= nd output the string themselves? Pushing the burden to the
Sh= ell apps does preserve the "drawback" that a full loading is
= required (which honestly does not matter for a debugging application
like Shell), however it does relieve the burden of PE/COFF HII pars= ing
from the core dispatcher (which matters a lot in my opini= on to keep
the core simple). It would simply be a normal Shel= l app execution as
any other however. If someone wants to avo= id the PE/COFF burden
altogether, they can still provide a .m= an file.

As for my points 6. and 7., maybe I s= hould provide some context. Due
to many issues with TE files,= platforms started abandoning them and
returned to PE/COFF Im= ages. I think a big reason for this is that TE
is not really = a sound and complete format, but a stripped version of
PE/COF= F with none of the necessary fixups applied. I'm currently
sk= etching a possible alternative [4], and I would really like to not
having to specify a HII section type, while still preserving
compatibility with all of the UEFI Image types and use-cases [4].

Thanks again!

Best r= egards,
Marvin


[1= ]
https://github.com/tianocore/edk2/blob/7= b4a99be8a39c12d3a7fc4b8db9f0ea
b4ac688d5/MdeModulePkg/App= lication/BootManagerMenuAp
p/BootManagerMenu.c#L929-L934

[2]
https://github.com/tianocore/edk= 2/blob/7b4a99be8a39c12d3a7fc4b8db9f0ea
b4ac688d5/MdeModulePkg= /Logo/LogoDxe.inf#L23

[3]

https://github.com/tianocore/edk2/blob/7b4a99= be8a39c12d3a7fc4b8db9f0eab4ac688d5/ShellPkg/Application/Shell/ShellManParse= r.
c#L646-L671

[4]
https://github.com/mhaeuser/edk2/blob/ue_poc/MdePk= g/Include/IndustrySt
andard/UeImage.h

26.08.2021 00:34:12 Michael D Kinney <michael.d.kinney@intel.c= om>:

H= i Marvin,

I think this feature is already ther= e and supported.

HII can either be in a global= variable or in a PE/COFF resource section.
The default is a = global variable because HII was implemented before
the PE/COF= F resource section feature was added to the UEFI Specification.

There is an INF [Defines] section statement to enable the= PE/COFF
section. See UefiHiiResource in the following link.<= br class=3D"">
https://tianocore-docs.github.io/edk2-InfSpeci= fication/draft/3_edk_i
i_inf_file_format/34_[defines]_section= .html#34-
defines-section
How is your proposal different than this exist= ing capability?

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <= devel@edk2.groups.io> On Behalf Of
Marvin H=C3=A4user
Sent: Wednesday, August 25, 2021 2:21 PM
To: devel@= edk2.groups.io
Cc: Andrew Fish <afish@apple.com>; leif@= nuviainc.com; Kinney,
Michael D <michael.d.kinney@intel.co= m>; Ni, Ray <ray.ni@intel.com>;
Gao, Zhichao <zhi= chao.gao@intel.com>; Wang, Jian J
<jian.j.wang@intel.co= m>; Wu, Hao A <hao.a.wu@intel.com>; Bi, Dandan
<d= andan.bi@intel.com>; Dong, Eric <eric.dong@intel.com>; Bret
Barkelew <Bret.Barkelew@microsoft.com>; Vitaly Cheptsov
<vit9696@protonmail.com>
Subject: [edk2-devel]= [RFC] Expose HII package list via C variables

Good day everyone,

Currently, the HII package= list is stored in a PE/COFF resource
section [1]. I propose = to store it in a C variable (byte array with
a pointer to it = and its size exposed) instead. DxeCore would have a
guard to = toggle the deprecated support for the automatic protocol
inst= allation. This has the following advantages:

1= . Fixes BZ (incl. future toolchains):
https://bugzilla.tianoc= ore.org/show_bug.cgi?id=3D557
2. Universal method across all = toolchains and output file formats
3. Saves error-prone parsi= ng work 4. Saves protocol install/locate
work, the data is av= ailable right away 5. The omission of a
dedicated section can= save space 6. Terse file formats can support
this and remain= terse :) 7. Removes a dependency on the PE/COFF
format speci= fically

A *very rough* PoC diff can be found h= ere:
https://github.com/mhaeuser/edk2/compare/master...wip_hi= i_cvar
If the feedback is positive, I will clean it up of cou= rse. OVMF
boots with everything working fine.
<= br class=3D"">I'd explicitly like feedback on the following:
= A. Is this an acceptable solution to BZ 557 (Andrew?)?
B. Is = this an acceptable solution for the "HII workflow" (MdeModule
maintainers?)?
C. Is it acceptable to make support UEFI-side= support for the old
mechanism optional (Stewards?)?
D. Can an acceptable alternative be found for the removed ShellPkgcode (Shell maintainers?)?

As you = can see the BaseTools part also is rough, but that is more a
= question of "how" rather than "whether", so I'll postpone asking about it.<= br class=3D"">
Thanks for your time and feedback!

Best regards,
Marvin


[1] "Once the image is loaded, LoadImage() installsEFI_HII_PACKAGE_LIST_PROTOCOL on the handle if the image conta= ins a
custom PE/COFF resource with the type 'HII'."
- UEFI 2.9, 7.4, "EFI_BOOT_SERVICES.LoadImage()"













--Apple-Mail=_8007B5E6-1026-4680-81BA-0298483AF551--