From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by mx.groups.io with SMTP id smtpd.web09.33494.1629997384389199654 for ; Thu, 26 Aug 2021 10:03:05 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@posteo.de header.s=2017 header.b=J/CfB6iz; spf=pass (domain: posteo.de, ip: 185.67.36.65, mailfrom: mhaeuser@posteo.de) Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id AC26E24002F for ; Thu, 26 Aug 2021 19:03:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017; t=1629997382; bh=d0Ww0RiiqYWW+Wvz9va8qSlnNedkdNq5Uijh0ofdZ+s=; h=Subject:To:Cc:From:Date:From; b=J/CfB6iz8r90H15xTWLjCHbWN4/Bh+e+8Kn0SkKdFgNN3O3SleyZspCVfrsIl/rmB QZ4SdVDAP6lTZFCxQkx4SA2AaHDlYfEv/XkQGoQhM+DzITVVv4YM2zEcjlGeR7VvWO ZqaYOrPBW/Oukb84c2uRO3mQu4nmTOxN5bvetwREftUEdEFB0fjqJyDC+i/+DX+PVn 5VUphO74rebtMsUp6EpTJQkf0gsaRfwO7Poe9m46AQPXAF6ToOYUjeyd5vnAamXxAI ICg2GlRrJJ4HhqCxU09mnCu0XbXi7gpYxFQqnh5HgG6kfIq0GgTmIq6vRsplZgjgyu CnL3W3GAR4H3Q== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4GwTgS21HDz6tmR; Thu, 26 Aug 2021 19:03:00 +0200 (CEST) Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables To: Samer El-Haj-Mahmoud , "devel@edk2.groups.io" , "tim.lewis@insyde.com" , "michael.d.kinney@intel.com" Cc: 'Andrew Fish' , "leif@nuviainc.com" , "'Ni, Ray'" , "'Gao, Zhichao'" , "'Wang, Jian J'" , "'Wu, Hao A'" , "'Bi, Dandan'" , "'Dong, Eric'" , 'Bret Barkelew' , 'Vitaly Cheptsov' References: <1e7201d79a93$b5386140$1fa923c0$@insyde.com> <8ea3fa57-3091-b2c5-dba2-c3eca153b697@posteo.de> From: =?UTF-8?B?TWFydmluIEjDpHVzZXI=?= Message-ID: Date: Thu, 26 Aug 2021 17:02:59 +0000 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: quoted-printable Good day Samer, Thanks a lot for your reply! On 26/08/2021 18:49, Samer El-Haj-Mahmoud wrote: > I am aware of some proprietary tools (not open source) that depend on thi= s functionality. > > The feature has been used, especially on some server designs, to allow ex= porting HII packages and consuming them offline by out-of-band or OS-based = management applications for instance. OS-based, so we are talking about DXE Runtime Drivers with HII support?=20 Can you share how they retrieve the data, i.e. do they parse the PE/COFF=20 header at OS runtime, or is there maybe some OS loader support that=20 collects all installed packages before kernel handover? As I mentioned,=20 part of the rationale is to support formats other than PE/COFF=20 (especially a new terse format), so the former mechanisms would break=20 for the long-term future no matter what (if adoption is progressing of=20 course). In the latter case, functionality should not really be affected=20 I believe, or am I missing something? The goal is not only to reduce core dispatcher parsing work, but also to=20 abstract away file format specifics with UEFI concepts (e.g. I replaced=20 all parsing of PE/COFF to retrieve the PDB with a modification of the=20 Debug Image Info Table to store the PDB file path). Publishing to the OS=20 could work with some OS-proprietary mechanism, or maybe with the help of=20 the SystemTable ConfigTable? > The need for this seems to be declining though, as implementations move t= o using standard Redfish data instead, which in turn can be produced from H= II packages, either during boot (like what is done in EDK2 RedfishPkg), or = at build / release time. How is this data retrieved then, some dynamic database the OS loader=20 passes along? Thanks! Best regards, Marvin > > Thanks, > --Samer > > >> -----Original Message----- >> From: devel@edk2.groups.io On Behalf Of Marvin >> H=C3=A4user via groups.io >> Sent: Thursday, August 26, 2021 12:07 PM >> To: tim.lewis@insyde.com; devel@edk2.groups.io; >> michael.d.kinney@intel.com >> Cc: 'Andrew Fish' ; leif@nuviainc.com; 'Ni, Ray' >> ; 'Gao, Zhichao' ; 'Wang, Jian = J' >> ; 'Wu, Hao A' ; 'Bi, Dandan' >> ; 'Dong, Eric' ; 'Bret Barkele= w' >> ; 'Vitaly Cheptsov' >> >> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables >> >> Hey Tim, >> >> Interesting, do you happen to have some tool or code that performs such >> edits at hand to take a look at? Seems like most modules already use >> variables and thus cannot be modified in such a way even right now? >> >> That's the kind of information I am looking for, thanks a lot! >> >> Best regards, >> Marvin >> >> On 26/08/2021 18:01, tim.lewis@insyde.com wrote: >>> Hi Marvin -- >>> >>> I would like to add some historical perspective on this. One of the des= ign >> requirements back when HII was first introduced into the UEFI specificat= ion >> after Intel's initial contribution was that of binary editability. In or= der to be >> able to reliably find, extract and then re-insert the HII data into the = binary, it >> needed to be discoverable and not affect the offsets of the code and dat= a in >> the binary. >>> While it was possible to put some sort of signature in the read-only da= ta >> sections of the binary and leave padding for possible future edits, it w= as felt >> that using a PE/COFF section similar to the resource sections was better= . >> Resource sections are commonly in use for PE/COFF files (in Windows) and >> the similar idea existed in the then-extant Mac binary format (resource >> fork?). >>> Thanks, >>> >>> Tim Lewis >>> CTO, Insyde Software >>> www.insyde.com >>> >>> -----Original Message----- >>> From: devel@edk2.groups.io On Behalf Of >> Michael D Kinney >>> Sent: Thursday, August 26, 2021 7:37 AM >>> To: devel@edk2.groups.io; mhaeuser@posteo.de; Kinney, Michael D >> >>> Cc: Andrew Fish ; leif@nuviainc.com; Ni, Ray >> ; Gao, Zhichao ; Wang, Jian J >> ; Wu, Hao A ; Bi, Dandan >> ; Dong, Eric ; Bret Barkelew >> ; Vitaly Cheptsov >> >>> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables >>> >>> Marvin, >>> >>> One constraint in this discussion is that the HII content in a PE/COFF >> resource section is defined in the UEFI Specification, Which means UEFI = Apps >> and UEFI Drivers from media and option ROMs that are not part of the >> system FW image are allowed to use this feature, This means the system = FW >> PE/COFF loader must support loading HII content from this PE/COFF resour= ce >> section to be UEFI conformant. So we cannot remove this feature from th= e >> PE/COFF loader without changes to the UEFI Specification. Even if chang= es to >> the UEFI Specification we made, we would have to continue to support thi= s >> feature for backward compatibility with existing UEFI Apps/Drivers that = may >> be using this feature. >>> Thanks, >>> >>> Mike >>> >>>> -----Original Message----- >>>> From: devel@edk2.groups.io On Behalf Of >> Marvin >>>> H=C3=A4user >>>> Sent: Thursday, August 26, 2021 1:51 AM >>>> To: devel@edk2.groups.io; Kinney, Michael D >>>> >>>> Cc: devel@edk2.groups.io; Andrew Fish ; >>>> leif@nuviainc.com; Ni, Ray ; Gao, Zhichao >>>> ; Wang, Jian J ; Wu, >> Hao >>>> A ; Bi, Dandan ; Dong, Eric >>>> ; Bret Barkelew ; >>>> Vitaly Cheptsov >>>> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C >>>> variables >>>> >>>> Hey Mike, >>>> >>>> Thanks for your reply! >>>> >>>> Well, this switch is not well-documented. Looking now at both the >>>> generation code and the emitted code, it does not generate a package >>>> list like my code does, but separate data variables (strings and >>>> images) that cannot easily be passed to HiiDatabase as-is. However >>>> apparently there are drivers that use this functionality successfully >>>> by composing the package list at runtime [1]. >>>> >>>> Looking with this information now at the pattern of using HII C >>>> variables (which I did not know existed before) vs the PE/COFF HII >>>> section, most that use latter are Shell applications, which I guess >>>> means the section has actually been introduced to resolve D.? There >>>> are exceptions such as LogoDxe [2], which use the PE/COFF section whil= e >> D. >>>> is not a problem, hence I got confused, sorry. I think these modules >>>> should be updated in any case. Do you agree? >>>> >>>> So, for modules that use C variables already, my patch would save some >>>> runtime generation code and dynamic memory allocation for the HII >>>> package list. This was not my goal (as I said, I didn't realise HII C >>>> variables already were a thing in the first place), but the changes >>>> are small enough that it might be worth considering anyway, in my >> opinion. >>>> If a HII package list is generated for both Shell and non-Shell apps, >>>> this also means code paths can be unified. For example, there could be >>>> a library class with constructor and destructor to add/remove packages >>>> from the HII database for all modules that use such, Shell or not. For >>>> BaseTools it means that there is no real need for separate Python and >>>> C paths as ideally they just generate the exact same data. >>>> >>>> Now to D., the only usage for this seems to be that Shell can locate >>>> the help text in the executable without executing it, yet it is fully >>>> loaded anyway [3]. To be honest, I find it hard to justify loading an >>>> executable (PE/COFF loading, memory permission application, the full >>>> process) to retrieve a help text and then unloading it again, >>>> especially with the HII code being on a core dispatcher level. 1. to >>>> 7. still hold true in my opinion. Was there any discussion I could >>>> read through why Shell apps cannot simply support a "--help" or "-?" >>>> command and output the string themselves? Pushing the burden to the >>>> Shell apps does preserve the "drawback" that a full loading is >>>> required (which honestly does not matter for a debugging application >>>> like Shell), however it does relieve the burden of PE/COFF HII parsing >>>> from the core dispatcher (which matters a lot in my opinion to keep >>>> the core simple). It would simply be a normal Shell app execution as >>>> any other however. If someone wants to avoid the PE/COFF burden >>>> altogether, they can still provide a .man file. >>>> >>>> As for my points 6. and 7., maybe I should provide some context. Due >>>> to many issues with TE files, platforms started abandoning them and >>>> returned to PE/COFF Images. I think a big reason for this is that TE >>>> is not really a sound and complete format, but a stripped version of >>>> PE/COFF with none of the necessary fixups applied. I'm currently >>>> sketching a possible alternative [4], and I would really like to not >>>> having to specify a HII section type, while still preserving >>>> compatibility with all of the UEFI Image types and use-cases [4]. >>>> >>>> Thanks again! >>>> >>>> Best regards, >>>> Marvin >>>> >>>> >>>> [1] >>>> >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0e >> a >>>> b4ac688d5/MdeModulePkg/Application/BootManagerMenuAp >>>> p/BootManagerMenu.c#L929-L934 >>>> >>>> [2] >>>> >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0e >> a >>>> b4ac688d5/MdeModulePkg/Logo/LogoDxe.inf#L23 >>>> >>>> [3] >>>> >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0e >> ab4ac688d5/ShellPkg/Application/Shell/ShellManParser. >>>> c#L646-L671 >>>> >>>> [4] >>>> >> https://github.com/mhaeuser/edk2/blob/ue_poc/MdePkg/Include/Industr >> ySt >>>> andard/UeImage.h >>>> >>>> 26.08.2021 00:34:12 Michael D Kinney : >>>> >>>>> Hi Marvin, >>>>> >>>>> I think this feature is already there and supported. >>>>> >>>>> HII can either be in a global variable or in a PE/COFF resource secti= on. >>>>> The default is a global variable because HII was implemented before >>>>> the PE/COFF resource section feature was added to the UEFI >> Specification. >>>>> There is an INF [Defines] section statement to enable the PE/COFF >>>>> section. See UefiHiiResource in the following link. >>>>> >>>>> https://tianocore-docs.github.io/edk2-InfSpecification/draft/3_edk_i >>>>> i_inf_file_format/34_[defines]_section.html#34- >>>> defines-section >>>>> How is your proposal different than this existing capability? >>>>> >>>>> Thanks, >>>>> >>>>> Mike >>>>> >>>>>> -----Original Message----- >>>>>> From: devel@edk2.groups.io On Behalf Of >>>>>> Marvin H=C3=A4user >>>>>> Sent: Wednesday, August 25, 2021 2:21 PM >>>>>> To: devel@edk2.groups.io >>>>>> Cc: Andrew Fish ; leif@nuviainc.com; Kinney, >>>>>> Michael D ; Ni, Ray ; >>>>>> Gao, Zhichao ; Wang, Jian J >>>>>> ; Wu, Hao A ; Bi, >> Dandan >>>>>> ; Dong, Eric ; Bret >>>>>> Barkelew ; Vitaly Cheptsov >>>>>> >>>>>> Subject: [edk2-devel] [RFC] Expose HII package list via C variables >>>>>> >>>>>> Good day everyone, >>>>>> >>>>>> Currently, the HII package list is stored in a PE/COFF resource >>>>>> section [1]. I propose to store it in a C variable (byte array with >>>>>> a pointer to it and its size exposed) instead. DxeCore would have a >>>>>> guard to toggle the deprecated support for the automatic protocol >>>>>> installation. This has the following advantages: >>>>>> >>>>>> 1. Fixes BZ (incl. future toolchains): >>>>>> https://bugzilla.tianocore.org/show_bug.cgi?id=3D557 >>>>>> 2. Universal method across all toolchains and output file formats >>>>>> 3. Saves error-prone parsing work 4. Saves protocol install/locate >>>>>> work, the data is available right away 5. The omission of a >>>>>> dedicated section can save space 6. Terse file formats can support >>>>>> this and remain terse :) 7. Removes a dependency on the PE/COFF >>>>>> format specifically >>>>>> >>>>>> A *very rough* PoC diff can be found here: >>>>>> https://github.com/mhaeuser/edk2/compare/master...wip_hii_cvar >>>>>> If the feedback is positive, I will clean it up of course. OVMF >>>>>> boots with everything working fine. >>>>>> >>>>>> I'd explicitly like feedback on the following: >>>>>> A. Is this an acceptable solution to BZ 557 (Andrew?)? >>>>>> B. Is this an acceptable solution for the "HII workflow" (MdeModule >>>>>> maintainers?)? >>>>>> C. Is it acceptable to make support UEFI-side support for the old >>>>>> mechanism optional (Stewards?)? >>>>>> D. Can an acceptable alternative be found for the removed ShellPkg >>>>>> code (Shell maintainers?)? >>>>>> >>>>>> As you can see the BaseTools part also is rough, but that is more a >>>>>> question of "how" rather than "whether", so I'll postpone asking abo= ut >> it. >>>>>> Thanks for your time and feedback! >>>>>> >>>>>> Best regards, >>>>>> Marvin >>>>>> >>>>>> >>>>>> [1] "Once the image is loaded, LoadImage() installs >>>>>> EFI_HII_PACKAGE_LIST_PROTOCOL on the handle if the image contains >> a >>>>>> custom PE/COFF resource with the type 'HII'." >>>>>> - UEFI 2.9, 7.4, "EFI_BOOT_SERVICES.LoadImage()" >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> >>> >> >> >>=20 >> > IMPORTANT NOTICE: The contents of this email and any attachments are conf= idential and may also be privileged. If you are not the intended recipient,= please notify the sender immediately and do not disclose the contents to a= ny other person, use it for any purpose, or store or copy the information i= n any medium. Thank you.