From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out03.hibox.biz (out03.hibox.biz [210.71.195.41]) by mx.groups.io with SMTP id smtpd.web12.32777.1629993730678755348 for ; Thu, 26 Aug 2021 09:02:11 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: insyde.com, ip: 210.71.195.41, mailfrom: tim.lewis@insyde.com) IronPort-SDR: dLoj2yGk5Yx+OTHVqq3bawwdgZuKsmPGA4P4NIeJLKDWdNudvtMp70kOFMfzDZh93PqgjBtS1d 4p2hxwkEcisw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2B2AQBqtydh/ws0GKxaHAEBAQEBAQc?= =?us-ascii?q?BARIBAQQEAQFACYE/BAEBCwEBgyBWF1WEAUaJBIgGAzgBj0JcGosYgREDVAs?= =?us-ascii?q?BAQEBAQEBAQEJOQUBAgQBASsBgUuCMD8EAgKCMSY3Bg4BAgQBAQESAQEGAQE?= =?us-ascii?q?BAQEGBIEkhWgNQAEQAYVwAQECAwgCGSYFCCMMAQUGAw0EBAEBAwImAhsYHgg?= =?us-ascii?q?CBAESCwULAgSCUYJVAy8PqWZ6gTEaAmWDTQGDUxmCPIEQKgGBZIUchnmCUIE?= =?us-ascii?q?VgihRMD6BTwGBEgEBAhiBEByDMYJkBIU7E0gdFzQGFBMBGwoEAgUdLgM9CAE?= =?us-ascii?q?CFBYIDQQKFQUCBAcuCyAPkRgigwFGiyebSD6BGAeNbYdAB4xTLINlgUiQSAM?= =?us-ascii?q?XkG2QX4U4jEKTZ4UogXdrgRNwUIJpCTYRGY1UZxaCP4ERgmSCBSuDGYJRJDA?= =?us-ascii?q?CCS0CBgEKAQEDCTIBhS0IFQGCcIZYAQE?= X-IronPort-AV: E=Sophos;i="5.84,353,1620662400"; d="scan'208";a="58518930" Received: from unknown (HELO hb3-BKT201.hibox.biz) ([172.24.52.11]) by out03.hibox.biz with ESMTP; 27 Aug 2021 00:02:05 +0800 IronPort-SDR: f/eadVIWDxDbYyYvC8e97Ot1nDienKSHouyCcq9P25Vz96V7/aI0PAvF1Ogi9tRaOaiTzZYxnG pMmrlHpoMxYw== Received: from unknown (HELO hb3-BKT102.hibox.biz) ([172.24.51.12]) by hb3-BKT201.hibox.biz with ESMTP; 27 Aug 2021 00:02:04 +0800 IronPort-SDR: /YJGqx/5+Kzv53nNHB25GAkSn8/1nrk4ywzcAPW2wykuDdHmASEfRV5xGi19DP9jY8X6iqm9E2 MUAAgheYwEHA== Received: from unknown (HELO hb3-IN03.hibox.biz) ([172.24.12.13]) by hb3-BKT102.hibox.biz with ESMTP; 27 Aug 2021 00:02:04 +0800 IronPort-SDR: LycEVLPLhLsImC9yL0f0LY8V6CETQCkXzwg30WkyJ4Im0GvQvr8eYlzmZmKfHoNWA2ZbC4zfsc B9kjqb6DaOnw== X-Remote-IP: 73.116.1.175 X-Remote-Host: c-73-116-1-175.hsd1.ca.comcast.net X-SBRS: -10.0 X-MID: 63284996 X-Auth-ID: tim.lewis@insyde.com X-EnvelopeFrom: tim.lewis@insyde.com hiBox-Sender: 1 IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AmDWobKg5Y1YtpOkb/2cFRbtbvnBQXu0ji2?= =?us-ascii?q?hC6mlwRA09TyVXra2TdZMgpHrJYVcqKRMdsPqHP7SNRm6Z8JZz75UYM7vKZn?= =?us-ascii?q?iBhILGFuBfBOfZqAEIXheQygc/79YCT0EdMr3N5DFB5K7HCUuDf+rIq+PozE?= =?us-ascii?q?ncv5a4854Cd2tXgtlbnmNENjo=3D?= Received: from c-73-116-1-175.hsd1.ca.comcast.net (HELO DESKTOPHG9V3E8) ([73.116.1.175]) by hb3-IN03.hibox.biz with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 00:02:00 +0800 From: "Tim Lewis" To: , , Cc: "'Andrew Fish'" , , "'Ni, Ray'" , "'Gao, Zhichao'" , "'Wang, Jian J'" , "'Wu, Hao A'" , "'Bi, Dandan'" , "'Dong, Eric'" , "'Bret Barkelew'" , "'Vitaly Cheptsov'" In-Reply-To: Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables Date: Thu, 26 Aug 2021 09:01:57 -0700 Message-ID: <1e7201d79a93$b5386140$1fa923c0$@insyde.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQFjmrNAgbvveKoQmN5BPy4Md9rRPKxt8EPA Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-language: en-us Hi Marvin -- I would like to add some historical perspective on this. One of the design = requirements back when HII was first introduced into the UEFI specification= after Intel's initial contribution was that of binary editability. In orde= r to be able to reliably find, extract and then re-insert the HII data into= the binary, it needed to be discoverable and not affect the offsets of the= code and data in the binary.=20 While it was possible to put some sort of signature in the read-only data s= ections of the binary and leave padding for possible future edits, it was f= elt that using a PE/COFF section similar to the resource sections was bette= r. Resource sections are commonly in use for PE/COFF files (in Windows) and= the similar idea existed in the then-extant Mac binary format (resource fo= rk?).=20 Thanks, Tim Lewis CTO, Insyde Software www.insyde.com -----Original Message----- From: devel@edk2.groups.io On Behalf Of Michael D Ki= nney Sent: Thursday, August 26, 2021 7:37 AM To: devel@edk2.groups.io; mhaeuser@posteo.de; Kinney, Michael D Cc: Andrew Fish ; leif@nuviainc.com; Ni, Ray ; Gao, Zhichao ; Wang, Jian J ; Wu, Hao A ; Bi, Dandan ; D= ong, Eric ; Bret Barkelew ; Vitaly Cheptsov Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables Marvin, One constraint in this discussion is that the HII content in a PE/COFF reso= urce section is defined in the UEFI Specification, Which means UEFI Apps an= d UEFI Drivers from media and option ROMs that are not part of the system F= W image are allowed to use this feature, This means the system FW PE/COFF = loader must support loading HII content from this PE/COFF resource section = to be UEFI conformant. So we cannot remove this feature from the PE/COFF l= oader without changes to the UEFI Specification. Even if changes to the UE= FI Specification we made, we would have to continue to support this feature= for backward compatibility with existing UEFI Apps/Drivers that may be usi= ng this feature. Thanks, Mike > -----Original Message----- > From: devel@edk2.groups.io On Behalf Of Marvin=20 > H=C3=A4user > Sent: Thursday, August 26, 2021 1:51 AM > To: devel@edk2.groups.io; Kinney, Michael D=20 > > Cc: devel@edk2.groups.io; Andrew Fish ;=20 > leif@nuviainc.com; Ni, Ray ; Gao, Zhichao=20 > ; Wang, Jian J ; Wu, Hao=20 > A ; Bi, Dandan ; Dong, Eric=20 > ; Bret Barkelew ;=20 > Vitaly Cheptsov > Subject: Re: [edk2-devel] [RFC] Expose HII package list via C=20 > variables >=20 > Hey Mike, >=20 > Thanks for your reply! >=20 > Well, this switch is not well-documented. Looking now at both the=20 > generation code and the emitted code, it does not generate a package=20 > list like my code does, but separate data variables (strings and=20 > images) that cannot easily be passed to HiiDatabase as-is. However=20 > apparently there are drivers that use this functionality successfully=20 > by composing the package list at runtime [1]. >=20 > Looking with this information now at the pattern of using HII C=20 > variables (which I did not know existed before) vs the PE/COFF HII=20 > section, most that use latter are Shell applications, which I guess=20 > means the section has actually been introduced to resolve D.? There=20 > are exceptions such as LogoDxe [2], which use the PE/COFF section while D= . > is not a problem, hence I got confused, sorry. I think these modules=20 > should be updated in any case. Do you agree? >=20 > So, for modules that use C variables already, my patch would save some=20 > runtime generation code and dynamic memory allocation for the HII=20 > package list. This was not my goal (as I said, I didn't realise HII C=20 > variables already were a thing in the first place), but the changes=20 > are small enough that it might be worth considering anyway, in my opinion= . > If a HII package list is generated for both Shell and non-Shell apps,=20 > this also means code paths can be unified. For example, there could be=20 > a library class with constructor and destructor to add/remove packages=20 > from the HII database for all modules that use such, Shell or not. For=20 > BaseTools it means that there is no real need for separate Python and=20 > C paths as ideally they just generate the exact same data. >=20 > Now to D., the only usage for this seems to be that Shell can locate=20 > the help text in the executable without executing it, yet it is fully=20 > loaded anyway [3]. To be honest, I find it hard to justify loading an=20 > executable (PE/COFF loading, memory permission application, the full > process) to retrieve a help text and then unloading it again,=20 > especially with the HII code being on a core dispatcher level. 1. to=20 > 7. still hold true in my opinion. Was there any discussion I could=20 > read through why Shell apps cannot simply support a "--help" or "-?"=20 > command and output the string themselves? Pushing the burden to the=20 > Shell apps does preserve the "drawback" that a full loading is=20 > required (which honestly does not matter for a debugging application=20 > like Shell), however it does relieve the burden of PE/COFF HII parsing=20 > from the core dispatcher (which matters a lot in my opinion to keep=20 > the core simple). It would simply be a normal Shell app execution as=20 > any other however. If someone wants to avoid the PE/COFF burden=20 > altogether, they can still provide a .man file. >=20 > As for my points 6. and 7., maybe I should provide some context. Due=20 > to many issues with TE files, platforms started abandoning them and=20 > returned to PE/COFF Images. I think a big reason for this is that TE=20 > is not really a sound and complete format, but a stripped version of=20 > PE/COFF with none of the necessary fixups applied. I'm currently=20 > sketching a possible alternative [4], and I would really like to not=20 > having to specify a HII section type, while still preserving=20 > compatibility with all of the UEFI Image types and use-cases [4]. >=20 > Thanks again! >=20 > Best regards, > Marvin >=20 >=20 > [1] > https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0ea > b4ac688d5/MdeModulePkg/Application/BootManagerMenuAp > p/BootManagerMenu.c#L929-L934 >=20 > [2] > https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0ea > b4ac688d5/MdeModulePkg/Logo/LogoDxe.inf#L23 >=20 > [3] > https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0eab4a= c688d5/ShellPkg/Application/Shell/ShellManParser. > c#L646-L671 >=20 > [4] > https://github.com/mhaeuser/edk2/blob/ue_poc/MdePkg/Include/IndustrySt > andard/UeImage.h >=20 > 26.08.2021 00:34:12 Michael D Kinney : >=20 > > Hi Marvin, > > > > I think this feature is already there and supported. > > > > HII can either be in a global variable or in a PE/COFF resource section= . > > The default is a global variable because HII was implemented before=20 > > the PE/COFF resource section feature was added to the UEFI Specificatio= n. > > > > There is an INF [Defines] section statement to enable the PE/COFF=20 > > section. See UefiHiiResource in the following link. > > > > https://tianocore-docs.github.io/edk2-InfSpecification/draft/3_edk_i > > i_inf_file_format/34_[defines]_section.html#34- > defines-section > > > > How is your proposal different than this existing capability? > > > > Thanks, > > > > Mike > > > >> -----Original Message----- > >> From: devel@edk2.groups.io On Behalf Of=20 > >> Marvin H=C3=A4user > >> Sent: Wednesday, August 25, 2021 2:21 PM > >> To: devel@edk2.groups.io > >> Cc: Andrew Fish ; leif@nuviainc.com; Kinney,=20 > >> Michael D ; Ni, Ray ;=20 > >> Gao, Zhichao ; Wang, Jian J=20 > >> ; Wu, Hao A ; Bi, Dandan=20 > >> ; Dong, Eric ; Bret=20 > >> Barkelew ; Vitaly Cheptsov=20 > >> > >> Subject: [edk2-devel] [RFC] Expose HII package list via C variables > >> > >> Good day everyone, > >> > >> Currently, the HII package list is stored in a PE/COFF resource=20 > >> section [1]. I propose to store it in a C variable (byte array with=20 > >> a pointer to it and its size exposed) instead. DxeCore would have a=20 > >> guard to toggle the deprecated support for the automatic protocol=20 > >> installation. This has the following advantages: > >> > >> 1. Fixes BZ (incl. future toolchains): > >> https://bugzilla.tianocore.org/show_bug.cgi?id=3D557 > >> 2. Universal method across all toolchains and output file formats=20 > >> 3. Saves error-prone parsing work 4. Saves protocol install/locate=20 > >> work, the data is available right away 5. The omission of a=20 > >> dedicated section can save space 6. Terse file formats can support=20 > >> this and remain terse :) 7. Removes a dependency on the PE/COFF=20 > >> format specifically > >> > >> A *very rough* PoC diff can be found here: > >> https://github.com/mhaeuser/edk2/compare/master...wip_hii_cvar > >> If the feedback is positive, I will clean it up of course. OVMF=20 > >> boots with everything working fine. > >> > >> I'd explicitly like feedback on the following: > >> A. Is this an acceptable solution to BZ 557 (Andrew?)? > >> B. Is this an acceptable solution for the "HII workflow" (MdeModule=20 > >> maintainers?)? > >> C. Is it acceptable to make support UEFI-side support for the old=20 > >> mechanism optional (Stewards?)? > >> D. Can an acceptable alternative be found for the removed ShellPkg=20 > >> code (Shell maintainers?)? > >> > >> As you can see the BaseTools part also is rough, but that is more a=20 > >> question of "how" rather than "whether", so I'll postpone asking about= it. > >> > >> Thanks for your time and feedback! > >> > >> Best regards, > >> Marvin > >> > >> > >> [1] "Once the image is loaded, LoadImage() installs=20 > >> EFI_HII_PACKAGE_LIST_PROTOCOL on the handle if the image contains a=20 > >> custom PE/COFF resource with the type 'HII'." > >> - UEFI 2.9, 7.4, "EFI_BOOT_SERVICES.LoadImage()" > >> > >> > >> > >> > > > > > > > > >=20 >=20 >=20 >=20