Re: [PATCH v3 1/7] MdeModulePkg: introduce PE/COFF image emulator protocol

["Kinney, Michael D" <michael.d.kinney@intel.com>]

Hi Ard,

Yes.  I think it is simpler if an emulator module registers
one machine type at a time.  If an emulator module wants
to do a single registration and that registration supports
multiple machine types, then the emulator implementation
is more complex, but it is still feasible.

I think I see where things got confused.  The proposal from
Star for a new protocol produced by the DXE Core has some issues.
A refined proposal is shown below.  It also supports the idea that
an emulator module can be loaded and unloaded to support testing
from environments like the UEFI Shell to soft load an emulator,
test some images, and unload it.

Also, by having a protocol produced by the DXE Core, the emulator
module can tell if the DXE Core supports emulators or not 
by checking for the presence of this protocol.  The emulator
module can fail gracefully with some DEBUG() message and unload
if DXE Core does not support emulators.  This also allows the
option to update the DXE Core module to optionally support 
emulators using a PCD Feature Flag and remove some logic if
emulators are not required on a specific platform.

typedef
EFI_STATUS
(EFIAPI *EDKII_PECOFF_IMAGE_EMULATOR_ADD_EMULATOR) (
  IN EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL            *This,
  IN EDKII_PECOFF_IMAGE_EMULATOR_IS_IMAGE_SUPPORTED  IsImageSupported,
  IN EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_IMAGE      RegisterImage,
  IN EDKII_PECOFF_IMAGE_EMULATOR_UNREGISTER_IMAGE    UnregisterImage
  );

typedef
EFI_STATUS
(EFIAPI *EDKII_PECOFF_IMAGE_EMULATOR_REMOVE_EMULATOR) (
  IN EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL            *This,
  IN EDKII_PECOFF_IMAGE_EMULATOR_IS_IMAGE_SUPPORTED  IsImageSupported,
  IN EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_IMAGE      RegisterImage,
  IN EDKII_PECOFF_IMAGE_EMULATOR_UNREGISTER_IMAGE    UnregisterImage
  );

typedef struct _EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL {
  EDKII_PECOFF_IMAGE_EMULATOR_ADD_EMULATOR     AddEmulator;
  EDKII_PECOFF_IMAGE_EMULATOR_REMOVE_EMULATOR  RemoveEmulator;
} EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL

The IsImageSupported(), RegisterImage(), and UnRegisterImage()
are added in a set in a single call to AddEmulator() from the
emulator module.  In LoadImage(), if IsImageSupported() returns
TRUE, LoadImage() uses the RegisterImage() and UnRegisterImage()
APIs that were added with the IsImageSupported() API in the
call to AddEmulator().

Thanks,

Mike

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> Sent: Thursday, September 27, 2018 3:59 AM
> To: Zeng, Star <star.zeng@intel.com>
> Cc: Kinney, Michael D <michael.d.kinney@intel.com>;
> edk2-devel@lists.01.org; Zimmer, Vincent
> <vincent.zimmer@intel.com>; Richardson, Brian
> <brian.richardson@intel.com>; Andrew Fish
> <afish@apple.com>; Leif Lindholm
> <leif.lindholm@linaro.org>; Dong, Eric
> <eric.dong@intel.com>; Ni, Ruiyu <ruiyu.ni@intel.com>;
> Gao, Liming <liming.gao@intel.com>; Carsey, Jaben
> <jaben.carsey@intel.com>; Shi, Steven
> <steven.shi@intel.com>
> Subject: Re: [PATCH v3 1/7] MdeModulePkg: introduce
> PE/COFF image emulator protocol
> 
> On 27 September 2018 at 02:48, Zeng, Star
> <star.zeng@intel.com> wrote:
> > Yes. This idea also came to my mind last night, then
> no need introduce PeCoffEmulatorDxe and no platform
> change is needed.
> >
> 
> The only problem with this approach is that we cannot
> keep track of
> which emulator returned TRUE for IsSupported(), and so
> RegisterImage()
> and also UnregisterImage() will have to do some
> internal bookkeeping
> in the 'manager' protocol implementation that
> duplicates code in the
> emulators.
> 
> So could we flesh this out a bit before I dive into the
> code again and
> turn everything upside down?
> 
> - the PCI bus driver no longer checks the machine type
> - the BDS no longer checks the machine type for
> Driver#### images
> - CoreLoadImage() checks IsSupported() to establish
> whether the
> emulator manager has an emulator available that is
> willing to take
> charge of the image
> - what happens next? CoreLoadImage() calls
> RegisterImage() on what?
> Should IsSupported() return some kind of handle?
> 
> Also, perhaps we should simply stick with one emulator
> per machine
> type (which doesn't preclude an implementation from
> registering itself
> for multiple ones). That simplifies the IsSupported vs
> RegisterImage
> issue above as well (i.e., it is unambiguous *which*
> RegisterImage()
> should be called)
> 
> 
> 
> > -----Original Message-----
> > From: Kinney, Michael D
> > Sent: Thursday, September 27, 2018 1:32 AM
> > To: Ard Biesheuvel <ard.biesheuvel@linaro.org>; Zeng,
> Star <star.zeng@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>
> > Cc: edk2-devel@lists.01.org; Zimmer, Vincent
> <vincent.zimmer@intel.com>; Richardson, Brian
> <brian.richardson@intel.com>; Andrew Fish
> <afish@apple.com>; Leif Lindholm
> <leif.lindholm@linaro.org>; Dong, Eric
> <eric.dong@intel.com>; Ni, Ruiyu <ruiyu.ni@intel.com>;
> Gao, Liming <liming.gao@intel.com>; Carsey, Jaben
> <jaben.carsey@intel.com>; Shi, Steven
> <steven.shi@intel.com>
> > Subject: RE: [PATCH v3 1/7] MdeModulePkg: introduce
> PE/COFF image emulator protocol
> >
> > Hi Ard,
> >
> > I think this registration protocol would be a new
> protocol in the MdeModulePkg and the protocol would be
> produced by the DXE Core.  The emulation drivers,
> including EBC would consume this protocol to register
> their services.  This removes the need for the DXE Core
> to do a Register Protocol Notify event for the emulator
> protocol.  The DXE Core would build a table of
> registered services, so it can quickly loop through
> them to check if an emulator supports a specific
> PE/COFF image or not.
> >
> > Best regards,
> >
> > Mike
> >
> >> -----Original Message-----
> >> From: Ard Biesheuvel
> [mailto:ard.biesheuvel@linaro.org]
> >> Sent: Wednesday, September 26, 2018 3:14 AM
> >> To: Zeng, Star <star.zeng@intel.com>
> >> Cc: edk2-devel@lists.01.org; Zimmer, Vincent
> >> <vincent.zimmer@intel.com>; Richardson, Brian
> >> <brian.richardson@intel.com>; Kinney, Michael D
> >> <michael.d.kinney@intel.com>; Andrew Fish
> <afish@apple.com>; Leif
> >> Lindholm <leif.lindholm@linaro.org>; Dong, Eric
> <eric.dong@intel.com>;
> >> Ni, Ruiyu <ruiyu.ni@intel.com>; Gao, Liming
> <liming.gao@intel.com>;
> >> Carsey, Jaben <jaben.carsey@intel.com>; Shi, Steven
> >> <steven.shi@intel.com>
> >> Subject: Re: [PATCH v3 1/7] MdeModulePkg: introduce
> PE/COFF image
> >> emulator protocol
> >>
> >> On Wed, 26 Sep 2018 at 12:07, Zeng, Star
> <star.zeng@intel.com> wrote:
> >> >
> >> > A little late feedback. Just an idea.
> >> >
> >> > Do you think whether it can make the consumer
> simpler
> >> like below or not?
> >> >
> >> > Add a new API RegisterInterfaces in the protocol
> and
> >> add a wrapper driver PeCoffEmulatorDxe.
> >> > The emulators can call RegisterInterfaces, and
> >> consumer will call other APIs simply,
> PeCoffEmulatorDxe will be a
> >> wrapper as manager.
> >> >
> >> > typedef
> >> > EFI_STATUS
> >> > (EFIAPI
> >> *EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_INTERFACES) (
> >> >   IN EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> *This,
> >> >   IN
> EDKII_PECOFF_IMAGE_EMULATOR_IS_IMAGE_SUPPORTED
> >> IsImageSupported,
> >> >   IN EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_IMAGE
> >> RegisterImage,
> >> >   IN EDKII_PECOFF_IMAGE_EMULATOR_UNREGISTER_IMAGE
> >> UnregisterImage
> >> >   );
> >> >
> >> > typedef struct
> _EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL {
> >> >   EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_INTERFACES
> >> RegisterInterfaces;
> >> >   EDKII_PECOFF_IMAGE_EMULATOR_IS_IMAGE_SUPPORTED
> >> IsImageSupported;
> >> >   EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_IMAGE
> >> RegisterImage;
> >> >   EDKII_PECOFF_IMAGE_EMULATOR_UNREGISTER_IMAGE
> >> UnregisterImage;
> >> > } EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL;
> >> >
> >>
> >> Hi Star,
> >>
> >> Thanks for taking a look.
> >>
> >> I tried to avoid introducing new drivers or library
> classes because it
> >> will break existing platforms that incorporate
> EbcDxe.
> >> Do you think
> >> this is not an issue?
> >>
> >> > -----Original Message-----
> >> > From: Ard Biesheuvel
> >> [mailto:ard.biesheuvel@linaro.org]
> >> > Sent: Friday, September 21, 2018 7:02 AM
> >> > To: edk2-devel@lists.01.org
> >> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>;
> >> Zimmer, Vincent <vincent.zimmer@intel.com>;
> Richardson, Brian
> >> <brian.richardson@intel.com>; Kinney, Michael D
> >> <michael.d.kinney@intel.com>; Andrew Fish
> <afish@apple.com>; Leif
> >> Lindholm <leif.lindholm@linaro.org>; Zeng, Star
> <star.zeng@intel.com>;
> >> Dong, Eric <eric.dong@intel.com>; Ni, Ruiyu
> <ruiyu.ni@intel.com>; Gao,
> >> Liming <liming.gao@intel.com>; Carsey, Jaben
> <jaben.carsey@intel.com>;
> >> Shi, Steven <steven.shi@intel.com>
> >> > Subject: [PATCH v3 1/7] MdeModulePkg: introduce
> >> PE/COFF image emulator protocol
> >> >
> >> > Introduce a protocol that can be invoked by the
> image
> >> loading services to execute foreign architecture
> PE/COFF images via an
> >> emulator.
> >> >
> >> > Contributed-under: TianoCore Contribution
> Agreement
> >> 1.1
> >> > Signed-off-by: Ard Biesheuvel
> >> <ard.biesheuvel@linaro.org>
> >> > ---
> >> >
> MdeModulePkg/Include/Protocol/PeCoffImageEmulator.h |
> >> 102 ++++++++++++++++++++
> >> >  MdeModulePkg/MdeModulePkg.dec
> |
> >> 4 +
> >> >  2 files changed, 106 insertions(+)
> >> >
> >> > diff --git
> >>
> a/MdeModulePkg/Include/Protocol/PeCoffImageEmulator.h
> >>
> b/MdeModulePkg/Include/Protocol/PeCoffImageEmulator.h
> >> > new file mode 100644
> >> > index 000000000000..27bad556209c
> >> > --- /dev/null
> >> > +++
> >>
> b/MdeModulePkg/Include/Protocol/PeCoffImageEmulator.h
> >> > @@ -0,0 +1,102 @@
> >> > +/** @file
> >> > +  Copyright (c) 2018, Linaro, Ltd. All rights
> >> reserved.<BR>
> >> > +
> >> > +  This program and the accompanying materials are
> >> licensed and made
> >> > + available  under the terms and conditions of the
> BSD
> >> License which
> >> > + accompanies this  distribution.  The full text
> of
> >> the license may be
> >> > + found at  http://opensource.org/licenses/bsd-
> >> license.php
> >> > +
> >> > +  THE PROGRAM IS DISTRIBUTED UNDER THE BSD
> LICENSE ON
> >> AN "AS IS" BASIS,
> >> > + WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY
> KIND,
> >> EITHER EXPRESS OR IMPLIED.
> >> > +
> >> > +**/
> >> > +
> >> > +#ifndef PECOFF_IMAGE_EMULATOR_PROTOCOL_GUID_H
> >> > +#define PECOFF_IMAGE_EMULATOR_PROTOCOL_GUID_H
> >> > +
> >> > +#define EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL_GUID
> \
> >> > +  { 0x96F46153, 0x97A7, 0x4793, { 0xAC, 0xC1,
> 0xFA,
> >> 0x19, 0xBF, 0x78,
> >> > +0xEA, 0x97 } }
> >> > +
> >> > +typedef struct
> _EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> >> > +EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL;
> >> > +
> >> > +/**
> >> > +  Check whether the emulator supports executing a
> >> certain PE/COFF image
> >> > +
> >> > +  @param[in] This         This pointer for
> >> EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> >> > +                          structure  @param[in]
> MachineType  The
> >> > + machine type for which
> >> the image was built
> >> > +  @param[in] ImageType    Whether the image is an
> >> application, a boot time
> >> > +                          driver or a runtime
> driver.
> >> > +  @param[in] DevicePath   Path to device where
> the
> >> image originated
> >> > +                          (e.g., a PCI option
> ROM)
> >> > +
> >> > +  @retval TRUE            The image is supported
> by
> >> the emulator
> >> > +  @retval FALSE           The image is not
> supported
> >> by the emulator.
> >> > +**/
> >> > +typedef
> >> > +BOOLEAN
> >> > +(EFIAPI
> >> *EDKII_PECOFF_IMAGE_EMULATOR_IS_IMAGE_SUPPORTED) (
> >> > +  IN  EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> *This,
> >> > +  IN  UINT16
> >> MachineType,
> >> > +  IN  UINT16
> >> ImageType,
> >> > +  IN  EFI_DEVICE_PATH_PROTOCOL
> >> *DevicePath   OPTIONAL
> >> > +  );
> >> > +
> >> > +/**
> >> > +  Register a supported PE/COFF image with the
> >> emulator. After this call
> >> > +  completes successfully, the PE/COFF image may
> be
> >> started as usual,
> >> > +and
> >> > +  it is the responsibility of the emulator
> >> implementation that any
> >> > +branch
> >> > +  into the code section of the image (including
> >> returns from functions
> >> > +called
> >> > +  from the foreign code) is executed as if it
> were
> >> running on the
> >> > +machine
> >> > +  type it was built for.
> >> > +
> >> > +  @param[in]      This          This pointer for
> >> > +
> >> EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL structure
> >> > +  @param[in]      ImageBase     The base address
> in
> >> memory of the PE/COFF image
> >> > +  @param[in]      ImageSize     The size in
> memory of
> >> the PE/COFF image
> >> > +  @param[in,out]  EntryPoint    The entry point
> of
> >> the PE/COFF image. Passed by
> >> > +                                reference so that
> the
> >> emulator may modify it.
> >> > +
> >> > +  @retval EFI_SUCCESS           The image was
> >> registered with the emulator and
> >> > +                                can be started as
> >> usual.
> >> > +  @retval other                 The image could
> not
> >> be registered.
> >> > +
> >> > +  If the PE/COFF machine type or image type are
> not
> >> supported by the
> >> > +emulator,
> >> > +  then ASSERT().
> >> > +**/
> >> > +typedef
> >> > +EFI_STATUS
> >> > +(EFIAPI
> *EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_IMAGE)
> >> (
> >> > +  IN      EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> >> *This,
> >> > +  IN      EFI_PHYSICAL_ADDRESS
> >> ImageBase,
> >> > +  IN      UINT64
> >> ImageSize,
> >> > +  IN  OUT EFI_IMAGE_ENTRY_POINT
> >> *EntryPoint
> >> > +  );
> >> > +
> >> > +/**
> >> > +  Unregister a PE/COFF image that has been
> registered
> >> with the emulator.
> >> > +  This should be done before the image is
> unloaded
> >> from memory.
> >> > +
> >> > +  @param[in] This         This pointer for
> >> EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> >> > +                          structure
> >> > +  @param[in] ImageBase    The base address in
> memory
> >> of the PE/COFF image
> >> > +
> >> > +  @retval EFI_SUCCESS     The image was
> unregistered
> >> with the emulator.
> >> > +  @retval other           Image could not be
> >> unloaded.
> >> > +**/
> >> > +typedef
> >> > +EFI_STATUS
> >> > +(EFIAPI
> >> *EDKII_PECOFF_IMAGE_EMULATOR_UNREGISTER_IMAGE) (
> >> > +  IN  EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> *This,
> >> > +  IN  EFI_PHYSICAL_ADDRESS
> >> ImageBase
> >> > +  );
> >> > +
> >> > +typedef struct
> _EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL
> >> {
> >> > +  EDKII_PECOFF_IMAGE_EMULATOR_IS_IMAGE_SUPPORTED
> >> IsImageSupported;
> >> > +  EDKII_PECOFF_IMAGE_EMULATOR_REGISTER_IMAGE
> >> RegisterImage;
> >> > +  EDKII_PECOFF_IMAGE_EMULATOR_UNREGISTER_IMAGE
> >> UnregisterImage;
> >> > +} EDKII_PECOFF_IMAGE_EMULATOR_PROTOCOL;
> >> > +
> >> > +extern EFI_GUID
> >> gEdkiiPeCoffImageEmulatorProtocolGuid;
> >> > +
> >> > +#endif
> >> > diff --git a/MdeModulePkg/MdeModulePkg.dec
> >> b/MdeModulePkg/MdeModulePkg.dec index
> >> 6a6d9660edc2..be307329f901 100644
> >> > --- a/MdeModulePkg/MdeModulePkg.dec
> >> > +++ b/MdeModulePkg/MdeModulePkg.dec
> >> > @@ -617,6 +617,10 @@
> >> >
> >> >    ## Include/Protocol/AtaAtapiPolicy.h
> >> >    gEdkiiAtaAtapiPolicyProtocolGuid = {
> 0xe59cd769,
> >> 0x5083, 0x4f26,{ 0x90, 0x94, 0x6c, 0x91, 0x9f, 0x91,
> 0x6c, 0x4e } }
> >> > +
> >> > +  ## Include/Protocol/PeCoffImageEmulator.h
> >> > +  gEdkiiPeCoffImageEmulatorProtocolGuid = {
> >> 0x96f46153, 0x97a7, 0x4793,
> >> > + { 0xac, 0xc1, 0xfa, 0x19, 0xbf, 0x78, 0xea, 0x97
> } }
> >> > +
> >> >  #
> >> >  # [Error.gEfiMdeModulePkgTokenSpaceGuid]
> >> >  #   0x80000001 | Invalid value provided.
> >> > --
> >> > 2.17.1
> >> >