BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified

BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified

[Leif Lindholm]

Hi Bob, Liming,

So, I've been playing around with merging the .dsc/.fdf files in
OvmfPkg. Merging IA32 and X64 is simple enough, since you can test on

!if $(ARCH) == "X64"

However, from what I can tell, when trying to merge in the
OvmfPkgIa32X64 platform - what you actually end up with in $(ARCH)
when specifying multiple -a is a python list rather than a single
string.

To explain the problem in code, tryin to build the platform described by

---
[Defines]
  PLATFORM_NAME                  = Ovmf
  PLATFORM_GUID                  = 5a9e7754-d81b-49ea-85ad-69eaa7b1539b
  PLATFORM_VERSION               = 0.1
  DSC_SPECIFICATION              = 0x00010005
  SUPPORTED_ARCHITECTURES        = IA32|X64
  BUILD_TARGETS                  = DEBUG|NOOPT|RELEASE
!if "$(ARCH)" == "IA32 X64" or "$(ARCH)" == "X64 IA32"
  OUTPUT_DIRECTORY               = Build/OvmfDummy
!else
  OUTPUT_DIRECTORY               = Build/Ovmf$(ARCH)Real
!endif
---

using the command line 'build -t GCC5 -a IA32 -a X64 -b DEBUG -p Test.dsc'
inevitably results in an output like

---
Build environment: Linux-4.9.0-3-amd64-x86_64-with-debian-9.7
Build start time: 18:21:33, Feb.04 2019

WORKSPACE        = /work/git/edk2
EDK_TOOLS_PATH   = /work/git/edk2/BaseTools
CONF_PATH        = /work/git/edk2/Conf
PYTHON_COMMAND   = /usr/bin/python3.5



build.py...
/work/git/edk2/Test.dsc(11): error 3001: No space is allowed in
OUTPUT_DIRECTORY
	Build/OvmfIA32 X64Real

- Failed -
---

Now, the Ia32X64 target is very much of a special case, which I don't
necessarily see as usefully supported by the current .dsc
specification. But I believe we need to do one of
- banning (simultaneous) multi-architecture platforms
- treating them the same as multi-target (-b) builds (build them separately)
- have a defined way of handling them

So, am I missing something, or does this require a change in
BaseTools?

Best Regards,

Leif

Re: BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified

[Laszlo Ersek]

On 02/04/19 19:31, Leif Lindholm wrote:
> Hi Bob, Liming,
> 
> So, I've been playing around with merging the .dsc/.fdf files in
> OvmfPkg. Merging IA32 and X64 is simple enough, since you can test on
> 
> !if $(ARCH) == "X64"
> 
> However, from what I can tell, when trying to merge in the
> OvmfPkgIa32X64 platform - what you actually end up with in $(ARCH)
> when specifying multiple -a is a python list rather than a single
> string.
> 
> To explain the problem in code, tryin to build the platform described by
> 
> ---
> [Defines]
>   PLATFORM_NAME                  = Ovmf
>   PLATFORM_GUID                  = 5a9e7754-d81b-49ea-85ad-69eaa7b1539b
>   PLATFORM_VERSION               = 0.1
>   DSC_SPECIFICATION              = 0x00010005
>   SUPPORTED_ARCHITECTURES        = IA32|X64
>   BUILD_TARGETS                  = DEBUG|NOOPT|RELEASE
> !if "$(ARCH)" == "IA32 X64" or "$(ARCH)" == "X64 IA32"
>   OUTPUT_DIRECTORY               = Build/OvmfDummy
> !else
>   OUTPUT_DIRECTORY               = Build/Ovmf$(ARCH)Real
> !endif
> ---
> 
> using the command line 'build -t GCC5 -a IA32 -a X64 -b DEBUG -p Test.dsc'
> inevitably results in an output like
> 
> ---
> Build environment: Linux-4.9.0-3-amd64-x86_64-with-debian-9.7
> Build start time: 18:21:33, Feb.04 2019
> 
> WORKSPACE        = /work/git/edk2
> EDK_TOOLS_PATH   = /work/git/edk2/BaseTools
> CONF_PATH        = /work/git/edk2/Conf
> PYTHON_COMMAND   = /usr/bin/python3.5
> 
> 
> 
> build.py...
> /work/git/edk2/Test.dsc(11): error 3001: No space is allowed in
> OUTPUT_DIRECTORY
> 	Build/OvmfIA32 X64Real
> 
> - Failed -
> ---
> 
> Now, the Ia32X64 target is very much of a special case, which I don't
> necessarily see as usefully supported by the current .dsc
> specification. But I believe we need to do one of
> - banning (simultaneous) multi-architecture platforms
> - treating them the same as multi-target (-b) builds (build them separately)
> - have a defined way of handling them

Only option #3 can work here. OvmfPkgIa32X64.dsc is a platform where the
Reset Vector, SEC and PEI phases are built for IA32, the rest (i.e.
DXE+) is built for X64, and the images are finally organized into a
single flash device (FD).

The key PCD is

  gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|TRUE

which you won't see in either the pure IA32 or the pure X64 OVMF
platform -- as none of those have to switch from 32-bit mode to 64-bit
(long) mode at the PEI/DXE boundary:

- OvmfPkgIa32.dsc:
  gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|FALSE

- OvmfPkgIa32X64.dsc:
  gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|TRUE

- OvmfPkgX64.dsc:
  gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|FALSE


Regarding the usefulness of the OvmfPkgIa32X64.dsc, that's presently the
*only* OVMF platform that is suitable for production use. It is the only
platform that:

(a) supports Secure Boot, *and*
(b) includes ACPI S3 suspend/resume support, *and*
(c) protects sensitive data related to (a) and (b), i.e. non-volatile
UEFI variables, and the LockBox data structure, respectively, by
inclusion of the SMM driver stack, and usage of SMRAM, *and*
(d) supports 64-bit OS-es.

> So, am I missing something, or does this require a change in
> BaseTools?

I believe you may have missed that "-a IA32 -a X64" stands for more than
just "build this set of modules for each of IA32 and X64, separately".

AIUI, "-a IA32 -a X64" it means "build the Components.* sections
(plural) as dictated by the '-a' flags, and then organize the full
(multi-arch) set of modules into the flash device(s), described by
FLASH_DEFINITION".

Thanks,
Laszlo

Re: BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified

[Leif Lindholm]

On Mon, Feb 04, 2019 at 08:02:55PM +0100, Laszlo Ersek wrote:
> > Now, the Ia32X64 target is very much of a special case, which I don't
> > necessarily see as usefully supported by the current .dsc
> > specification. But I believe we need to do one of
> > - banning (simultaneous) multi-architecture platforms
> > - treating them the same as multi-target (-b) builds (build them separately)
> > - have a defined way of handling them
> 
> Only option #3 can work here. OvmfPkgIa32X64.dsc is a platform where the
> Reset Vector, SEC and PEI phases are built for IA32, the rest (i.e.
> DXE+) is built for X64, and the images are finally organized into a
> single flash device (FD).
> 
> The key PCD is
> 
>   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|TRUE
> 
> which you won't see in either the pure IA32 or the pure X64 OVMF
> platform -- as none of those have to switch from 32-bit mode to 64-bit
> (long) mode at the PEI/DXE boundary:
> 
> - OvmfPkgIa32.dsc:
>   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|FALSE
> 
> - OvmfPkgIa32X64.dsc:
>   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|TRUE
> 
> - OvmfPkgX64.dsc:
>   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode|FALSE
> 
> 
> Regarding the usefulness of the OvmfPkgIa32X64.dsc, that's presently the
> *only* OVMF platform that is suitable for production use. It is the only
> platform that:
> 
> (a) supports Secure Boot, *and*
> (b) includes ACPI S3 suspend/resume support, *and*
> (c) protects sensitive data related to (a) and (b), i.e. non-volatile
> UEFI variables, and the LockBox data structure, respectively, by
> inclusion of the SMM driver stack, and usage of SMRAM, *and*
> (d) supports 64-bit OS-es.

OK, at that point I totally agree only option #3 is workable.

> > So, am I missing something, or does this require a change in
> > BaseTools?
> 
> I believe you may have missed that "-a IA32 -a X64" stands for more than
> just "build this set of modules for each of IA32 and X64, separately".
> 
> AIUI, "-a IA32 -a X64" it means "build the Components.* sections
> (plural) as dictated by the '-a' flags, and then organize the full
> (multi-arch) set of modules into the flash device(s), described by
> FLASH_DEFINITION".

Nope, I did spot that actually :)
Although I was certainly confused the first 10 times I looked at those
.dscs.

And it could still be squashed into a common .dsc/.fdf if there was a
useful way of handling $(ARCH) when multiple -a have been specified to
the build command.

/
    Leif

Re: BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified

[Laszlo Ersek]

On 02/05/19 12:08, Leif Lindholm wrote:
> On Mon, Feb 04, 2019 at 08:02:55PM +0100, Laszlo Ersek wrote:

>> AIUI, "-a IA32 -a X64" it means "build the Components.* sections
>> (plural) as dictated by the '-a' flags, and then organize the full
>> (multi-arch) set of modules into the flash device(s), described by
>> FLASH_DEFINITION".
> 
> Nope, I did spot that actually :)
> Although I was certainly confused the first 10 times I looked at those
> .dscs.
> 
> And it could still be squashed into a common .dsc/.fdf if there was a
> useful way of handling $(ARCH) when multiple -a have been specified to
> the build command.

I agree.

Thanks
Laszlo

Re: BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified

[Gao, Liming]

Leif:
  I will check BaseTools implementation and share more information on how to handle ARCH when multiple ARCH are specified. 

Thanks
Liming
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Laszlo Ersek
> Sent: Tuesday, February 5, 2019 7:36 PM
> To: Leif Lindholm <leif.lindholm@linaro.org>
> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
> Subject: Re: [edk2] BaseTools: handling of $(ARCH) in .dsc/.fdf when multiple -a specified
> 
> On 02/05/19 12:08, Leif Lindholm wrote:
> > On Mon, Feb 04, 2019 at 08:02:55PM +0100, Laszlo Ersek wrote:
> 
> >> AIUI, "-a IA32 -a X64" it means "build the Components.* sections
> >> (plural) as dictated by the '-a' flags, and then organize the full
> >> (multi-arch) set of modules into the flash device(s), described by
> >> FLASH_DEFINITION".
> >
> > Nope, I did spot that actually :)
> > Although I was certainly confused the first 10 times I looked at those
> > .dscs.
> >
> > And it could still be squashed into a common .dsc/.fdf if there was a
> > useful way of handling $(ARCH) when multiple -a have been specified to
> > the build command.
> 
> I agree.
> 
> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel