[edk2-devel] Expansion of ARCH in .dsc files

[edk2-devel] Expansion of ARCH in .dsc files

[Leif Lindholm via groups.io]

So... I originally tripped over my misunderstanding of <subject> when writing
https://github.com/tianocore/edk2/pull/6262 to get around something I
wanted to test in UefiPayloadPkg.

Now Ajan has run into at least an adjacent problem when trying to rework
https://github.com/tianocore/edk2/pull/6545 based on feedback I among
others provided.

Now, the thing I had misunderstood before is that when build is called
with multiple -a options,
the ARCH variable will expand to a space-separated list of *all* the
specified architectures.
A lot of Arm/AArch64 modules assume this variable only ever contains a
single architecture.

The thing is, I don't think this is clearly documented in the .dsc
specification at all. And it is
frequently used as a shorthand in there in a form that would only work
if build was called with
a single -a.
Now, this also means that there are nearly no correct users of ARCH in
edk2 or edk2-platforms.
Because "the list of all architectures being built simultaneously"
simply isn't very useful information
build-time. There are two correct uses in edk2, in EmulatorPkg.dsc and
in UefiPayloadPkg.dsc,
using '!if "IA32" in "$ARCH"', which as far as I can tell is the only
possible way to use the variable
correctly.
Slightly amusingly, EmulatorPkg.dsc also contains an incorrect use.

I guess the behaviour is a direct result of how the build command is
architected.
I guess the one question I have is - is this the intended behaviour?

Ard: I think we need to update some of our .dsc files...

Ajan: one somewhat hacky workaround I can think of to separate the
output images is to hardcode
the output image name in arch-specific Defines sections, like so:

diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
index 0bd8d85cab03..f07f2b91764e 100644
--- a/ArmVirtPkg/ArmVirtQemu.dsc
+++ b/ArmVirtPkg/ArmVirtQemu.dsc
@@ -12,12 +12,17 @@
 # Defines Section - statements that will be processed to create a Makefile.
 #
 ################################################################################
+[Defines.ARM]
+  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-ARM
+
+[Defines.AARCH64]
+  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-AARCH64
+
 [Defines]
   PLATFORM_NAME                  = ArmVirtQemu
   PLATFORM_GUID                  = 37d7e986-f7e9-45c2-8067-e371421a626c
   PLATFORM_VERSION               = 0.1
   DSC_SPECIFICATION              = 0x00010005
-  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-$(ARCH)
   SUPPORTED_ARCHITECTURES        = AARCH64|ARM
   BUILD_TARGETS                  = DEBUG|RELEASE|NOOPT
   SKUID_IDENTIFIER               = DEFAULT


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120967): https://edk2.groups.io/g/devel/message/120967
Mute This Topic: https://groups.io/mt/110483263/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Expansion of ARCH in .dsc files

[Joey Vagedes via groups.io]

Leif,

For your awareness, Section 3.4 of the INF specification states that architectural modifiers for the [Defines] section tag are not permitted.

So that you can Ctrl+F and to find the exact language, here is a quote:

"architectural modifiers for the [Defines] section tag are not permitted"

https://tianocore-docs.github.io/edk2-InfSpecification/release-1.27/3_edk_ii_inf_file_format/34_[defines]_section.html#34-defines-section

Thanks,
Joey

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Leif Lindholm via groups.io
Sent: Tuesday, January 7, 2025 11:05 AM
To: devel@edk2.groups.io
Cc: Ajan Zhong <ajan.zhong@newfw.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Kinney, Michael D <michael.d.kinney@intel.com>
Subject: [EXTERNAL] [edk2-devel] Expansion of ARCH in .dsc files

So... I originally tripped over my misunderstanding of <subject> when writing
https://github.com/tianocore/edk2/pull/6262 to get around something I wanted to test in UefiPayloadPkg.

Now Ajan has run into at least an adjacent problem when trying to rework
https://github.com/tianocore/edk2/pull/6545 based on feedback I among others provided.

Now, the thing I had misunderstood before is that when build is called with multiple -a options, the ARCH variable will expand to a space-separated list of *all* the specified architectures.
A lot of Arm/AArch64 modules assume this variable only ever contains a single architecture.

The thing is, I don't think this is clearly documented in the .dsc specification at all. And it is frequently used as a shorthand in there in a form that would only work if build was called with a single -a.
Now, this also means that there are nearly no correct users of ARCH in
edk2 or edk2-platforms.
Because "the list of all architectures being built simultaneously"
simply isn't very useful information
build-time. There are two correct uses in edk2, in EmulatorPkg.dsc and in UefiPayloadPkg.dsc, using '!if "IA32" in "$ARCH"', which as far as I can tell is the only possible way to use the variable correctly.
Slightly amusingly, EmulatorPkg.dsc also contains an incorrect use.

I guess the behaviour is a direct result of how the build command is architected.
I guess the one question I have is - is this the intended behaviour?

Ard: I think we need to update some of our .dsc files...

Ajan: one somewhat hacky workaround I can think of to separate the output images is to hardcode the output image name in arch-specific Defines sections, like so:

diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc index 0bd8d85cab03..f07f2b91764e 100644
--- a/ArmVirtPkg/ArmVirtQemu.dsc
+++ b/ArmVirtPkg/ArmVirtQemu.dsc
@@ -12,12 +12,17 @@
 # Defines Section - statements that will be processed to create a Makefile.
 #
 ################################################################################
+[Defines.ARM]
+  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-ARM
+
+[Defines.AARCH64]
+  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-AARCH64
+
 [Defines]
   PLATFORM_NAME                  = ArmVirtQemu
   PLATFORM_GUID                  = 37d7e986-f7e9-45c2-8067-e371421a626c
   PLATFORM_VERSION               = 0.1
   DSC_SPECIFICATION              = 0x00010005
-  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-$(ARCH)
   SUPPORTED_ARCHITECTURES        = AARCH64|ARM
   BUILD_TARGETS                  = DEBUG|RELEASE|NOOPT
   SKUID_IDENTIFIER               = DEFAULT







-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120969): https://edk2.groups.io/g/devel/message/120969
Mute This Topic: https://groups.io/mt/110483263/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Expansion of ARCH in .dsc files

[Leif Lindholm via groups.io]

Hi Joey,

I appreciate the pointer.
However, that's the .inf spec.

The same language is not in the corresponding section of
https://tianocore-docs.github.io/edk2-DscSpecification/release-1.28/3_edk_ii_dsc_file_format/35_[defines]_section.html#35-defines-section

Regards,

Leif

On Tue, 7 Jan 2025 at 19:43, Joey Vagedes <joeyvagedes@microsoft.com> wrote:
>
> Leif,
>
> For your awareness, Section 3.4 of the INF specification states that architectural modifiers for the [Defines] section tag are not permitted.
>
> So that you can Ctrl+F and to find the exact language, here is a quote:
>
> "architectural modifiers for the [Defines] section tag are not permitted"
>
> https://tianocore-docs.github.io/edk2-InfSpecification/release-1.27/3_edk_ii_inf_file_format/34_[defines]_section.html#34-defines-section
>
> Thanks,
> Joey
>
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Leif Lindholm via groups.io
> Sent: Tuesday, January 7, 2025 11:05 AM
> To: devel@edk2.groups.io
> Cc: Ajan Zhong <ajan.zhong@newfw.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: [EXTERNAL] [edk2-devel] Expansion of ARCH in .dsc files
>
> So... I originally tripped over my misunderstanding of <subject> when writing
> https://github.com/tianocore/edk2/pull/6262 to get around something I wanted to test in UefiPayloadPkg.
>
> Now Ajan has run into at least an adjacent problem when trying to rework
> https://github.com/tianocore/edk2/pull/6545 based on feedback I among others provided.
>
> Now, the thing I had misunderstood before is that when build is called with multiple -a options, the ARCH variable will expand to a space-separated list of *all* the specified architectures.
> A lot of Arm/AArch64 modules assume this variable only ever contains a single architecture.
>
> The thing is, I don't think this is clearly documented in the .dsc specification at all. And it is frequently used as a shorthand in there in a form that would only work if build was called with a single -a.
> Now, this also means that there are nearly no correct users of ARCH in
> edk2 or edk2-platforms.
> Because "the list of all architectures being built simultaneously"
> simply isn't very useful information
> build-time. There are two correct uses in edk2, in EmulatorPkg.dsc and in UefiPayloadPkg.dsc, using '!if "IA32" in "$ARCH"', which as far as I can tell is the only possible way to use the variable correctly.
> Slightly amusingly, EmulatorPkg.dsc also contains an incorrect use.
>
> I guess the behaviour is a direct result of how the build command is architected.
> I guess the one question I have is - is this the intended behaviour?
>
> Ard: I think we need to update some of our .dsc files...
>
> Ajan: one somewhat hacky workaround I can think of to separate the output images is to hardcode the output image name in arch-specific Defines sections, like so:
>
> diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc index 0bd8d85cab03..f07f2b91764e 100644
> --- a/ArmVirtPkg/ArmVirtQemu.dsc
> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> @@ -12,12 +12,17 @@
>  # Defines Section - statements that will be processed to create a Makefile.
>  #
>  ################################################################################
> +[Defines.ARM]
> +  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-ARM
> +
> +[Defines.AARCH64]
> +  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-AARCH64
> +
>  [Defines]
>    PLATFORM_NAME                  = ArmVirtQemu
>    PLATFORM_GUID                  = 37d7e986-f7e9-45c2-8067-e371421a626c
>    PLATFORM_VERSION               = 0.1
>    DSC_SPECIFICATION              = 0x00010005
> -  OUTPUT_DIRECTORY               = Build/ArmVirtQemu-$(ARCH)
>    SUPPORTED_ARCHITECTURES        = AARCH64|ARM
>    BUILD_TARGETS                  = DEBUG|RELEASE|NOOPT
>    SKUID_IDENTIFIER               = DEFAULT
>
>
> 
>
>


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120970): https://edk2.groups.io/g/devel/message/120970
Mute This Topic: https://groups.io/mt/110483263/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Expansion of ARCH in .dsc files

[Gerd Hoffmann via groups.io]

  Hi,

> Because "the list of all architectures being built simultaneously"
> simply isn't very useful information build-time.

I think this is not intended for building all archs in simultaneously
(even though that often works too), but for firmware images using code
from multiple architectures.  That is (or used to be?) common in the x86
world, where firmware can run the PEI phase in 32-bit mode (aka IA32)
and the DXE phase in 64-bit mode (aka X64).

See OvmfPkg/OvmfPkgIa32X64.dsc

take care,
  Gerd



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120972): https://edk2.groups.io/g/devel/message/120972
Mute This Topic: https://groups.io/mt/110483263/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] Expansion of ARCH in .dsc files

[Leif Lindholm via groups.io]

On Wed, 8 Jan 2025 at 15:01, Gerd Hoffmann <kraxel@redhat.com> wrote:
> > Because "the list of all architectures being built simultaneously"
> > simply isn't very useful information build-time.
>
> I think this is not intended for building all archs in simultaneously
> (even though that often works too), but for firmware images using code
> from multiple architectures.  That is (or used to be?) common in the x86
> world, where firmware can run the PEI phase in 32-bit mode (aka IA32)
> and the DXE phase in 64-bit mode (aka X64).
>
> See OvmfPkg/OvmfPkgIa32X64.dsc

Yes, indeed.
It feels like a bit of a hack introduced specifically for that
specific case (but not documented as such).

So, Ajan - I guess that means arguably the CI job should be reworked
to call the different arch builds independently. Then the problem
would go away even without the Defines.$ARCH change.

/
    Leif


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120973): https://edk2.groups.io/g/devel/message/120973
Mute This Topic: https://groups.io/mt/110483263/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-