Re: [RFC] migration of OpenPlatformPkg to tianocore

[Leif Lindholm <leif.lindholm@linaro.org>]

Hi all,

OK, so I have now updated both edk2-non-osi and edk2-platforms
(devel-OpenPlatformPkg branches) with package renaming, and updating
.dsc/.fdf files to . 

It appears the problems I'm facing are mainly caused by the GenFds
stage not seeing a view consistent with the actual compilation stage.

To demonstrate, I build the Juno platform:
$ . edksetup.sh
$ make -C BaseTools
$ PACKAGES_PATH=/work/git/edk2-platforms/Platform/ARM \
   GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -n 9 -a AARCH64 \
   -t GCC5 -p JunoPkg/ArmJuno.dsc -b RELEASE

This build fails with:
---
Fd File Name:BL33_AP_UEFI

Generate Region at Offset 0x0
   Region Size = 0xF8000
      Region Name = FV

Generating FVMAIN_COMPACT FV
####
Generating FVMAIN FV


GenFds.py...
 : error F003: Output file for RAW section could not be found for
 JunoPkg/AcpiTables/AcpiTables.inf



build.py...
 : error 7000: Failed to execute command
   GenFds -f /work/git/edk2-platforms/Platform/ARM/JunoPkg/ArmJuno.fdf
   --conf=/work/git/edk2/Conf -o
   /work/git/edk2/Build/ArmJuno/RELEASE_GCC5 -t GCC5 -b RELEASE -p
   /work/git/edk2-platforms/Platform/ARM/JunoPkg/ArmJuno.dsc -a
   AARCH64 -D "EFI_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
   "EDK_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
   "TOOL_CHAIN_TAG=GCC5" -D "TOOLCHAIN=GCC5" -D "TARGET=RELEASE" -D
   "FAMILY=GCC" -D "WORKSPACE=/work/git/edk2" -D
   "EDK_TOOLS_PATH=/work/git/edk2/BaseTools" -D "ARCH=AARCH64" -D
   "ECP_SOURCE=/work/git/edk2/EdkCompatibilityPkg" [/work/git/edk2]

- Failed -
---
, but if I add the symlink
ln -s edk2-platforms/Platform/ARM/JunoPkg/ Build/ArmJuno/RELEASE_GCC5/AARCH64/
it completes successfully on the next pass.

Same when building VExpressPkg/ArmVExpress-FVP-AArch64.dsc
ln -s edk2-platforms/Platform/ARM/VExpressPkg \
 Build/ArmVExpress-FVP-AArch64/RELEASE_GCC5/AARCH64/
resolves the build error.


Similarly, when I build the Overdrive/Overdrive.dsc, using:
$ PACKAGES_PATH=/work/git/edk2-platforms/Platform/AMD:/work/git/edk2-platforms/Silicon/AMD:/work/git/edk2-non-osi/Silicon/AMD/Styx:/work/git/edk2-non-osi/Platform/AMD \
 GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -n 9 -a AARCH64 -t GCC5
 -p OverdriveBoard/OverdriveBoard.dsc -b RELEASE
this fails with
---
Fd File Name:STYX_ROM

Generate Region at Offset 0x0
   Region Size = 0x200000
      Region File Name =
      /work/git/edk2/../edk2-non-osi/Platform/AMD/OverdriveBoard/PreUefiFirmware.bin

Generate Region at Offset 0x200000
   Region Size = 0x260000
      Region Name = FV

Generating STYX_EFI FV
################Return Value = 2
GenFw: ERROR 0001: Error opening file

  /work/git/edk2/Build/Overdrive/RELEASE_GCC5/AARCH64/StyxPkg/Drivers/PlatInitPei/PlatInitPei/OUTPUT/PlatInitPei.efi



GenFds.py...
### ['GenFw', '-t', '-o',
### '/work/git/edk2/Build/Overdrive/RELEASE_GCC5/FV/Ffs/769694a4-2572-4f29-a5bb-33d7df7be001PlatInitPei/769694a4-2572-4f29-a5bb-33d7df7be001Te.raw',
### '/work/git/edk2/Build/Overdrive/RELEASE_GCC5/AARCH64/StyxPkg/Drivers/PlatInitPei/PlatInitPei/OUTPUT/PlatInitPei.efi']
 : error 7000: Failed to generate firmware image



build.py...
 : error 7000: Failed to execute command
   GenFds -f
   /work/git/edk2-platforms/Platform/AMD/OverdriveBoard/OverdriveBoard.fdf
   --conf=/work/git/edk2/Conf -o
   /work/git/edk2/Build/Overdrive/RELEASE_GCC5 -t GCC5 -b RELEASE -p
   /work/git/edk2-platforms/Platform/AMD/OverdriveBoard/OverdriveBoard.dsc
   -a AARCH64 -D "EFI_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
   "EDK_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
   "TOOL_CHAIN_TAG=GCC5" -D "TOOLCHAIN=GCC5" -D "TARGET=RELEASE" -D
   "FAMILY=GCC" -D "WORKSPACE=/work/git/edk2" -D
   "FIRMWARE_VER=b941c34" -D "EDK_TOOLS_PATH=/work/git/edk2/BaseTools"
   -D "ARCH=AARCH64" -D
   "ECP_SOURCE=/work/git/edk2/EdkCompatibilityPkg" [/work/git/edk2]

- Failed -
---

Again, resolved by
ln -s edk2-platforms/Silicon/AMD/StyxPkg/ Build/Overdrive/RELEASE_GCC5/AARCH64/

Am I doing something wrong here? Are there any magic runes that could
help out?

/
    Leif

On Fri, May 05, 2017 at 01:45:51PM +0000, Yao, Jiewen wrote:
> HI Leif
> It is great that you are adding more platform to edk2-platforms.
> 
> We (Intel) also have plan to add more boards there. In general, we are very close on having silicon and platform folder.
> 
> I have a quick look. One minor suggestions here:
> Take Arm folder as example. (I assume Juno is one package, and VExpress is the other package.)
> Can we name Juno to be JunoPkg, and VExpress to be VExpressPkg ?
> We do not add "Pkg" to a folder. And we usually add "Pkg" suffix to a package.
> 
> In general, I think it is a very good start.
> I may review the content in more detail and provide more feedback.
> 
> 
> Thank you
> Yao Jiewen
> 
> 
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Leif Lindholm
> Sent: Thursday, May 4, 2017 6:56 AM
> To: edk2-devel@lists.01.org
> Cc: Ryan Harkin <ryan.harkin@linaro.org>; Ard Biesheuvel <ard.biesheuvel@linaro.org>; Chenhui Sun <chenhui.sun@linaro.org>; Andrew Fish <afish@apple.com>; Alan Ott <alan@softiron.co.uk>; Richardson, Brian <brian.richardson@intel.com>; Duran, Leo <leo.duran@amd.com>; haojian.zhuang@linaro.org; Linaro UEFI <linaro-uefi@lists.linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Heyi Guo <heyi.guo@linaro.org>
> Subject: [edk2] [RFC] migration of OpenPlatformPkg to tianocore
> 
> Hi all,
> 
> As some of you may be aware, I have been working around the lack of
> a clear upstreaming strategy for platform support by keeping such code
> in a dedicated repository I set up at Linaro for that purpose:
> https://git.linaro.org/uefi/OpenPlatformPkg.git
> 
> During discussions at the last Seattle plugfest we finally agreed on
> the (theoretical) details of how to use the edk2-platforms repository.
> After that I promised to migrate OpenPlatformPkg across to the
> edk2-platforms and edk2-non-osi structure, with the explicit end goal
> from my side that this should become the master branch for each.
> 
> And now, before the release of HURD 1.0, I have.
> 
> Current limitations (that I can remember):
> - A few references to OpenPlatformPkg remain, in ways that do not
>   appear to break any of the platform builds. Most likely this affects
>   dead code, but in case it's been accidentally orphaned, I thought it
>   best to
> - I have simply nuked all references to Ebl (used in _addition_ to the
>   UEFI shell, which was never the intent) and the efi-toolkit
>   ramdisk driver.
> - The Marvell Yukon driver that I sent out for review last week has
>   not migrated anywhere, and so has been temporarily disabled
>   Mike suggested I should
> - USB support on the LeMaker Cello board depends on the patch
>   "OptionRomPkg: add firmware loader driver for Renesas PD72020x"
>   sent out by Ard on 18th of April.
> - I have dropped some of the binary-only modules from OpenPlatformPkg,
>   and contacted the platform owners with requests for modifications.
> - The git history is quite messy and will be cleaned up, but I wanted
>   to keep the transition quite visible in the RFC.
> - I haven't filled anything into the Maintainers.txt files - I am in
>   favour of moving to a fully machine-readable format with wildcards
>   as Laszlo has proposed in the past, and think this would be an
>   excellent point to have that discussion (which can be had separately
>   for edk2-platforms and edk2-non-osi from edk2).
> - Few of the platforms complete the FV generation stage, and I've
>   inserted a couple of silly hacks to get them to get as far as they
>   do. I think that either I am missing some points of how
>   PACKAGES_PATH is intended to work, or I'm simply hitting corner
>   cases no one has come across before. I could really use some help
>   debugging these issues. (examples below).
> 
> The below depends on the 3-part series I sent out today for importing
> DwEmmcDxe and EfiTimeBaseLib from OpenPlatformPkg. But apart from
> that, I have uploaded branches called devel-OpenPlatformPkg to:
> 
> https://github.com/tianocore/edk2-platforms/tree/devel-OpenPlatformPkg
> https://github.com/tianocore/edk2-non-osi/tree/devel-OpenPlatformPkg
> 
> These branches _will_ be rebased occasionally until they get to a
> point where they can move out of devel- stage (and hopefully onto
> master).
> 
> 
> Build issue description
> =======================
> So, one of the hopefully easier ones is what I'm seeing when trying to
> build the Juno platform:
> 
> $ PACKAGES_PATH="/work/maint/edk2-platforms:/work/maint/edk2-non-osi" GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -a AARCH64 -t GCC5 -p Platform/ARM/Juno/ArmJuno.dsc -b RELEASE -n 9
> 
> results in:
> 
> <<<
> GenFds.py...
>  : error F003: Output file for RAW section could not be found for
>  Platform/ARM/Juno/AcpiTables/AcpiTables.inf
> 
> ###
> 
> 
> build.py...
>  : error 7000: Failed to execute command
>          GenFds -f /work/maint/edk2-platforms/Platform/ARM/Juno/ArmJuno.fdf --conf=/work/maint/edk2/Conf -o /work/maint/edk2/Build/ArmJuno/RELEASE_GCC5 -t GCC5 -b RELEASE -p /work/maint/edk2-platforms/Platform/ARM/Juno/ArmJuno.dsc -a AARCH64 -D "EFI_SOURCE=/work/maint/edk2/EdkCompatibilityPkg" -D "EDK_SOURCE=/work/maint/edk2/EdkCompatibilityPkg" -D "TOOL_CHAIN_TAG=GCC5" -D "TOOLCHAIN=GCC5" -D "TARGET=RELEASE" -D "FAMILY=GCC" -D "WORKSPACE=/work/maint/edk2" -D "EDK_TOOLS_PATH=/work/maint/edk2/BaseTools" -D "ARCH=AARCH64" -D "ECP_SOURCE=/work/maint/edk2/EdkCompatibilityPkg"
>  [/work/maint/edk2]
> 
> - Failed -
> >>>
> 
> And when I copy and paste the above command manually, I get:
> 
> <<<
> GenFds.py...
> /work/maint/edk2-platforms/Platform/ARM/Juno/ArmJuno.dsc(34): error
> 000E: File/directory not found in workspace
>         /work/maint/edk2-platforms/Platform/ARM/Juno/Platform/ARM/VExpress/ArmVExpress.dsc.inc
> /work/maint/edk2/Platform/ARM/VExpress/ArmVExpress.dsc.inc
> >>>
> 
> So, to an uniformed observer, it seems the portion
> !include Platform/ARM/VExpress/ArmVExpress.dsc.inc
> from ArmJuno.dsc
> gets expanded to "directory ArmJuno.dsc is in" + "Platform/ARM/VExpress/ArmVExpress.dsc.inc"
> whereas I was hoping for it to try to find a match for
> "Platform/ARM/VExpress/ArmVExpress.dsc.inc" along PACKAGES_PATH (and
> find one in edk2-platforms).
> 
> I also have the impression that something similar is happening in
> ArmJuno.fdf for the line
>   INF RuleOverride=ACPITABLE Platform/ARM/Juno/AcpiTables/AcpiTables.inf
> generating the error message from the original build command.
> 
> But I'm not quite sure how to debug these issues (short of fully
> figuring out the innards of MultipleWorkspace.py, MetaFileParser.py
> and a few others).
> Any ideas of where to look, or even what is going on?
> Would these cases be expected to work?
> 
> /
>     Leif
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
> https://lists.01.org/mailman/listinfo/edk2-devel