Re: [RFC] migration of OpenPlatformPkg to tianocore

[Leif Lindholm <leif.lindholm@linaro.org>]

Hi Mike,

Yes, this way I am able to build all of my packages! Many thanks!

This also works with the simplified PACKAGES_PATH usage model I had
initially been trying for (just point to edk2-platforms and
edk2-non-osi top level directories).

I have pushed updated and slightly cleaned up versions to
devel-OpenPlatformPkg branches on edk2-platforms and edk2-non-osi.

I'm working on updated helper scripts in my uefi-tools repository and
some documentation for this, and will post a new thread when I have
this ready.

/
    Leif

On Tue, Jun 13, 2017 at 11:13:08PM +0000, Kinney, Michael D wrote:
> Hi Leif,
> 
> I pulled the latest versions of the repos and I was able to complete
> a build with no errors and without using symbolic links.
> 
> The change I made is to set WORKSPACE to the directory immediately
> above the repos and set PACKAGES_PATH to directories that contain the
> packages required for the platform to build.
> 
> This is the same technique used to build QuarkPlatformPkg that uses
> content from both the edk2 and the edk2-non-osi repos.  See the
> Readme.md in the QuarkPlatformPkg with instructions for setting up
> the build env for Linux and Windows for an example.
> 
> https://github.com/tianocore/edk2/tree/master/QuarkPlatformPkg
> 
> I wrote a small shell script that I put in the directory above the
> repos to setup the build environment for the JunoPkg platform.
> 
> export GCC5_AARCH64_PREFIX=aarch64-linux-gnu-
> 
> export WORKSPACE=$PWD
> 
> export PACKAGES_PATH=\
> $WORKSPACE/edk2:\
> $WORKSPACE/edk2-platforms/Platform/ARM:\
> $WORKSPACE/edk2-non-osi
> 
> cd edk2
> make -C BaseTools
> . edksetup.sh BaseTools
> 
> build -a AARCH64 -t GCC5 -p JunoPkg/ArmJuno.dsc
> 
> === end of build log ===
> Generate Region at Offset 0x0
>    Region Size = 0xF8000
>    Region Name = FV
> 
> Generating FVMAIN_COMPACT FV
> ####
> Generating FVMAIN FV
> ####################################
> ########################################
> ########################################
> ########################################
> ################################
> Generate Region at Offset 0x0
>    Region Size = 0xF8000
>    Region Name = FV
> 
> GUID cross reference file can be found at /home/mdkinney/GitHub/tianocore/Build/ArmJuno/DEBUG_GCC5/FV/Guid.xref
> 
> FV Space Information
> FVMAIN_COMPACT [94%Full] 1015808 total, 956200 used, 59608 free
> FVMAIN [99%Full] 8237696 total, 8237688 used, 8 free
> 
> - Done -
> Build end time: 18:49:57, Jun.13 2017
> Build total time: 00:04:22
> ========================
> 
> 
> If you want to support building any of the platforms, then
> PACKAGES_PATH can be set as follows:
> 
> export PACKAGES_PATH=\
> $WORKSPACE/edk2:\
> $WORKSPACE/edk2-platforms/Silicon/Hisilicon:\
> $WORKSPACE/edk2-platforms/Silicon/AMD:\
> $WORKSPACE/edk2-platforms/Platform/Hisilicon:\
> $WORKSPACE/edk2-platforms/Platform/Marvell:\
> $WORKSPACE/edk2-platforms/Platform/ARM:\
> $WORKSPACE/edk2-non-osi/Silicon/Intel:\
> $WORKSPACE/edk2-non-osi/Silicon/AMD/Styx:
> 
> Best regards,
> 
> Mike
> 
> -----Original Message-----
> From: Leif Lindholm [mailto:leif.lindholm@linaro.org] 
> Sent: Wednesday, June 7, 2017 7:58 AM
> To: Yao, Jiewen <jiewen.yao@intel.com>
> Cc: edk2-devel@lists.01.org; Ryan Harkin <ryan.harkin@linaro.org>; Ard Biesheuvel <ard.biesheuvel@linaro.org>; Chenhui Sun <chenhui.sun@linaro.org>; Andrew Fish <afish@apple.com>; Alan Ott <alan@softiron.co.uk>; Richardson, Brian <brian.richardson@intel.com>; Duran, Leo <leo.duran@amd.com>; haojian.zhuang@linaro.org; Linaro UEFI <linaro-uefi@lists.linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Heyi Guo <heyi.guo@linaro.org>
> Subject: Re: [edk2] [RFC] migration of OpenPlatformPkg to tianocore
> 
> Hi all,
> 
> OK, so I have now updated both edk2-non-osi and edk2-platforms
> (devel-OpenPlatformPkg branches) with package renaming, and updating
> .dsc/.fdf files to . 
> 
> It appears the problems I'm facing are mainly caused by the GenFds
> stage not seeing a view consistent with the actual compilation stage.
> 
> To demonstrate, I build the Juno platform:
> $ . edksetup.sh
> $ make -C BaseTools
> $ PACKAGES_PATH=/work/git/edk2-platforms/Platform/ARM \
>    GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -n 9 -a AARCH64 \
>    -t GCC5 -p JunoPkg/ArmJuno.dsc -b RELEASE
> 
> This build fails with:
> ---
> Fd File Name:BL33_AP_UEFI
> 
> Generate Region at Offset 0x0
>    Region Size = 0xF8000
>       Region Name = FV
> 
> Generating FVMAIN_COMPACT FV
> ####
> Generating FVMAIN FV
> 
> 
> GenFds.py...
>  : error F003: Output file for RAW section could not be found for
>  JunoPkg/AcpiTables/AcpiTables.inf
> 
> 
> 
> build.py...
>  : error 7000: Failed to execute command
>    GenFds -f /work/git/edk2-platforms/Platform/ARM/JunoPkg/ArmJuno.fdf
>    --conf=/work/git/edk2/Conf -o
>    /work/git/edk2/Build/ArmJuno/RELEASE_GCC5 -t GCC5 -b RELEASE -p
>    /work/git/edk2-platforms/Platform/ARM/JunoPkg/ArmJuno.dsc -a
>    AARCH64 -D "EFI_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
>    "EDK_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
>    "TOOL_CHAIN_TAG=GCC5" -D "TOOLCHAIN=GCC5" -D "TARGET=RELEASE" -D
>    "FAMILY=GCC" -D "WORKSPACE=/work/git/edk2" -D
>    "EDK_TOOLS_PATH=/work/git/edk2/BaseTools" -D "ARCH=AARCH64" -D
>    "ECP_SOURCE=/work/git/edk2/EdkCompatibilityPkg" [/work/git/edk2]
> 
> - Failed -
> ---
> , but if I add the symlink
> ln -s edk2-platforms/Platform/ARM/JunoPkg/ Build/ArmJuno/RELEASE_GCC5/AARCH64/
> it completes successfully on the next pass.
> 
> Same when building VExpressPkg/ArmVExpress-FVP-AArch64.dsc
> ln -s edk2-platforms/Platform/ARM/VExpressPkg \
>  Build/ArmVExpress-FVP-AArch64/RELEASE_GCC5/AARCH64/
> resolves the build error.
> 
> 
> Similarly, when I build the Overdrive/Overdrive.dsc, using:
> $ PACKAGES_PATH=/work/git/edk2-platforms/Platform/AMD:/work/git/edk2-platforms/Silicon/AMD:/work/git/edk2-non-osi/Silicon/AMD/Styx:/work/git/edk2-non-osi/Platform/AMD \
>  GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -n 9 -a AARCH64 -t GCC5
>  -p OverdriveBoard/OverdriveBoard.dsc -b RELEASE
> this fails with
> ---
> Fd File Name:STYX_ROM
> 
> Generate Region at Offset 0x0
>    Region Size = 0x200000
>       Region File Name =
>       /work/git/edk2/../edk2-non-osi/Platform/AMD/OverdriveBoard/PreUefiFirmware.bin
> 
> Generate Region at Offset 0x200000
>    Region Size = 0x260000
>       Region Name = FV
> 
> Generating STYX_EFI FV
> ################Return Value = 2
> GenFw: ERROR 0001: Error opening file
> 
>   /work/git/edk2/Build/Overdrive/RELEASE_GCC5/AARCH64/StyxPkg/Drivers/PlatInitPei/PlatInitPei/OUTPUT/PlatInitPei.efi
> 
> 
> 
> GenFds.py...
> ### ['GenFw', '-t', '-o',
> ### '/work/git/edk2/Build/Overdrive/RELEASE_GCC5/FV/Ffs/769694a4-2572-4f29-a5bb-33d7df7be001PlatInitPei/769694a4-2572-4f29-a5bb-33d7df7be001Te.raw',
> ### '/work/git/edk2/Build/Overdrive/RELEASE_GCC5/AARCH64/StyxPkg/Drivers/PlatInitPei/PlatInitPei/OUTPUT/PlatInitPei.efi']
>  : error 7000: Failed to generate firmware image
> 
> 
> 
> build.py...
>  : error 7000: Failed to execute command
>    GenFds -f
>    /work/git/edk2-platforms/Platform/AMD/OverdriveBoard/OverdriveBoard.fdf
>    --conf=/work/git/edk2/Conf -o
>    /work/git/edk2/Build/Overdrive/RELEASE_GCC5 -t GCC5 -b RELEASE -p
>    /work/git/edk2-platforms/Platform/AMD/OverdriveBoard/OverdriveBoard.dsc
>    -a AARCH64 -D "EFI_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
>    "EDK_SOURCE=/work/git/edk2/EdkCompatibilityPkg" -D
>    "TOOL_CHAIN_TAG=GCC5" -D "TOOLCHAIN=GCC5" -D "TARGET=RELEASE" -D
>    "FAMILY=GCC" -D "WORKSPACE=/work/git/edk2" -D
>    "FIRMWARE_VER=b941c34" -D "EDK_TOOLS_PATH=/work/git/edk2/BaseTools"
>    -D "ARCH=AARCH64" -D
>    "ECP_SOURCE=/work/git/edk2/EdkCompatibilityPkg" [/work/git/edk2]
> 
> - Failed -
> ---
> 
> Again, resolved by
> ln -s edk2-platforms/Silicon/AMD/StyxPkg/ Build/Overdrive/RELEASE_GCC5/AARCH64/
> 
> Am I doing something wrong here? Are there any magic runes that could
> help out?
> 
> /
>     Leif
> 
> On Fri, May 05, 2017 at 01:45:51PM +0000, Yao, Jiewen wrote:
> > HI Leif
> > It is great that you are adding more platform to edk2-platforms.
> > 
> > We (Intel) also have plan to add more boards there. In general, we are very close on having silicon and platform folder.
> > 
> > I have a quick look. One minor suggestions here:
> > Take Arm folder as example. (I assume Juno is one package, and VExpress is the other package.)
> > Can we name Juno to be JunoPkg, and VExpress to be VExpressPkg ?
> > We do not add "Pkg" to a folder. And we usually add "Pkg" suffix to a package.
> > 
> > In general, I think it is a very good start.
> > I may review the content in more detail and provide more feedback.
> > 
> > 
> > Thank you
> > Yao Jiewen
> > 
> > 
> > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Leif Lindholm
> > Sent: Thursday, May 4, 2017 6:56 AM
> > To: edk2-devel@lists.01.org
> > Cc: Ryan Harkin <ryan.harkin@linaro.org>; Ard Biesheuvel <ard.biesheuvel@linaro.org>; Chenhui Sun <chenhui.sun@linaro.org>; Andrew Fish <afish@apple.com>; Alan Ott <alan@softiron.co.uk>; Richardson, Brian <brian.richardson@intel.com>; Duran, Leo <leo.duran@amd.com>; haojian.zhuang@linaro.org; Linaro UEFI <linaro-uefi@lists.linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Heyi Guo <heyi.guo@linaro.org>
> > Subject: [edk2] [RFC] migration of OpenPlatformPkg to tianocore
> > 
> > Hi all,
> > 
> > As some of you may be aware, I have been working around the lack of
> > a clear upstreaming strategy for platform support by keeping such code
> > in a dedicated repository I set up at Linaro for that purpose:
> > https://git.linaro.org/uefi/OpenPlatformPkg.git
> > 
> > During discussions at the last Seattle plugfest we finally agreed on
> > the (theoretical) details of how to use the edk2-platforms repository.
> > After that I promised to migrate OpenPlatformPkg across to the
> > edk2-platforms and edk2-non-osi structure, with the explicit end goal
> > from my side that this should become the master branch for each.
> > 
> > And now, before the release of HURD 1.0, I have.
> > 
> > Current limitations (that I can remember):
> > - A few references to OpenPlatformPkg remain, in ways that do not
> >   appear to break any of the platform builds. Most likely this affects
> >   dead code, but in case it's been accidentally orphaned, I thought it
> >   best to
> > - I have simply nuked all references to Ebl (used in _addition_ to the
> >   UEFI shell, which was never the intent) and the efi-toolkit
> >   ramdisk driver.
> > - The Marvell Yukon driver that I sent out for review last week has
> >   not migrated anywhere, and so has been temporarily disabled
> >   Mike suggested I should
> > - USB support on the LeMaker Cello board depends on the patch
> >   "OptionRomPkg: add firmware loader driver for Renesas PD72020x"
> >   sent out by Ard on 18th of April.
> > - I have dropped some of the binary-only modules from OpenPlatformPkg,
> >   and contacted the platform owners with requests for modifications.
> > - The git history is quite messy and will be cleaned up, but I wanted
> >   to keep the transition quite visible in the RFC.
> > - I haven't filled anything into the Maintainers.txt files - I am in
> >   favour of moving to a fully machine-readable format with wildcards
> >   as Laszlo has proposed in the past, and think this would be an
> >   excellent point to have that discussion (which can be had separately
> >   for edk2-platforms and edk2-non-osi from edk2).
> > - Few of the platforms complete the FV generation stage, and I've
> >   inserted a couple of silly hacks to get them to get as far as they
> >   do. I think that either I am missing some points of how
> >   PACKAGES_PATH is intended to work, or I'm simply hitting corner
> >   cases no one has come across before. I could really use some help
> >   debugging these issues. (examples below).
> > 
> > The below depends on the 3-part series I sent out today for importing
> > DwEmmcDxe and EfiTimeBaseLib from OpenPlatformPkg. But apart from
> > that, I have uploaded branches called devel-OpenPlatformPkg to:
> > 
> > https://github.com/tianocore/edk2-platforms/tree/devel-OpenPlatformPkg
> > https://github.com/tianocore/edk2-non-osi/tree/devel-OpenPlatformPkg
> > 
> > These branches _will_ be rebased occasionally until they get to a
> > point where they can move out of devel- stage (and hopefully onto
> > master).
> > 
> > 
> > Build issue description
> > =======================
> > So, one of the hopefully easier ones is what I'm seeing when trying to
> > build the Juno platform:
> > 
> > $ PACKAGES_PATH="/work/maint/edk2-platforms:/work/maint/edk2-non-osi" GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -a AARCH64 -t GCC5 -p Platform/ARM/Juno/ArmJuno.dsc -b RELEASE -n 9
> > 
> > results in:
> > 
> > <<<
> > GenFds.py...
> >  : error F003: Output file for RAW section could not be found for
> >  Platform/ARM/Juno/AcpiTables/AcpiTables.inf
> > 
> > ###
> > 
> > 
> > build.py...
> >  : error 7000: Failed to execute command
> >          GenFds -f /work/maint/edk2-platforms/Platform/ARM/Juno/ArmJuno.fdf --conf=/work/maint/edk2/Conf -o /work/maint/edk2/Build/ArmJuno/RELEASE_GCC5 -t GCC5 -b RELEASE -p /work/maint/edk2-platforms/Platform/ARM/Juno/ArmJuno.dsc -a AARCH64 -D "EFI_SOURCE=/work/maint/edk2/EdkCompatibilityPkg" -D "EDK_SOURCE=/work/maint/edk2/EdkCompatibilityPkg" -D "TOOL_CHAIN_TAG=GCC5" -D "TOOLCHAIN=GCC5" -D "TARGET=RELEASE" -D "FAMILY=GCC" -D "WORKSPACE=/work/maint/edk2" -D "EDK_TOOLS_PATH=/work/maint/edk2/BaseTools" -D "ARCH=AARCH64" -D "ECP_SOURCE=/work/maint/edk2/EdkCompatibilityPkg"
> >  [/work/maint/edk2]
> > 
> > - Failed -
> > >>>
> > 
> > And when I copy and paste the above command manually, I get:
> > 
> > <<<
> > GenFds.py...
> > /work/maint/edk2-platforms/Platform/ARM/Juno/ArmJuno.dsc(34): error
> > 000E: File/directory not found in workspace
> >         /work/maint/edk2-platforms/Platform/ARM/Juno/Platform/ARM/VExpress/ArmVExpress.dsc.inc
> > /work/maint/edk2/Platform/ARM/VExpress/ArmVExpress.dsc.inc
> > >>>
> > 
> > So, to an uniformed observer, it seems the portion
> > !include Platform/ARM/VExpress/ArmVExpress.dsc.inc
> > from ArmJuno.dsc
> > gets expanded to "directory ArmJuno.dsc is in" + "Platform/ARM/VExpress/ArmVExpress.dsc.inc"
> > whereas I was hoping for it to try to find a match for
> > "Platform/ARM/VExpress/ArmVExpress.dsc.inc" along PACKAGES_PATH (and
> > find one in edk2-platforms).
> > 
> > I also have the impression that something similar is happening in
> > ArmJuno.fdf for the line
> >   INF RuleOverride=ACPITABLE Platform/ARM/Juno/AcpiTables/AcpiTables.inf
> > generating the error message from the original build command.
> > 
> > But I'm not quite sure how to debug these issues (short of fully
> > figuring out the innards of MultipleWorkspace.py, MetaFileParser.py
> > and a few others).
> > Any ideas of where to look, or even what is going on?
> > Would these cases be expected to work?
> > 
> > /
> >     Leif
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
> > https://lists.01.org/mailman/listinfo/edk2-devel