Re: [PATCH v4 edk2-platforms 20/23] Platform/Raspberry/Pi3: Add platform readme

[Ard Biesheuvel <ard.biesheuvel@linaro.org>]

On Thu, 31 Jan 2019 at 15:13, Leif Lindholm <leif.lindholm@linaro.org> wrote:
>
> On Thu, Jan 31, 2019 at 12:30:22PM +0000, Pete Batard wrote:
> > Hi Leif. Thanks for reviewing this patchset.
> >
> > On 2019.01.30 21:50, Leif Lindholm wrote:
> > > Hi Pete,
> > >
> > > I will only have minor comments on this set, but I'll start with this
> > > documentation.
> > >
> > > On Tue, Jan 29, 2019 at 04:26:52PM +0000, Pete Batard wrote:
> > > > Contributed-under: TianoCore Contribution Agreement 1.1
> > > > Signed-off-by: Pete Batard <pete@akeo.ie>
> > > > ---
> > > >   Platform/Raspberry/Pi3/Readme.md | 259 ++++++++++++++++++++
> > > >   Readme.md                        |   3 +
> > > >   2 files changed, 262 insertions(+)
> > > >
> > > > diff --git a/Platform/Raspberry/Pi3/Readme.md b/Platform/Raspberry/Pi3/Readme.md
> > > > new file mode 100644
> > > > index 000000000000..7fb59ccdc321
> > > > --- /dev/null
> > > > +++ b/Platform/Raspberry/Pi3/Readme.md
> > > > @@ -0,0 +1,259 @@
> > > > +Raspberry Pi 3 EDK2 Platform Support
> > > > +====================================
> > > > +
> > > > +# Summary
> > > > +
> > > > +This is a port of 64-bit Tiano Core UEFI firmware for the Raspberry Pi 3/3B+ platforms,
> > > > +based on [Ard Bisheuvel's 64-bit](http://www.workofard.com/2017/02/uefi-on-the-pi/)
> > > > +and [Microsoft's 32-bit](https://github.com/ms-iot/RPi-UEFI/tree/ms-iot/Pi3BoardPkg)
> > > > +implementations, as maintained by [Andrei Warkentin](https://github.com/andreiw/RaspberryPiPkg).
> > > > +
> > > > +This is meant as a generally useful 64-bit ATF + UEFI implementation for the Raspberry
> > > > +Pi 3/3B+ which should be good enough for most kind of UEFI development, as well as for
> > > > +running consummer Operating Systems in such as Linux or Windows.
> > > > +
> > > > +Raspberry Pi is a trademark of the [Raspberry Pi Foundation](http://www.raspberrypi.org).
> > > > +
> > > > +# Status
> > > > +
> > > > +This firmware, that has been validated to compile against the current
> > > > +[edk2](https://github.com/tianocore/edk2)/[edk2-platforms](https://github.com/tianocore/edk2-platforms),
> > > > +should be able to boot Linux (SUSE, Ubuntu), NetBSD, FreeBSD as well as Windows 10 ARM64
> > > > +(full GUI version).
> > > > +
> > > > +It also provides support for ATF ([Arm Trusted Platform](https://github.com/ARM-software/arm-trusted-firmware)).
> > > > +
> > > > +HDMI and the mini-UART serial port can be used for output devices, with mirrored output.
> > > > +USB keyboards and the mini-UART serial port can be used as input.
> > > > +
> > > > +The boot order is currently hardcoded, first to the USB ports and then to the uSD card.
> > > > +If there no bootable media media is found, the UEFI Shell is launched.
> > > > +<kbd>Esc</kbd> enters platform setup. <kbd>F1</kbd> boots the UEFI Shell.
> > > > +
> > > > +# Building
> > > > +
> > > > +(These instructions were validated against the latest edk2 / edk2-platforms /
> > > > +edk2-non-osi as of 2019.01.27, on a Debian 9.6 x64 system).
> > > > +
> > > > +You may need to install the relevant compilation tools. Especially you should have the
> > > > +ACPI Source Language (ASL) compiler, `nasm` as well as a native compiler installed.
> > >
> > > nasm? The x86 assembler?
> >
> > I'll remove that.
> >
> > > > +On a Debian system, you can get these prerequisites installed with:
> > > > +```
> > > > +sudo apt-get install build-essential acpica-tools nasm uuid-dev
> > > > +```
> > > > +
> > > > +**IMPORTANT:** We recommend the use of the Linaro GCC for compilation instead of
> > > > +your system's native ARM64 GCC cross compiler.
> > >
> > > This sounds like something written in the days of GCC 4.8. I doubt it
> > > has any relevance today.
> >
> > It very much had until circa one month ago, as we observed early Synchronous
> > Exceptions when trying to use the native Debian ARM64 compiler, which we did
> > not observe with Linaro's toolchain. We even had trouble (similar issue)
> > with recent Linaro toolchains at some stage, which is why, until v3, we
> > recommended an older version, but recent tests showed that the latest Linaro
> > GCC (2019.02) appeared to be okay, which is why I removed the previous
> > requirement to use exclusively Linaro's 2017.10 GCC 5.5.
>
> Urgh. But that actually makes the above statement even more
> misleading. What you have isn't an issue with non-Linaro toolchains,
> you have an unidentified toolchain issue that you've triggered more
> frequently
>

Could you please check whether the broken toolchain in question has
the workaround for Cortex-A53 erratum 843419 enabled? (gcc -v will
tell you)

While playing with this port the other day, I noticed that the RPi3 is
affected by this, and the Debian kernels don't enable mitigations for
it either.

> I mean, it's not like the
> (Also, Linaro no longer releases GCC toolchains - if you try to grab a
> GCC8 toolchain from releases.linaro.org, you get redirected to
> https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads.)
>
> > Besides, I think it's preferable when a project highlights precisely how
> > they build their own binaries, and with which toolchain, so that, if anybody
> > experiences an issue with their own build, they can compare their setup with
> > the maintainer's "official" one.
>
> Given what you say below, I think we actually understand each other
> quite well So let's try to find a compromise we're both OK with.
>
> My bottom line is that what I want in _this_ file is a high-level
> overview of the platform and the port. Things like the toolchain issue
> definitely fits in there, but as a notice stating very clearly that
> "we have an unidentified toolchain issue - if you see issues with your
> default toolchain, ...".
>
> I'm also fully OK with listing specific versions of toolchains that
> have been tested with success.
>
> But I don't want the full-on handholding documentation for someone who
> has never seen EDK2 before in the Readme.md (and in every other platform).
> Ideally, I would like to see that documentation "somewhere else",
> which could be referenced from here, or from the wiki pages like
> https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Platforms).
>
> If you find that less-than-ideal, I'd take a separate file (probably
> the same as the OS installation one) linked to from this one.
>
> > > Also, and this applies both above and below: I am trying very hard to
> > > get rid of (mostly unnecessary) platform-specific build instructions.
> > > The top-level Readme.md in this repository contains basic build
> > > instructions. I would much prefer if you can refer to that instead and
> > > drop everything after the # Building header above...
> >
> > Okay. This is a bit at odds with the goal I'm trying to achieve here, which
> > is to save time and bewilderment from developers who might be trying to
> > build this specific platform and encounter an issue where they'll want to
> > eliminate the possibility that their setup/configuration is the problem.
> >
> > I've ran in too many of "works on my machine" not top want to also provide
> > "here is exactly how the official developer's machine was set up when they
> > ran their built" from the get go, to try to alleviate the usual headaches of
> > trying to solve environmental issues... Furthermore, given the popularity of
> > the Raspberry Pi platform, my guess is that we're going to get quite a few
> > people who aren't that familiar with the EDK2, or even building things, in
> > general and who'll want a set of "copy/paste exactly this in your shell and
> > you *should* end up with a firmware binary" set of instructions, which is
> > why I've attempted to provide that, in a single location.
> >
> > Now, I agree that this may run contrary to what you (and possibly other
> > people) want, so I'm not going to push the matter further if you think
> > having a set of duplicate "hand holding" set of build instructions in this
> > readme is uncalled for. But I'd rather take pre-emptive steps to avoid
> > having to deal with a potentially large number of "I tried to build and it
> > didn't work" requests that might come from providing firmware builds for a
> > very popular platform...
>
> So to be clear, in addition to what I said above: I strongly support
> the idea of that documentation existing. I am less in favour of it
> residing in the firmware source tree.
>
> But the thing I _will_ keep going on about is the assumption of cross
> compilation. Now, I have x86 machines, and sometimes I use them for
> building. But I also have AArch64 servers, our CI infrastructure lives
> on AArch64 vms, I'm typing this on an AArch64-based laptop (which also
> holds Visual Studio and GCC/CLANG).
> And, partly because of this, I have not used Linaro-released
> toolchains since early GCC5 days.
>
> > > > +
> > > > +You can then build the firmware as follows:
> > > > +
> > > > +* Standalone instructions
> > > > +
> > > > +```
> > > > +mkdir ~/workspace
> > > > +cd ~/workspace
> > > > +git clone https://github.com/tianocore/edk2.git
> > > > +# The following is only needed once, after you cloned edk2
> > > > +make -C edk2/BaseTools
> > > > +git clone https://github.com/tianocore/edk2-platforms.git
> > > > +git clone https://github.com/tianocore/edk2-non-osi.git
> > > > +wget https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz
> > > > +tar -xJvf gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz
> > > > +# If you have multiple AARCH64 toolchains, make sure the above one comes first in your path
> > > > +export PATH=$PWD/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu/bin:$PATH
> > > > +export GCC5_AARCH64_PREFIX=aarch64-linux-gnu-
> > > > +export WORKSPACE=$PWD
> > > > +export PACKAGES_PATH=$WORKSPACE/edk2:$WORKSPACE/edk2-platforms:$WORKSPACE/edk2-non-osi
> > > > +. edk2/edksetup.sh
> > >
> > > ...down to here.
> > > (I would certainly not object if you felt the need to improve on the
> > > contents of the top-level Readme.md. For example with the explicit
> > > prerequisite installation steps, rather than the brief list provided
> > > under the Prerequisites section currently on that page.)
> > >
> > > > +build -a AARCH64 -t GCC5 -p edk2-platforms/Platform/Raspberry/Pi3/RPi3.dsc -DBUILD_EPOCH=`date +%s` -b RELEASE
> > > > +```
> > > > +
> > >
> > > The below _is_ platform-specific, so clearly I have no issue with
> > > it. Not sure that needed poinging out, but...
> > > And the above line points out the location of the .dsc, so no
> > > objection there.
> > >
> > > > +# Booting the firmware
> > > > +
> > > > +1. Format a uSD card as FAT32
> > > > +2. Copy the generated `RPI_EFI.fd` firmware onto the partition
> > > > +3. Download and copy the following files from https://github.com/raspberrypi/firmware/tree/master/boot
> > > > +  - `bootcode.bin`
> > > > +  - `fixup.dat`
> > > > +  - `start.elf`
> > > > +4. Create a `config.txt` with the following content:
> > > > +  ```
> > > > +  arm_control=0x200
> > > > +  enable_uart=1
> > > > +  armstub=RPI_EFI.fd
> > > > +  disable_commandline_tags=1
> > > > +  ```
> > > > +5. Insert the uSD card and power up the Pi.
> > > > +
> > > > +Note that if you have a model 3+ or a model 3 where you enabled USB boot through OTP
> > > > +(see [here](https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md))
> > > > +you may also be able to boot from a FAT32 USB driver rather than uSD.
> > > > +
> > > > +# Notes
> > > > +
> > > > +## ARM Trusted Firmware (ATF)
> > > > +
> > > > +The ATF binaries being used were compiled from the ATF mainline.
> > >
> > > With additional modifications?
> >
> > No. It would have been mentioned otherwise.
> >
> > > > +
> > > > +For more details on the ATF compilation, see the [README](./Binary/README.md) in the `Binary/` directory.
> > >
> > > This directory no longer exists.
> > > You could point at the full URL in the edk2-non-osi repository.
> >
> > Yes, I will do that, since it contains important data about the compilation
> > options that were used for ATF. Ultimately though, unless you changed your
> > mind (since I believe this is something you wanted), we want to remove the
> > ATF binaries in non-osi and simply point to build instructions (as well as
> > links to automated built binaries from AppVeyor or something similar, for
> > people who don't want to have to go through the extra build step). I am
> > still waiting on a new dot release of ATF to work on that.
>
> Yeah, sure, we covered that before.
>
> > > > +
> > > > +## Custom Device Tree
> > > > +
> > > > +The default Device Tree included in the firmware is the one for a Raspberry Pi 3 Model B (not B+).
> > > > +If you want to use a different Device Tree, to boot a Pi 3 Model B+ for instance (for which a
> > > > +DTB is also provided under `DeviceTree/`), you should copy the relevant `.dtb` into the root of
> > > > +the SD or USB, and then edit your `config.txt` so that it looks like:
> > > > +
> > > > +```
> > > > +(...)
> > > > +disable_commandline_tags=2
> > > > +device_tree_address=0x10000
> > > > +device_tree_end=0x20000
> > > > +device_tree=bcm2710-rpi-3-b-plus.dtb
> > > > +```
> > > > +
> > > > +Note: the address range **must** be `[0x10000:0x20000]`.
> > > > +`dtoverlay` and `dtparam` parameters are also supported **when** providing a Device Tree`.
> > > > +
> > > > +## Custom `bootargs`
> > > > +
> > > > +This firmware will honor the command line passed by the GPU via `cmdline.txt`.
> > > > +
> > > > +Note, that the ultimate contents of `/chosen/bootargs` are a combination of several pieces:
> > > > +- Original `/chosen/bootargs` if using the internal DTB. Seems to be completely discarded by GPU when booting with a custom device tree.
> > > > +- GPU-passed hardware configuration. This one is always present.
> > > > +- Additional boot options passed via `cmdline.txt`.
> > > > +
> > > > +# Tested Platforms
> > >
> > > I won't press the issue, but I would prefer for the operating system
> > > installation instructions not to be included.
> >
> > Okay. My reasoning here is similar to the build instructions: As opposed to
> > what might be the case for other platforms, due to the Raspberry Pi's
> > popularity, a lot of people might be coming to this readme for help on how
> > to use the firmware to install an OS and they may start to spam the list or
> > package maintainers if they don't find the information they want. So we want
> > to ensure we provide at least one good walk-through example that people can
> > refer to.
> >
> > > If you do want it in this repo, please break it out into a separate
> > > Readme.md, which you can link to from here.
> >
> > I'll do that then, as I believe it's important to have at least getting to a
> > fully configured Linux installation documented, especially this early in the
> > game. Once relying on this firmware (hopefully) starts to become the norm
> > for popular Linux distros, having these instructions should be a lot less
> > important.
>
> Yes, this is what I am hoping. Once we get the toolchain issue figured
> out, and once distros start assuming UEFI (whether EDK2 or U-Boot), we
> shouldn't need this level of device-specific info. Only how to get
> your firmware installed on your device.
>
> Regards,
>
> Leif
>
> > > Basically, skp straight to "Limitations" from here.
> > >
> > > > +
> > > > +## Ubuntu
> > > > +
> > > > +[Ubuntu 18.04 LTS](http://releases.ubuntu.com/18.04/) has been tested and confirmed to work,
> > > > +on a Raspberry 3 Model B, including the installation process. Note however that network
> > > > +installation and networking may not work on the Model B+, due to the `lan78xx` Linux driver
> > > > +still requiring some support.
> > > > +
> > > > +Below are the steps you can follow to install Ubuntu LTS onto SD/USB:
> > > > +* Download the latest Ubuntu LTS ARM64 [`mini.iso`](http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/installer-arm64/current/images/netboot/mini.iso).
> > > > +* Partition the media as MBR and create a ~200 MB FAT32 partition on it with MBR type `0x0c`.
> > > > +  Note: Do not be tempted to use GPT partition scheme or `0xef` (EFI System Partition) for the
> > > > +  type, as none of these are supported by the Raspberry Pi's internal boot rom.
> > > > +* Extract the full content of the ISO onto the partition you created.
> > > > +* Also extract the GRUB EFI bootloader `bootaa64.efi` from `/boot/grub/efi.img` to `/boot/grub/`.
> > > > +  Note: Do not be tempted to copy this file to another directory (such as `/efi/boot/`) as GRUB looks for its
> > > > +  modules and configuration data in the same directory as the EFI loader and also, the installation
> > > > +  process will create a `bootaa64.efi` into `/efi/boot/`.
> > > > +* If needed, copy the UEFI firmware files (`RPI_EFI.fd`, `bootcode.bin`, `fixup.dat` and `start.elf`)
> > > > +  onto the FAT partition.
> > > > +* Boot the pi and let it go into the UEFI shell.
> > > > +* Navigate to `fs0:` then `/boot/grub/` and launch the GRUB efi loader.
> > > > +* Follow the Ubuntu installation process.
> > > > +
> > > > +Note: Because Ubuntu operates in quiet mode by default (no boot messages), you may think the system is frozen
> > > > +on first reboot after installation. However, if you wait long enough you **will** get to a login prompt.
> > > > +
> > > > +Once Linux is running, if desired, you can disable quiet boot, as well as force the display
> > > > +of the GRUB selector, by editing `/etc/default/grub` and changing:
> > > > +* `GRUB_TIMEOUT_STYLE=hidden` &rarr; `GRUB_TIMEOUT_STYLE=menu`
> > > > +* `GRUB_CMDLINE_LINUX_DEFAULT="splash quiet"` &rarr; `GRUB_CMDLINE_LINUX_DEFAULT=""`
> > > > +
> > > > +Then, to have your changes applied run `update-grub` and reboot.
> > > > +
> > > > +## Other Linux distributions
> > > > +
> > > > +* Debian ARM64 does not currently work, most likely due to missing required module support
> > > > +  in its kernel. However its installation process works, so it may be possible to get it
> > > > +  running with a custom kernel.
> > > > +
> > > > +* OpenSUSE Leap 42.3 has been reported to work on Raspberry 3 Model B.
> > > > +
> > > > +* Other ARM64 Linux releases, that support UEFI boot and have the required hardware support
> > > > +  for Pi hardware are expected to run, though their installation process might require some
> > > > +  cajoling.
> > > > +
> > > > +## Windows
> > > > +
> > > > +Windows 10 1809 for ARM64 (build 17763) has been tested and confirmed to work (after replacing
> > > > +`C:\Windows\System32\Drivers\WppRecorder.sys` with an older version, since the one from 1809
> > > > +appears to be buggy across all archs, and results in a similar BSOD when trying to run Windows
> > > > +To Go on x64 with native drivers for instance).
> > > > +
> > > > +Windows 10 1803 for ARM64 and earlier do not work due to the presence of a hardware ASSERT check
> > > > +in the Windows kernel, that was removed in later versions.
> > > > +
> > > > +You probably want to look at https://www.worproject.ml/ as well as the
> > > > +[Windows thread in the original RaspberryPiPkg](https://github.com/andreiw/RaspberryPiPkg/issues/12)
> > > > +for installation details.
> > > > +
> > > > +## Other platforms
> > > > +
> > > > +Details you may need to run other platforms, including FreeBSD, is provided in the
> > > > +[Readme from the original RaspberryPiPkg](https://github.com/andreiw/RaspberryPiPkg).
> > > > +
> > > > +# Limitations
> > > > +
> > > > +## HDMI
> > > > +
> > > > +The UEFI HDMI video support relies on the VC (that's the GPU)
> > > > +firmware to correctly detect and configure the attached screen.
> > > > +Some screens are slow, and this detection may not occur fast
> > > > +enough. Finally, you may wish to be able to boot your Pi
> > > > +headless, yet be able to attach a display to it later for
> > > > +debugging.
> > > > +
> > > > +To accommodate these issues, the following extra lines
> > > > +are recommended for your `config.txt`:
> > > > +- `hdmi_force_hotplug=1` to allow plugging in video after system is booted.
> > > > +- `hdmi_group=1` and `hdmi_mode=4` to force a specific mode, both to accommodate
> > > > +   late-plugged screens or buggy/slow screens. See [official documentation](https://www.raspberrypi.org/documentation/configuration/config-txt/video.md)
> > > > +   to make sense of these parameters (example above sets up 720p 60Hz).
> > > > +
> > > > +## NVRAM
> > > > +
> > > > +The Raspberry Pi has no NVRAM.
> > > > +
> > > > +NVRAM is emulated, with the non-volatile store backed by the UEFI image itself. This means
> > > > +that any changes made in UEFI proper will be persisted, but changes made in HLOS will not.
> > > > +It would be nice to implement ATF-assisted warm reboot, to allow persisting HLOS
> > > > +NVRAM changes.
> > > > +
> > > > +## RTC
> > > > +
> > > > +The Rasberry Pi has no RTC.
> > > > +
> > > > +`RtcEpochSeconds` NVRAM variable is used to store the boot time
> > > > +This should allow you to set whatever date/time you
> > > > +want using the Shell date and time commands. While in UEFI
> > > > +or HLOS, the time will tick forward. `RtcEpochSeconds`
> > > > +is not updated on reboots.
> > > > +
> > > > +## uSD
> > > > +
> > > > +UEFI supports both the Arasan SDHCI and the Broadcom SDHost controllers to access the uSD slot.
> > > > +You can use either. The other controller gets routed to the SDIO card. The choice made will
> > > > +impact ACPI OSes booted (e.g. Windows 10). Arasan, being an SDIO controller, is usually used
> > > > +with the WiFi adapter where available. SDHost cannot be used with SDIO. In UEFI setup screen:
> > > > +- go to `Device Manager`
> > > > +- go to `Raspberry Pi Configuration`
> > > > +- go to `Chipset`
> > > > +- configure `Boot uSD Routing`
> > > > +
> > > > +Known issues:
> > > > +- Arasan HS/4bit support is missing.
> > > > +- No 8 bit mode support for (e)MMC (irrelevant for the Pi 3).
> > > > +- Hacky (e)MMC support (no HS).
> > > > +- No card removal/replacement detection, tons of timeouts and slow down during boot without an uSD card present.
> > > > +
> > > > +## USB
> > > > +
> > > > +- USB1 BBB mass storage devices untested (USB2 and USB3 devices are fine).
> > > > +- USB1 CBI mass storage devices don't work (e.g. HP FD-05PUB floppy).
> > > > +
> > > > +## ACPI
> > > > +
> > > > +ACPI should match the MS-IoT one. Both Arasan and SDHost SD controllers are exposed.
> > >
> > > It would be good if you could add a comment here about the limitations
> > > with regards to proper ACPI description and its unusability for Linux.
> >
> > Will do.
> >
> > > > +
> > > > +## Missing Functionality
> > > > +
> > > > +- Network booting via onboard NIC.
> > > > +- Ability to switch UART use to PL011.
> > > > diff --git a/Readme.md b/Readme.md
> > > > index 384b1d3c5e2b..d82b7581ba6d 100644
> > > > --- a/Readme.md
> > > > +++ b/Readme.md
> > > > @@ -217,6 +217,9 @@ they will be documented with the platform.
> > > >   ## Marvell
> > > >   * [Armada 70x0](Platform/Marvell/Armada)
> > > > +## Raspberry
> > >
> > > Raspberry Pi
> >
> > Yes, that makes sense with your further note.
> >
> > > /
> > >      Leif
> > >
> > > > +* [Pi 3](Platform/Raspberry/Pi3)
> > > > +
> > > >   ## Socionext
> > > >   * [SynQuacer](Platform/Socionext/DeveloperBox)
> > > > --
> > > > 2.17.0.windows.1
> > > >
> >