Re: [edk2-devel] EDK2 ArmVirtQemu behaviour with multiple UARTs

["Laszlo Ersek" <lersek@redhat.com>]

On 10/2/23 03:51, Laszlo Ersek wrote:
> On 9/21/23 12:50, peter.maydell at linaro.org (Peter Maydell) wrote:
> 
>> If you want to play around with this, I have some WIP patches at
>> https://git.linaro.org/people/pmaydell/qemu-arm.git uart-edk-investigation
>> (content wise they should be fine, but I haven't cleaned them up into
>> a coherent set of distinct patches yet, so they're a bit messy.)
>> A run of QEMU with both UARTs which sends all output to files looks like:
>>
>> ./build/arm-clang/qemu-system-aarch64 -display none -vga none \
>>   -machine virt,acpi=on,virtualization=on,mte=on,gic-version=max,iommu=smmuv3 \
>>   -smp 2 -m 1024 -cpu max,pauth-impdef=on \
>>   -bios ~/linaro/edk2/QEMU_EFI_DEBUG.fd \
>>   -drive file=/home/petmay01/avocado/data/cache/by_location/0154b7cd3a4f5e135299060c8cabbeec10b70b6d/alpine-standard-3.17.2-aarch64.iso,format=raw
>> \
>>   -device virtio-rng-pci,rng=rng0 \
>>   -object rng-random,id=rng0,filename=/dev/urandom \
>>   -chardev file,id=chr0,path=/tmp/uart0-rev.txt \
>>   -chardev file,id=chr1,path=/tmp/uart1-rev.txt \
>>   -serial chardev:chr0 -serial chardev:chr1
>>
>> (adjust -serial options to taste)
> 
> Is the "uart-edk-investigation" branch still the most recent one for
> this? (I've not checked it yet, but coming close.)

I checked out your branch and built it at commit "virt: Reverse order of UART dtb nodes".

I included my edk2 patches in my edk2 binary.

The QEMU command line is

/opt/qemu-installed-optimized/bin/qemu-system-aarch64 \
  -device virtio-gpu-pci \
  -machine virt,acpi=on,virtualization=on,mte=on,gic-version=max,iommu=smmuv3 \
  -smp 2 \
  -m 1024 \
  -cpu max,pauth-impdef=on \
  -drive if=pflash,file=/home/virt-images/arm/QEMU_EFI.fd.padded,format=raw,readonly=on \
  -drive if=pflash,file=/home/virt-images/arm/QEMU_VARS.fd.padded,format=raw,snapshot=on \
  -drive if=none,file=/home/lacos/tmp/alpine-standard-3.18.4-aarch64.iso,format=raw,readonly=on,id=alpine \
  -device virtio-scsi-pci,id=vscsi \
  -device scsi-cd,drive=alpine,bus=vscsi.0,bootindex=1 \
  -device virtio-rng-pci,rng=rng0 \
  -object rng-random,id=rng0,filename=/dev/urandom \
  -device qemu-xhci,id=xhci \
  -device usb-kbd,bus=xhci.0 \
  -chardev stdio,mux=on,id=monitor-and-console \
  -mon chardev=monitor-and-console,mode=readline \
  -serial chardev:monitor-and-console \
  -chardev file,id=debug-log,path=/home/lacos/tmp/debug.txt \
  -serial chardev:debug-log

With this command line, I have three windows to watch:

(1) a graphical guest display window (virtio-gpu-pci), supposed to serve the following purposes:
(1.1) multiplexed output from the UEFI console,
(1.2) Linux virtual console (in character mode)

(2) the terminal where I start the command from, supposed to serve (mux / C-a) the following purposes:
(2.1) the QEMU monitor,
(2.2) the guest serial port that is *supposed* to serve the following purposes:
(2.2.1) direct SerialPortLib usage from edk2 modules,
(2.2.2) the UEFI console,
(2.2.3) the Linux boot log and a serial login prompt

(3) a separate terminal where I run "tail -F debug.txt", intending to see:
(3.1) the edk2 DEBUG log

Also I intend to provide input via:
(4) USB-3 keyboard when focusing the graphical window (1)
(5) QEMU monitor commands in the terminal where I start QEMU (2)
(6) serial input to the guest in the terminal where I start QEMU (2)

Results:

* edk2 writes the message

  UEFI firmware (version  built at 03:38:24 on Oct  2 2023)

to window (2), with direct SerialPortLib usage.

* the edk2 DEBUG log goes solely to the file "debug.txt" / window (3), not messing up the UEFI console (i.e., windows (1) and (2))

* the edk2 DEBUG log contains the following two lines:

  PlatformPeim: PL011 UART (console) @ 0x9000000
  PlatformPeim: PL011 UART (debug) @ 0x9040000

* the UEFI console output is properly multiplexed to windows (1) and (2), and the UEFI console properly takes input from both (4) and (6). This covers both the edk2 Boot Manager, and Grub from the alpine ISO.

* in Grub, I replace alpine's default kernel option "quiet" with the two options "ignore_loglevel efi=debug". (Because grub uses the UEFI console, it properly takes input from both (4) and (6).)

* I see the guest kernel's EFI Stub's messages on the UEFI console (multiplexed to windows (1) and (2))

* I see the kernel boot log in window (2) -- that is, on the proper serial port. This includes both kernel debug messages, and userspace boot log messages

* Finally I get the following printout:

  Welcome to Alpine Linux 3.18
  Kernel 6.1.55-0-lts on an aarch64 (TTY_LINE)

in *all three* windows. Notably, the TTY_LINE value is the following:

- in window (1) -- virtio-gpu-pci --, it is "/dev/tty1"

- in window (2) (the "console" serial port) it is "/dev/ttyAMA0"

- in window (3) (that is, the "debug.txt" file) it is "/dev/ttyAMA1"

* Linux takes keyboard input in both windows (1) and (2) via (4) and (6) respectively; (3) is not testable (and is not expected to take input) because "-chardev file" is only good for output.

So this is *total success*; the only (positive) surprise is that Linux provides a login prompt on /dev/ttyAMA1 as well.

* Rebuilding QEMU at the *parent* patch -- namely "add to acpi table" --, that is, with patch "virt: Reverse order of UART dtb nodes" *popped*, all of the above results remain identical.

* dropping the "debug-log" chardev, and the serial port that references it, from the QEMU cmdline, there is one difference (and those results generally match the current upstream status): the edk2 DEBUG log is merged into the UEFI console output as the latter is multiplexed to the sole serial port -- making a mess for the reader. And of course Linux cannot offer a login prompt on "/dev/ttyAMA1".

I say we want this!

I have one patch under review ("[edk2-devel] [PATCH 1/1] ArmVirtPkg/FdtPL011SerialPortLib: initialize implicitly"), which is independent, but contextually a pre-requisite for my "armvirt-dual-serial" branch. Once that patch is merged, I'll post this branch. And I suggest going forward with the QEMU-side patches as well.

--*--

... Just for kicks, I also tested the following change to the QEMU command line:

@@ -19,6 +19,7 @@
   -device usb-kbd,bus=xhci.0 \
   -chardev stdio,mux=on,id=monitor-and-console \
   -mon chardev=monitor-and-console,mode=readline \
-  -serial chardev:monitor-and-console \
+  -device virtio-serial-pci,id=vserial \
+  -device virtconsole,bus=vserial.0,chardev=monitor-and-console \
   -chardev file,id=debug-log,path=/home/lacos/tmp/debug.txt \
   -serial chardev:debug-log

The behavior is not immediately intuitive, but it's correct after all. The *sole* PL011 UART -- now in "debug.txt" in window (3) -- unifies several purposes in edk2: UEFI console IO (including edk2 Boot manager and Grub), direct SerialPortLib output, and DEBUG messages. The virtio-gpu-pci window (1) functions undisturbed (UEFI console). The virtio serial console in window (2) properly works as a "head" of the UEFI console.

Even with the virtio serial console, I think that having *two* PL011 serial ports will be beneficial. With virtio-serial plus *one* PL011, we can have undisturbed non-graphical UEFI console, but there still isn't an undisturbed DEBUG log -- the sole PL011 still includes both UEFI console (multiplexed) and DEBUG. If we add in the second PL011, then the DEBUG log is pristine as well, same as with isa-debugcon under x86.

Furthermore, a disadvantage of virtio-serial plus *one* PL011 is that once I boot Linux, the kernel log + userspace log goes to the PL011. That's a bit of a UX disruption, because in this setup, we tend to look at the virtio-serial console, due to the PL011 still being a mixture of UEFI console + DEBUG. So the Linux boot log appears "elsewhere", and even the login prompt does not appear on virtio-serial (probably because in this build of Alpine, no virtio-serial driver is included; I guess anyway).

So either way, to mirror the x86 experience most closely (where we have undisturbed UEFI console on the ISA serial port *and/or* on virtio serial; *plus* undisturbed DEBUG output on isa-debugcon), we need two PL011 UARTs on aarch64. With one PL011 plus virtio-serial, the UEFI console is cleaned up (on virtio-serial), but the sole PL011 still contains a mixure of DEBUG and UEFI console -- and that's the only way we get a (somewhat garbled) DEBUG log in that setup.

Thanks,
Laszlo



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109268): https://edk2.groups.io/g/devel/message/109268
Mute This Topic: https://groups.io/mt/101498371/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/leave/12367111/7686176/1913456212/xyzzy [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-