The arm virtual machine displays problems in QXL during the UEFI phase

The arm virtual machine displays problems in QXL during the UEFI phase

[Lange Tang]

[-- Attachment #1: Type: text/plain, Size: 2821 bytes --]

Hi everyone:


In order to support QXL display during the UEFI phase of the arm64 virtual machine, the modified code is as follows (based on master branch, commit id:17143c4837393d42c484b42d1789b85b2cff1aaf):


diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
index 7ef5e7297b..d73cf6aba0 100644
--- a/ArmVirtPkg/ArmVirtQemu.dsc
+++ b/ArmVirtPkg/ArmVirtQemu.dsc
@@ -501,6 +501,7 @@
   OvmfPkg/QemuRamfbDxe/QemuRamfbDxe.inf
   OvmfPkg/VirtioGpuDxe/VirtioGpu.inf
   OvmfPkg/PlatformDxe/Platform.inf
+  OvmfPkg/QemuVideoDxe/QemuVideoDxe.inf
 
   #
   # USB Support
diff --git a/ArmVirtPkg/ArmVirtQemu.fdf b/ArmVirtPkg/ArmVirtQemu.fdf
index b5e2253295..289d0c580b 100644
--- a/ArmVirtPkg/ArmVirtQemu.fdf
+++ b/ArmVirtPkg/ArmVirtQemu.fdf
@@ -126,3 +126,8 @@ READ_LOCK_STATUS   = TRUE
   }
 
 !include ArmVirtRules.fdf.inc
+
+#
+# VGA Module
+#
+# INF OvmfPkg/QemuVideoDxe/QemuVideoDxe.inf


Compiled it and generate QEMU_EFI.fd, Create virtual machines by qemu, as follows:


./qemu-6.0.0/build/qemu-system-aarch64 \
-machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \
-cpu host \
-m 8192 \
-smp 16 \
-bios QEMU_EFI.fd \
-vnc 127.0.0.1:19 \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \
-device pcie-pci-bridge,id=pci.9,bus=pci.6,addr=0x0 \
-netdev user,id=hostnet0 \
-device virtio-net-pci,id=hostnet0,mac=52:54:00:14:72:20,bus=pci.1,addr=0x0 \
-drive file=/var/lib/libvirt/images/tlj-kylin.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pcie.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-device qxl-vga,id=video0,bus=pci.7,addr=0x0


The test results are as follows:
1. When the QXL device hangs on the bus=pci.9, addr=0x1 or bus=pci.7, addr=0x0, there is no display during the UEFI phase, and the following error is found by debug UEFI firmware log:
  QemuVideo: BochsID mismatch (got 0x0). From the OgmfPkg/QemuVideoDxe/Driver .c
2. When the qxl device hangs in the bus=pci.8, addr=0x0, it can be displayed normally during the UEFI phase. BochsID=0xb0c5


The way to read BochsID is PIO, and ARM does not support PIO commands, should it be changed to MMIO? Or others?
Hope to receive your suggestions or comments. Thank you!


Best regards



[-- Attachment #2: Type: text/html, Size: 5120 bytes --]

Re: [edk2-devel] The arm virtual machine displays problems in QXL during the UEFI phase

[Gerd Hoffmann]

On Tue, Nov 02, 2021 at 08:55:39PM +0800, Lange Tang wrote:
> Hi everyone:
> 
> 
> In order to support QXL display during the UEFI phase of the arm64 virtual machine,

Not going to fly.  Use virtio-gpu instead.

Anything with a virtual pci memory bar is going to have cache attribute
problems on arm.

take care,
  Gerd

Re: [edk2-devel] The arm virtual machine displays problems in QXL during the UEFI phase

[Lange Tang]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Hi Gerd:
Thanks for your reply. In fact, I have no choice, only QXL in my work.
1. I wonder why the device display be normal when it hung on bus=pci.8,addr=0x0, but it is abnormal when bus=pci.9,addr=0x1 or bus=pci.7,addr=0x0.
2. Why cache properties are going to cause QXL display abnormal on ARM. Are there any links or materials? Thanks.




take care
Lange





At 2021-11-02 21:35:06, "Gerd Hoffmann" <kraxel@redhat.com> wrote:
>On Tue, Nov 02, 2021 at 08:55:39PM +0800, Lange Tang wrote:
>> Hi everyone:
>> 
>> 
>> In order to support QXL display during the UEFI phase of the arm64 virtual machine,
>
>Not going to fly.  Use virtio-gpu instead.
>
>Anything with a virtual pci memory bar is going to have cache attribute
>problems on arm.
>
>take care,
>  Gerd
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1203 bytes --]

Re: [edk2-devel] The arm virtual machine displays problems in QXL during the UEFI phase

[Gerd Hoffmann]

On Wed, Nov 03, 2021 at 09:46:01AM +0800, Lange Tang wrote:
> Hi Gerd:
> Thanks for your reply. In fact, I have no choice, only QXL in my work.
> 1. I wonder why the device display be normal when it hung on bus=pci.8,addr=0x0, but it is abnormal when bus=pci.9,addr=0x1 or bus=pci.7,addr=0x0.

Placing qxl behind a pci bridge is problematic too (even on x86).
I'm surprised it works at all.  qxl needs io ports 0x01ce + 0x1cf for
programming video modes in vga compatibility mode, and pci bridges
typically don't pass that though.

Maybe the working bridge got the 0x0000 -> 0x0fff io window assigned.
Wouldn't happen on x86 because of the legacy hardware in the 0x000 ->
0x3ff range, but maybe it is used for pci-pci bridges on arm.

> 2. Why cache properties are going to cause QXL display abnormal on
> ARM. Are there any links or materials? Thanks.

The fundamental issue is that the pci memory br for vram is virtual.
The guest maps it as io memory, but it actually is normal ram, so guest
and host map the same memory with different attributes.  That just
doesn't work on arm and results in display corruption when actually
running on arm hardware (when running in emulation, for example using
qemu on a x86 machine, this doesn't happen).

This is the reason why QemuVideoDxe is not compiled into ArmVirt.

take care,
  Gerd

Re: [edk2-devel] The arm virtual machine displays problems in QXL during the UEFI phase

[Lange Tang]

[-- Attachment #1: Type: text/plain, Size: 5488 bytes --]

Hi Gerd:
Thanks for your reply.
>1. I wonder why the device display be normal when it hung on bus=pci.8,addr=0x0, but it is abnormal when bus=pci.9,addr=0x1 or bus=pci.7,addr=0x0.

> Placing qxl behind a pci bridge is problematic too (even on x86).
> I'm surprised it works at all.  qxl needs io ports 0x01ce + 0x1cf for
> programming video modes in vga compatibility mode, and pci bridges
> typically don't pass that though.
>
> Maybe the working bridge got the 0x0000 -> 0x0fff io window assigned.
> Wouldn't happen on x86 because of the legacy hardware in the 0x000 ->
> 0x3ff range, but maybe it is used for pci-pci bridges on arm.
qxl device msg as follow,when it behind bus=pci.8,addr=0x0:
```
VGA compatible controller [0300]: Red Hat, Inc. QXL paravirtual graphic card [1b36:0100] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Physical Slot: 0-8
        Flags: bus master, fast devsel, latency 0, IRQ 39
        Memory at 14000000 (32-bit, non-prefetchable) [size=64M]
        Memory at 10000000 (32-bit, non-prefetchable) [size=64M]
        Memory at 18000000 (32-bit, non-prefetchable) [size=8K]
        I/O ports at 0000 [virtual] [size=32]
        Expansion ROM at 18010000 [disabled] [size=64K]
        Kernel driver in use: qxl
        Kernel modules: qxl
```
his working bridge io window as follow:
```
PCI bridge [0604]: Red Hat, Inc. QEMU PCIe Root port [1b36:000c] (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 39
        Memory at 19440000 (32-bit, non-prefetchable) [size=4K]
        Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
        I/O behind bridge: 00000000-00000fff [size=4K]
        Memory behind bridge: 10000000-180fffff [size=129M]
        Prefetchable memory behind bridge: 0000008000b00000-0000008000cfffff [size=2M]
        Capabilities: [54] Express Root Port (Slot+), MSI 00
        Capabilities: [48] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [40] Subsystem: Red Hat, Inc. Device [1b36:0000]
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Access Control Services
        Kernel driver in use: pcieport
```


qxl device msg as follow,when it behind bus=pci.9,addr=0x1:
```
VGA compatible controller [0300]: Red Hat, Inc. QXL paravirtual graphic card [1b36:0100] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100]
        Physical Slot: 1
        Flags: fast devsel, IRQ 39
        Memory at 14000000 (32-bit, non-prefetchable) [size=64M]
        Memory at 10000000 (32-bit, non-prefetchable) [size=64M]
        Memory at 18000000 (32-bit, non-prefetchable) [size=8K]
        I/O ports at 2000 [size=32]
        Expansion ROM at 18010000 [disabled] [size=64K]
        Kernel driver in use: qxl
        Kernel modules: qxl
```

his working bridge io window as follow:
``` PCI bridge [0604]: Red Hat, Inc. Device [1b36:000e] (prog-if 00 [Normal decode]) Physical Slot: 0-6 Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 52 Memory at 18100000 (64-bit, non-prefetchable) [size=256] Bus: primary=07, secondary=08, subordinate=08, sec-latency=0 I/O behind bridge: 00002000-00002fff [size=4K] Memory behind bridge: 10000000-180fffff [size=129M] Prefetchable memory behind bridge: [disabled] Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable+ 64bit+ Capabilities: [84] Power Management version 3 Capabilities: [48] Express PCI-Express to PCI/PCI-X Bridge, MSI 00 Capabilities: [40] Hot-plug capable Capabilities: [100] Advanced Error Reporting Kernel driver in use: shpchp ```

From the above point of view, your speculation is right. excellent!
In this way, if a QXL device wants to work on ARM, the io window of the bridge it is mounted on must be allocated as 00000000-00000fff [size=4K].
In my work, by modifying the qemu code, the io window of the bridge mounted with the qxl device is fixed to 00000000-00000fff to solve this problem. what do you think?
take care
Lange




















At 2021-11-03 14:03:22, "Gerd Hoffmann" <kraxel@redhat.com> wrote:
>On Wed, Nov 03, 2021 at 09:46:01AM +0800, Lange Tang wrote:
>> Hi Gerd:
>> Thanks for your reply. In fact, I have no choice, only QXL in my work.
>> 1. I wonder why the device display be normal when it hung on bus=pci.8,addr=0x0, but it is abnormal when bus=pci.9,addr=0x1 or bus=pci.7,addr=0x0.
>
>Placing qxl behind a pci bridge is problematic too (even on x86).
>I'm surprised it works at all.  qxl needs io ports 0x01ce + 0x1cf for
>programming video modes in vga compatibility mode, and pci bridges
>typically don't pass that though.
>
>Maybe the working bridge got the 0x0000 -> 0x0fff io window assigned.
>Wouldn't happen on x86 because of the legacy hardware in the 0x000 ->
>0x3ff range, but maybe it is used for pci-pci bridges on arm.
>
>> 2. Why cache properties are going to cause QXL display abnormal on
>> ARM. Are there any links or materials? Thanks.
>
>The fundamental issue is that the pci memory br for vram is virtual.
>The guest maps it as io memory, but it actually is normal ram, so guest
>and host map the same memory with different attributes.  That just
>doesn't work on arm and results in display corruption when actually
>running on arm hardware (when running in emulation, for example using
>qemu on a x86 machine, this doesn't happen).
>
>This is the reason why QemuVideoDxe is not compiled into ArmVirt.
>
>take care,
>  Gerd
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 6288 bytes --]

Re: [edk2-devel] The arm virtual machine displays problems in QXL during the UEFI phase

[Gerd Hoffmann]

  Hi,

> From the above point of view, your speculation is right. excellent!
> In this way, if a QXL device wants to work on ARM, the io window of the bridge it is mounted on must be allocated as 00000000-00000fff [size=4K].

That doesn't solve the cache attribute issues though ...

> In my work, by modifying the qemu code, the io window of the bridge
> mounted with the qxl device is fixed to 00000000-00000fff to solve
> this problem. what do you think?

It's not qemu but tianocore who assigns bridge windows, should be
in pci scan order, i.e. lowest slot.func goes first and gets the
0000-0fff range.

take care,
  Gerd