Re: ARM virt machine boots fail with 14 ioh3420

[Laszlo Ersek <lersek@redhat.com>]

Adding Andrea, Ard, Drew and Marcel; and the main qemu list

On 04/13/17 09:37, Shannon Zhao wrote:
> Hi,
> 
> I'm testing the PCIe devices hotplug for ARM virt machine and using
> ioh3420 as root port. I found that below command line could work.
> 
> qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
> QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
> ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
> ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
> ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
> ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
> ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
> ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
> ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
> ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
> ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
> ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
> ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
> ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
> ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
> i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
> pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
> usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
> virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
> -drive
> file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
> -device
> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
> -netdev tap,id=hostnet1,vhost=on -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet2,vhost=on -device
> virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet3,vhost=on -device
> virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet4,vhost=on -device
> virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet5,vhost=on -device
> virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet6,vhost=on -device
> virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet7,vhost=on -device
> virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet8,vhost=on -device
> virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet9,vhost=on -device
> virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet10,vhost=on -device
> virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet11,vhost=on -device
> virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet12,vhost=on -device
> virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
> -nographic
> 
> But if I add one more ioh3420 device by appending above command with
> "-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
> the guest can't boot. It seems that the firmware doesn't recognize the
> PCIe devices and print "Connect: PciRoot(0x0): Not Found".
> 
> I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any limitation
> of the supported PCIe devices?

In one sentence: you are running out of (emulated) IO space.

Aarch64 does not have "IO space", but with QEMU, using the "virt"
machine type, we emulate 64KB of IO space, mapped to a special MMIO
range. This is available for PCI resource allocation, for such devices
that have IO BARs (and for such PCI bridges that reserve IO space for
hotplug purposes).

The ioh3420 PCI Express Root Port device represents such a bridge. Even
if you plug a PCI Express device into it that has only MMIO BARs, the
bridge still advertises IO support, and it causes the firmware (and/or
Linux) to reserve 4KB of IO space. With ~15-16 such ports, you run out
of the 64 KB IO aperture, and the resource assignment fails.

The solution to this problem comes together from several parts:

(1) New, vendor-independent device models in QEMU, for PCI Express Root
Ports and Downstream Ports, that (optionally) do not advertise any
support at all for IO BARs. This is on Marcel's task list. Please refer to:

  generic port device model:
    https://bugzilla.redhat.com/show_bug.cgi?id=1390316

  optional disablement of IO space:
    https://bugzilla.redhat.com/show_bug.cgi?id=1344299

(2) A similar question exists for MMIO space. While there the issue
isn't exactly "running out of available aperture", we still want to
control the MMIO reservation size for root ports and downstream ports --
different devices that are to be hot-plugged can require different
reservations in advance. For this, please refer to:

  sizing of MMIO reservation:
    https://bugzilla.redhat.com/show_bug.cgi?id=1437113

(3) In the firmware, platform code can control the aperture reservation
of bridges, for hotplug purposes, while the generic PCI Bus driver
enumerates devices and assigns resources. The driver that does this is

  OvmfPkg/PciHotPlugInitDxe

At the moment, this driver is only included in OVMF, not in ArmVirtQemu.
Plus, it doesn't really do the right thing in OVMF either.

Once items (1) and (2) above are solved, I will update PciHotPlugInitDxe
so that it adheres to the reservation sizes dictated by QEMU. For this,
please refer to:

  adhere to IO reservation disablement:
    https://bugzilla.redhat.com/show_bug.cgi?id=1434740
    (depends on BZ#1344299 above)

  adhere to MMIO reservation sizing:
    https://bugzilla.redhat.com/show_bug.cgi?id=1434747
    (depends on BZ#1437113 above)

We plan to work on these items in the following months (upstream first,
obviously).

--*--

Please read the following document in the QEMU source tree:

  docs/pcie.txt

It will give you the "grand vision" we have for PCI Express on the Q35
(x86) machine type, and most of that -- intentionally! -- applies to the
"virt" (aarch64) machine type as well.

See also the following QEMU commits (especially the second one):

  9ca019c1dd57 q35: Improve sample configuration files
  166d434685ed mach-virt: Provide sample configuration files

Thanks!
Laszlo