ARM virt machine boots fail with 14 ioh3420

ARM virt machine boots fail with 14 ioh3420

[Shannon Zhao]

Hi,

I'm testing the PCIe devices hotplug for ARM virt machine and using
ioh3420 as root port. I found that below command line could work.

qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
-drive
file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-netdev tap,id=hostnet1,vhost=on -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet2,vhost=on -device
virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet3,vhost=on -device
virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet4,vhost=on -device
virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet5,vhost=on -device
virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet6,vhost=on -device
virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet7,vhost=on -device
virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet8,vhost=on -device
virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet9,vhost=on -device
virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet10,vhost=on -device
virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet11,vhost=on -device
virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
-netdev tap,id=hostnet12,vhost=on -device
virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
-nographic

But if I add one more ioh3420 device by appending above command with
"-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
the guest can't boot. It seems that the firmware doesn't recognize the
PCIe devices and print "Connect: PciRoot(0x0): Not Found".

I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any limitation
of the supported PCIe devices?

Thanks,
-- 
Shannon

Re: ARM virt machine boots fail with 14 ioh3420

[Laszlo Ersek]

Adding Andrea, Ard, Drew and Marcel; and the main qemu list

On 04/13/17 09:37, Shannon Zhao wrote:
> Hi,
> 
> I'm testing the PCIe devices hotplug for ARM virt machine and using
> ioh3420 as root port. I found that below command line could work.
> 
> qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
> QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
> ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
> ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
> ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
> ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
> ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
> ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
> ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
> ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
> ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
> ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
> ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
> ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
> ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
> i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
> pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
> usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
> virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
> -drive
> file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
> -device
> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
> -netdev tap,id=hostnet1,vhost=on -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet2,vhost=on -device
> virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet3,vhost=on -device
> virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet4,vhost=on -device
> virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet5,vhost=on -device
> virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet6,vhost=on -device
> virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet7,vhost=on -device
> virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet8,vhost=on -device
> virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet9,vhost=on -device
> virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet10,vhost=on -device
> virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet11,vhost=on -device
> virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
> -netdev tap,id=hostnet12,vhost=on -device
> virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
> -nographic
> 
> But if I add one more ioh3420 device by appending above command with
> "-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
> the guest can't boot. It seems that the firmware doesn't recognize the
> PCIe devices and print "Connect: PciRoot(0x0): Not Found".
> 
> I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any limitation
> of the supported PCIe devices?

In one sentence: you are running out of (emulated) IO space.

Aarch64 does not have "IO space", but with QEMU, using the "virt"
machine type, we emulate 64KB of IO space, mapped to a special MMIO
range. This is available for PCI resource allocation, for such devices
that have IO BARs (and for such PCI bridges that reserve IO space for
hotplug purposes).

The ioh3420 PCI Express Root Port device represents such a bridge. Even
if you plug a PCI Express device into it that has only MMIO BARs, the
bridge still advertises IO support, and it causes the firmware (and/or
Linux) to reserve 4KB of IO space. With ~15-16 such ports, you run out
of the 64 KB IO aperture, and the resource assignment fails.

The solution to this problem comes together from several parts:

(1) New, vendor-independent device models in QEMU, for PCI Express Root
Ports and Downstream Ports, that (optionally) do not advertise any
support at all for IO BARs. This is on Marcel's task list. Please refer to:

  generic port device model:
    https://bugzilla.redhat.com/show_bug.cgi?id=1390316

  optional disablement of IO space:
    https://bugzilla.redhat.com/show_bug.cgi?id=1344299

(2) A similar question exists for MMIO space. While there the issue
isn't exactly "running out of available aperture", we still want to
control the MMIO reservation size for root ports and downstream ports --
different devices that are to be hot-plugged can require different
reservations in advance. For this, please refer to:

  sizing of MMIO reservation:
    https://bugzilla.redhat.com/show_bug.cgi?id=1437113

(3) In the firmware, platform code can control the aperture reservation
of bridges, for hotplug purposes, while the generic PCI Bus driver
enumerates devices and assigns resources. The driver that does this is

  OvmfPkg/PciHotPlugInitDxe

At the moment, this driver is only included in OVMF, not in ArmVirtQemu.
Plus, it doesn't really do the right thing in OVMF either.

Once items (1) and (2) above are solved, I will update PciHotPlugInitDxe
so that it adheres to the reservation sizes dictated by QEMU. For this,
please refer to:

  adhere to IO reservation disablement:
    https://bugzilla.redhat.com/show_bug.cgi?id=1434740
    (depends on BZ#1344299 above)

  adhere to MMIO reservation sizing:
    https://bugzilla.redhat.com/show_bug.cgi?id=1434747
    (depends on BZ#1437113 above)

We plan to work on these items in the following months (upstream first,
obviously).

--*--

Please read the following document in the QEMU source tree:

  docs/pcie.txt

It will give you the "grand vision" we have for PCI Express on the Q35
(x86) machine type, and most of that -- intentionally! -- applies to the
"virt" (aarch64) machine type as well.

See also the following QEMU commits (especially the second one):

  9ca019c1dd57 q35: Improve sample configuration files
  166d434685ed mach-virt: Provide sample configuration files

Thanks!
Laszlo

Re: ARM virt machine boots fail with 14 ioh3420

[Shannon Zhao]

Hi Laszlo,

Thanks a lot for your reply:)

On 2017/4/14 1:09, Laszlo Ersek wrote:
> Adding Andrea, Ard, Drew and Marcel; and the main qemu list
> 
> On 04/13/17 09:37, Shannon Zhao wrote:
>> Hi,
>>
>> I'm testing the PCIe devices hotplug for ARM virt machine and using
>> ioh3420 as root port. I found that below command line could work.
>>
>> qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
>> QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
>> ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
>> ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
>> ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
>> ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
>> ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
>> ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
>> ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
>> ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
>> ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
>> ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
>> ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
>> ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
>> ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
>> i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
>> pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
>> usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
>> virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
>> -drive
>> file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
>> -device
>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>> -netdev tap,id=hostnet1,vhost=on -device
>> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet2,vhost=on -device
>> virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet3,vhost=on -device
>> virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet4,vhost=on -device
>> virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet5,vhost=on -device
>> virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet6,vhost=on -device
>> virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet7,vhost=on -device
>> virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet8,vhost=on -device
>> virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet9,vhost=on -device
>> virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet10,vhost=on -device
>> virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet11,vhost=on -device
>> virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
>> -netdev tap,id=hostnet12,vhost=on -device
>> virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
>> -nographic
>>
>> But if I add one more ioh3420 device by appending above command with
>> "-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
>> the guest can't boot. It seems that the firmware doesn't recognize the
>> PCIe devices and print "Connect: PciRoot(0x0): Not Found".
>>
>> I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any limitation
>> of the supported PCIe devices?
> 
> In one sentence: you are running out of (emulated) IO space.
> 
> Aarch64 does not have "IO space", but with QEMU, using the "virt"
> machine type, we emulate 64KB of IO space, mapped to a special MMIO
> range. This is available for PCI resource allocation, for such devices
> that have IO BARs (and for such PCI bridges that reserve IO space for
> hotplug purposes).
> 
> The ioh3420 PCI Express Root Port device represents such a bridge. Even
> if you plug a PCI Express device into it that has only MMIO BARs, the
> bridge still advertises IO support, and it causes the firmware (and/or
> Linux) to reserve 4KB of IO space. With ~15-16 such ports, you run out
> of the 64 KB IO aperture, and the resource assignment fails.
> 
So currently if we want to support more than ~15 virtio-net-pci devices,
they can't connect to root port. They should connect to pcie root bus
directly, right? But this will not support hot-plug/remove.

BTW, I think even though the qemu assign more than ~15 root port, I
think the firmware should enable the first 15 ports and continue to work
instead of failing with silence.

> The solution to this problem comes together from several parts:
> 
> (1) New, vendor-independent device models in QEMU, for PCI Express Root
> Ports and Downstream Ports, that (optionally) do not advertise any
> support at all for IO BARs. This is on Marcel's task list. Please refer to:
> 
>   generic port device model:
>     https://bugzilla.redhat.com/show_bug.cgi?id=1390316
> 
I see this is in upstream qemu.

>   optional disablement of IO space:
>     https://bugzilla.redhat.com/show_bug.cgi?id=1344299
> 
Marcel, what's the status of this feature?

> (2) A similar question exists for MMIO space. While there the issue
> isn't exactly "running out of available aperture", we still want to
> control the MMIO reservation size for root ports and downstream ports --
> different devices that are to be hot-plugged can require different
> reservations in advance. For this, please refer to:
> 
>   sizing of MMIO reservation:
>     https://bugzilla.redhat.com/show_bug.cgi?id=1437113
> 
> (3) In the firmware, platform code can control the aperture reservation
> of bridges, for hotplug purposes, while the generic PCI Bus driver
> enumerates devices and assigns resources. The driver that does this is
> 
>   OvmfPkg/PciHotPlugInitDxe
> 
> At the moment, this driver is only included in OVMF, not in ArmVirtQemu.
> Plus, it doesn't really do the right thing in OVMF either.
> 
> Once items (1) and (2) above are solved, I will update PciHotPlugInitDxe
> so that it adheres to the reservation sizes dictated by QEMU. For this,
> please refer to:
> 
>   adhere to IO reservation disablement:
>     https://bugzilla.redhat.com/show_bug.cgi?id=1434740
>     (depends on BZ#1344299 above)
> 
>   adhere to MMIO reservation sizing:
>     https://bugzilla.redhat.com/show_bug.cgi?id=1434747
>     (depends on BZ#1437113 above)
> 
> We plan to work on these items in the following months (upstream first,
> obviously).
> 
> --*--
> 
> Please read the following document in the QEMU source tree:
> 
>   docs/pcie.txt
> 
> It will give you the "grand vision" we have for PCI Express on the Q35
> (x86) machine type, and most of that -- intentionally! -- applies to the
> "virt" (aarch64) machine type as well.
> 
> See also the following QEMU commits (especially the second one):
> 
>   9ca019c1dd57 q35: Improve sample configuration files
>   166d434685ed mach-virt: Provide sample configuration files
> 
> Thanks!
> Laszlo
> 
> .
> 

-- 
Shannon

Re: ARM virt machine boots fail with 14 ioh3420

[Laszlo Ersek]

On 04/14/17 04:41, Shannon Zhao wrote:
> Hi Laszlo,
> 
> Thanks a lot for your reply:)
> 
> On 2017/4/14 1:09, Laszlo Ersek wrote:
>> Adding Andrea, Ard, Drew and Marcel; and the main qemu list
>>
>> On 04/13/17 09:37, Shannon Zhao wrote:
>>> Hi,
>>>
>>> I'm testing the PCIe devices hotplug for ARM virt machine and using
>>> ioh3420 as root port. I found that below command line could work.
>>>
>>> qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
>>> QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
>>> ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
>>> ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
>>> ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
>>> ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
>>> ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
>>> ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
>>> ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
>>> ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
>>> ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
>>> ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
>>> ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
>>> ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
>>> ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
>>> i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
>>> pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
>>> usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
>>> virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
>>> -drive
>>> file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
>>> -device
>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>>> -netdev tap,id=hostnet1,vhost=on -device
>>> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet2,vhost=on -device
>>> virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet3,vhost=on -device
>>> virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet4,vhost=on -device
>>> virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet5,vhost=on -device
>>> virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet6,vhost=on -device
>>> virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet7,vhost=on -device
>>> virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet8,vhost=on -device
>>> virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet9,vhost=on -device
>>> virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet10,vhost=on -device
>>> virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet11,vhost=on -device
>>> virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
>>> -netdev tap,id=hostnet12,vhost=on -device
>>> virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
>>> -nographic
>>>
>>> But if I add one more ioh3420 device by appending above command with
>>> "-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
>>> the guest can't boot. It seems that the firmware doesn't recognize the
>>> PCIe devices and print "Connect: PciRoot(0x0): Not Found".
>>>
>>> I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any limitation
>>> of the supported PCIe devices?
>>
>> In one sentence: you are running out of (emulated) IO space.
>>
>> Aarch64 does not have "IO space", but with QEMU, using the "virt"
>> machine type, we emulate 64KB of IO space, mapped to a special MMIO
>> range. This is available for PCI resource allocation, for such devices
>> that have IO BARs (and for such PCI bridges that reserve IO space for
>> hotplug purposes).
>>
>> The ioh3420 PCI Express Root Port device represents such a bridge. Even
>> if you plug a PCI Express device into it that has only MMIO BARs, the
>> bridge still advertises IO support, and it causes the firmware (and/or
>> Linux) to reserve 4KB of IO space. With ~15-16 such ports, you run out
>> of the 64 KB IO aperture, and the resource assignment fails.
>>
> So currently if we want to support more than ~15 virtio-net-pci devices,
> they can't connect to root port.

Right.

> They should connect to pcie root bus
> directly, right?

Correct, that will make them integrated endpoints, and no PCI Express
ports will be necessary (hence no IO space will be wasted).

> But this will not support hot-plug/remove.

Correct.

> 
> BTW, I think even though the qemu assign more than ~15 root port, I
> think the firmware should enable the first 15 ports and continue to work
> instead of failing with silence.

PCI device enumeration & resource assignment are implemented in
"MdeModulePkg/Bus/Pci/PciBusDxe". It is a generic edk2 driver that is
built into OVMF, ArmVirtQemu, and (supposedly) most physical platform
firmware, without any changes. If you have improvements in mind, please
submit a patch for that driver.

> 
>> The solution to this problem comes together from several parts:
>>
>> (1) New, vendor-independent device models in QEMU, for PCI Express Root
>> Ports and Downstream Ports, that (optionally) do not advertise any
>> support at all for IO BARs. This is on Marcel's task list. Please refer to:
>>
>>   generic port device model:
>>     https://bugzilla.redhat.com/show_bug.cgi?id=1390316
>>
> I see this is in upstream qemu.

Yes. All of the pertaining work will be implemented upstream first.

> 
>>   optional disablement of IO space:
>>     https://bugzilla.redhat.com/show_bug.cgi?id=1344299
>>
> Marcel, what's the status of this feature?

(I think Marcel plans to answer your question, but AIUI he too might
have a bit of post-vacation email backlog to flush.)

Thanks
Laszlo

Re: ARM virt machine boots fail with 14 ioh3420

[Marcel Apfelbaum]

On 04/24/2017 01:02 PM, Laszlo Ersek wrote:
> On 04/14/17 04:41, Shannon Zhao wrote:
>> Hi Laszlo,
>>
>> Thanks a lot for your reply:)
>>
>> On 2017/4/14 1:09, Laszlo Ersek wrote:
>>> Adding Andrea, Ard, Drew and Marcel; and the main qemu list
>>>
>>> On 04/13/17 09:37, Shannon Zhao wrote:
>>>> Hi,
>>>>
>>>> I'm testing the PCIe devices hotplug for ARM virt machine and using
>>>> ioh3420 as root port. I found that below command line could work.
>>>>
>>>> qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
>>>> QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
>>>> ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
>>>> ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
>>>> ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
>>>> ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
>>>> ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
>>>> ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
>>>> ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
>>>> ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
>>>> ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
>>>> ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
>>>> ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
>>>> ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
>>>> ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
>>>> i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
>>>> pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
>>>> usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
>>>> virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -drive
>>>> file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
>>>> -device
>>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>>>> -netdev tap,id=hostnet1,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet2,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet3,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet4,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet5,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet6,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet7,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet8,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet9,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet10,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet11,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -netdev tap,id=hostnet12,vhost=on -device
>>>> virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
>>>> -nographic
>>>>
>>>> But if I add one more ioh3420 device by appending above command with
>>>> "-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
>>>> the guest can't boot. It seems that the firmware doesn't recognize the
>>>> PCIe devices and print "Connect: PciRoot(0x0): Not Found".
>>>>
>>>> I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any limitation
>>>> of the supported PCIe devices?
>>>
>>> In one sentence: you are running out of (emulated) IO space.
>>>
>>> Aarch64 does not have "IO space", but with QEMU, using the "virt"
>>> machine type, we emulate 64KB of IO space, mapped to a special MMIO
>>> range. This is available for PCI resource allocation, for such devices
>>> that have IO BARs (and for such PCI bridges that reserve IO space for
>>> hotplug purposes).
>>>
>>> The ioh3420 PCI Express Root Port device represents such a bridge. Even
>>> if you plug a PCI Express device into it that has only MMIO BARs, the
>>> bridge still advertises IO support, and it causes the firmware (and/or
>>> Linux) to reserve 4KB of IO space. With ~15-16 such ports, you run out
>>> of the 64 KB IO aperture, and the resource assignment fails.
>>>
>> So currently if we want to support more than ~15 virtio-net-pci devices,
>> they can't connect to root port.
>
> Right.
>
>> They should connect to pcie root bus
>> directly, right?
>
> Correct, that will make them integrated endpoints, and no PCI Express
> ports will be necessary (hence no IO space will be wasted).
>
>> But this will not support hot-plug/remove.
>
> Correct.
>
>>
>> BTW, I think even though the qemu assign more than ~15 root port, I
>> think the firmware should enable the first 15 ports and continue to work
>> instead of failing with silence.
>
> PCI device enumeration & resource assignment are implemented in
> "MdeModulePkg/Bus/Pci/PciBusDxe". It is a generic edk2 driver that is
> built into OVMF, ArmVirtQemu, and (supposedly) most physical platform
> firmware, without any changes. If you have improvements in mind, please
> submit a patch for that driver.
>
>>
>>> The solution to this problem comes together from several parts:
>>>
>>> (1) New, vendor-independent device models in QEMU, for PCI Express Root
>>> Ports and Downstream Ports, that (optionally) do not advertise any
>>> support at all for IO BARs. This is on Marcel's task list. Please refer to:
>>>
>>>   generic port device model:
>>>     https://bugzilla.redhat.com/show_bug.cgi?id=1390316
>>>
>> I see this is in upstream qemu.
>
> Yes. All of the pertaining work will be implemented upstream first.
>
>>
>>>   optional disablement of IO space:
>>>     https://bugzilla.redhat.com/show_bug.cgi?id=1344299
>>>
>> Marcel, what's the status of this feature?
>
> (I think Marcel plans to answer your question, but AIUI he too might
> have a bit of post-vacation email backlog to flush.)
>

Hi,

Sorry for the delay, I am working on it, I do have some patches that should
work, but they don't... I am checking the possibility that Firmwares/OSes do
not really check if the PCI bridge actually implements IO ports forwarding
and assumes it does instead.

I really hope I have a bug somewhere, I will update when I'll know more.

Thanks,
Marcel


> Thanks
> Laszlo
>

Re: ARM virt machine boots fail with 14 ioh3420

[Shannon Zhao]

On 2017/4/24 18:16, Marcel Apfelbaum wrote:
> On 04/24/2017 01:02 PM, Laszlo Ersek wrote:
>> On 04/14/17 04:41, Shannon Zhao wrote:
>>> Hi Laszlo,
>>>
>>> Thanks a lot for your reply:)
>>>
>>> On 2017/4/14 1:09, Laszlo Ersek wrote:
>>>> Adding Andrea, Ard, Drew and Marcel; and the main qemu list
>>>>
>>>> On 04/13/17 09:37, Shannon Zhao wrote:
>>>>> Hi,
>>>>>
>>>>> I'm testing the PCIe devices hotplug for ARM virt machine and using
>>>>> ioh3420 as root port. I found that below command line could work.
>>>>>
>>>>> qemu-system-aarch64 -machine virt,accel=kvm,usb=off -cpu host -bios
>>>>> QEMU_EFI.fd -m 12288 -smp 8,sockets=8,cores=1,threads=1  -device
>>>>> ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,addr=0x1 -device
>>>>> ioh3420,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x2 -device
>>>>> ioh3420,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x3 -device
>>>>> ioh3420,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x4 -device
>>>>> ioh3420,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x5 -device
>>>>> ioh3420,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x6 -device
>>>>> ioh3420,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 -device
>>>>> ioh3420,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x8 -device
>>>>> ioh3420,port=0x10,chassis=9,id=pci.9,bus=pcie.0,addr=0x9 -device
>>>>> ioh3420,port=0x11,chassis=10,id=pci.10,bus=pcie.0,addr=0xa -device
>>>>> ioh3420,port=0x12,chassis=11,id=pci.11,bus=pcie.0,addr=0xb -device
>>>>> ioh3420,port=0x13,chassis=12,id=pci.12,bus=pcie.0,addr=0xc -device
>>>>> ioh3420,port=0x14,chassis=13,id=pci.13,bus=pcie.0,addr=0xd -device
>>>>> i82801b11-bridge,id=pci.17,bus=pcie.0,addr=0x11 -device
>>>>> pci-bridge,chassis_nr=18,id=pci.18,bus=pci.17,addr=0x0 -device
>>>>> usb-ehci,id=usb,bus=pci.18,addr=0x1 -device
>>>>> virtio-scsi-pci,id=scsi0,bus=pci.1,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -drive
>>>>> file=/mnt/sdb/guest.raw,format=raw,if=none,id=drive-scsi0-0-0-0,cache=none,aio=native
>>>>>
>>>>> -device
>>>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>>>>>
>>>>> -netdev tap,id=hostnet1,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:16:3e:2b:cc:e1,bus=pci.2,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet2,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet2,id=net2,mac=00:16:3e:22:29:80,bus=pci.3,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet3,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet3,id=net3,mac=00:16:3e:28:07:9a,bus=pci.4,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet4,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet4,id=net4,mac=00:16:3e:3d:cd:b6,bus=pci.5,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet5,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet5,id=net5,mac=00:16:3e:64:9f:b0,bus=pci.6,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet6,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet6,id=net6,mac=00:16:3e:33:5b:d3,bus=pci.7,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet7,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet7,id=net7,mac=00:16:3e:39:7c:df,bus=pci.8,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet8,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet8,id=net8,mac=00:16:3e:0a:c1:4e,bus=pci.9,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet9,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet9,id=net9,mac=00:16:3e:0a:58:a6,bus=pci.10,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet10,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet10,id=net10,mac=00:16:3e:35:b5:80,bus=pci.11,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet11,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet11,id=net11,mac=00:16:3e:4d:b5:bb,bus=pci.12,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -netdev tap,id=hostnet12,vhost=on -device
>>>>> virtio-net-pci,netdev=hostnet12,id=net12,mac=00:16:3e:3b:69:e9,bus=pci.13,addr=0x0,disable-legacy=on,disable-modern=off
>>>>>
>>>>> -nographic
>>>>>
>>>>> But if I add one more ioh3420 device by appending above command with
>>>>> "-device ioh3420,port=0x15,chassis=14,id=pci.14,bus=pcie.0,addr=0xe",
>>>>> the guest can't boot. It seems that the firmware doesn't recognize the
>>>>> PCIe devices and print "Connect: PciRoot(0x0): Not Found".
>>>>>
>>>>> I'm using QEMU 2.8.1 and edk2 at commit 36a0d5c. Is there any
>>>>> limitation
>>>>> of the supported PCIe devices?
>>>>
>>>> In one sentence: you are running out of (emulated) IO space.
>>>>
>>>> Aarch64 does not have "IO space", but with QEMU, using the "virt"
>>>> machine type, we emulate 64KB of IO space, mapped to a special MMIO
>>>> range. This is available for PCI resource allocation, for such devices
>>>> that have IO BARs (and for such PCI bridges that reserve IO space for
>>>> hotplug purposes).
>>>>
>>>> The ioh3420 PCI Express Root Port device represents such a bridge. Even
>>>> if you plug a PCI Express device into it that has only MMIO BARs, the
>>>> bridge still advertises IO support, and it causes the firmware (and/or
>>>> Linux) to reserve 4KB of IO space. With ~15-16 such ports, you run out
>>>> of the 64 KB IO aperture, and the resource assignment fails.
>>>>
>>> So currently if we want to support more than ~15 virtio-net-pci devices,
>>> they can't connect to root port.
>>
>> Right.
>>
>>> They should connect to pcie root bus
>>> directly, right?
>>
>> Correct, that will make them integrated endpoints, and no PCI Express
>> ports will be necessary (hence no IO space will be wasted).
>>
>>> But this will not support hot-plug/remove.
>>
>> Correct.
>>
>>>
>>> BTW, I think even though the qemu assign more than ~15 root port, I
>>> think the firmware should enable the first 15 ports and continue to work
>>> instead of failing with silence.
>>
>> PCI device enumeration & resource assignment are implemented in
>> "MdeModulePkg/Bus/Pci/PciBusDxe". It is a generic edk2 driver that is
>> built into OVMF, ArmVirtQemu, and (supposedly) most physical platform
>> firmware, without any changes. If you have improvements in mind, please
>> submit a patch for that driver.
>>
>>>
>>>> The solution to this problem comes together from several parts:
>>>>
>>>> (1) New, vendor-independent device models in QEMU, for PCI Express Root
>>>> Ports and Downstream Ports, that (optionally) do not advertise any
>>>> support at all for IO BARs. This is on Marcel's task list. Please
>>>> refer to:
>>>>
>>>>   generic port device model:
>>>>     https://bugzilla.redhat.com/show_bug.cgi?id=1390316
>>>>
>>> I see this is in upstream qemu.
>>
>> Yes. All of the pertaining work will be implemented upstream first.
>>
>>>
>>>>   optional disablement of IO space:
>>>>     https://bugzilla.redhat.com/show_bug.cgi?id=1344299
>>>>
>>> Marcel, what's the status of this feature?
>>
>> (I think Marcel plans to answer your question, but AIUI he too might
>> have a bit of post-vacation email backlog to flush.)
>>
> 
> Hi,
> 
> Sorry for the delay, I am working on it, I do have some patches that should
> work, but they don't... I am checking the possibility that
> Firmwares/OSes do
> not really check if the PCI bridge actually implements IO ports forwarding
> and assumes it does instead.
> 
> I really hope I have a bug somewhere, I will update when I'll know more.
Ok, thanks in advance.

-- 
Shannon