Re: Question about hotplugging NIC devices to an empty pci-bridge

["Zhoujian (jay)" <jianjay.zhou@huawei.com>]

> -----Original Message-----
> From: Yao, Jiewen [mailto:jiewen.yao@intel.com]
> Sent: Friday, December 21, 2018 1:28 PM
> To: Zhoujian (jay) <jianjay.zhou@huawei.com>; edk2-devel@lists.01.org;
> lersek@redhat.com
> Cc: Huangweidong (C) <weidong.huang@huawei.com>; liujunjie (A)
> <liujunjie23@huawei.com>; wangxin (U) <wangxinxin.wang@huawei.com>; wujing (O)
> <wujing42@huawei.com>; dengkai (A) <dengkai1@huawei.com>
> Subject: RE: Question about hotplugging NIC devices to an empty pci-bridge
> 
> You need have a PciHotPlug driver to produce the
> EFI_PCI_HOT_PLUG_INIT_PROTOCOL
> 
> One example:
> https://github.com/tianocore/edk2/tree/master/OvmfPkg/PciHotPlugInitDxe
> Laszlo added it. He may provide comment on how to use it.
> 
> Another example:
> https://github.com/tianocore/edk2-platforms/tree/devel-
> MinPlatform/Platform/Intel/KabylakeOpenBoardPkg/Features/PciHotPlug
> This is to add Thunderbolt support in Kabylake platform.

I've checked the dsc, and confirmed that the OVMF.fd already had the PciHotPlug driver.
Then I found the resource info through the debug log like below:

InitRootBridge: populated root bus 0, with room for 255 subordinate bus(es)
RootBridge: PciRoot(0x0)
  Support/Attr: 70069 / 70069
    DmaAbove4G: No
NoExtConfSpace: Yes
     AllocAttr: 3 (CombineMemPMem Mem64Decode)
           Bus: 0 - FF Translation=0
            Io: C000 - FFFF Translation=0
           Mem: C0000000 - FBFFFFFF Translation=0
    MemAbove4G: 41800000000 - 41FFFFFFFFF Translation=0
          PMem: FFFFFFFFFFFFFFFF - 0 Translation=0
   PMemAbove4G: FFFFFFFFFFFFFFFF - 0 Translation=0

In the OvmfPkg/PlatformPei/Platform.c, the function MemMapInitialization sets the PciIoBase=0xC000 and PciIoSize=0x4000(On Q35, the PciIoBase=0x6000 and PciIoSize=0xA000).

So my question are:
1)Why the default value of PciIoBase is 0xC000, each pci-bridges needs 0x0fff IO window, which means only 4 pci-bridges can be reserved?
2)If I set the PciIoBase=0x1000, PciIoSize=0xA000 and start a vm with 8 empty pci-bridges, hotpluging a virtual nic to the pci-bridge, the problem is disappearing.
  But will this cause any side effects?
  
[    4.552337] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    4.552341] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    4.552343] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    4.552344] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
[    4.552346] pci_bus 0000:00: resource 8 [mem 0x41800000000-0x418001270ff window]
[    4.552348] pci_bus 0000:01: resource 0 [io  0x8000-0x8fff]
[    4.552350] pci_bus 0000:01: resource 1 [mem 0xc2e00000-0xc2ffffff]
[    4.552351] pci_bus 0000:02: resource 0 [io  0x7000-0x7fff]
[    4.552353] pci_bus 0000:02: resource 1 [mem 0xc2c00000-0xc2dfffff]
[    4.552354] pci_bus 0000:02: resource 2 [mem 0x41800000000-0x418000fffff 64bit pref]
[    4.552356] pci_bus 0000:03: resource 0 [io  0x6000-0x6fff]
[    4.552357] pci_bus 0000:03: resource 1 [mem 0xc2a00000-0xc2bfffff]
[    4.552359] pci_bus 0000:04: resource 0 [io  0x5000-0x5fff]
[    4.552360] pci_bus 0000:04: resource 1 [mem 0xc2800000-0xc29fffff]
[    4.552362] pci_bus 0000:05: resource 0 [io  0x4000-0x4fff]
[    4.552363] pci_bus 0000:05: resource 1 [mem 0xc2600000-0xc27fffff]
[    4.552364] pci_bus 0000:06: resource 0 [io  0x3000-0x3fff]
[    4.552366] pci_bus 0000:06: resource 1 [mem 0xc2400000-0xc25fffff]
[    4.552368] pci_bus 0000:07: resource 0 [io  0x2000-0x2fff]
[    4.552369] pci_bus 0000:07: resource 1 [mem 0xc2200000-0xc23fffff]
[    4.552370] pci_bus 0000:08: resource 0 [io  0x1000-0x1fff]
[    4.552372] pci_bus 0000:08: resource 1 [mem 0xc2000000-0xc21fffff]


Regards,
Jay Zhou

> 
> Thank you
> Yao Jiewen
> 
> 
> > -----Original Message-----
> > From: Zhoujian (jay) [mailto:jianjay.zhou@huawei.com]
> > Sent: Friday, December 21, 2018 11:04 AM
> > To: Yao, Jiewen <jiewen.yao@intel.com>; edk2-devel@lists.01.org;
> > lersek@redhat.com
> > Cc: Huangweidong (C) <weidong.huang@huawei.com>; liujunjie (A)
> > <liujunjie23@huawei.com>; wangxin (U) <wangxinxin.wang@huawei.com>;
> > wujing (O) <wujing42@huawei.com>; dengkai (A) <dengkai1@huawei.com>
> > Subject: RE: Question about hotplugging NIC devices to an empty
> > pci-bridge
> >
> > I've tried to set PcdPciBusHotplugDeviceSupport to be true in
> > MdeModulePkg.dec like below:
> > gEfiMdeModulePkgTokenSpaceGuid.PcdPciBusHotplugDeviceSupport|TRUE
> > |BOOLEAN|0x0001003d
> > But the problem still exists. Is there any steps I missed? Or some
> > infos need to populate to OVMF by Qemu?
> >
> > Could you give me more infos?
> >
> > Thanks,
> > Jay Zhou
> >
> > > -----Original Message-----
> > > From: Yao, Jiewen [mailto:jiewen.yao@intel.com]
> > > Sent: Thursday, December 20, 2018 8:09 PM
> > > To: Zhoujian (jay) <jianjay.zhou@huawei.com>;
> > > edk2-devel@lists.01.org
> > > Cc: Huangweidong (C) <weidong.huang@huawei.com>; liujunjie (A)
> > > <liujunjie23@huawei.com>; wangxin (U)
> > <wangxinxin.wang@huawei.com>; wujing (O)
> > > <wujing42@huawei.com>; dengkai (A) <dengkai1@huawei.com>
> > > Subject: RE: Question about hotplugging NIC devices to an empty
> > pci-bridge
> > >
> > > Maybe you can use EFI_PCI_HOT_PLUG_INIT_PROTOCOL to reserve some
> > resource.
> > >
> > > See MdePkg\Include\Protocol\PciHotPlugInit.h
> > >
> > > Thank you
> > > Yao Jiewen
> > >
> > > > -----Original Message-----
> > > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On
> > > > Behalf
> > Of
> > > > Zhoujian (jay)
> > > > Sent: Thursday, December 20, 2018 7:34 PM
> > > > To: edk2-devel@lists.01.org
> > > > Cc: Huangweidong (C) <weidong.huang@huawei.com>; liujunjie (A)
> > > > <liujunjie23@huawei.com>; wangxin (U)
> > <wangxinxin.wang@huawei.com>;
> > > > wujing (O) <wujing42@huawei.com>; dengkai (A)
> > <dengkai1@huawei.com>
> > > > Subject: [edk2] Question about hotplugging NIC devices to an empty
> > > > pci-bridge
> > > >
> > > > Hi all,
> > > >
> > > > The issue occurs when I started a virtual machine in UEFI way by
> > > > libvirt on qemu-kvm platform, the vm is configured with 8
> > > > pci-bridges on root bus0. I hotplug a device like virtual nic to
> > > > an empty pci-bridge which has no device connected. Login the vm, I
> > > > can see the device by "lspci"", but it didn't show by "ifconfig
> > > > -a". Dmesg shows like
> > > below:
> > > > pci 0000:04:01.0: BAR 0: no space for [mem size 0x00010000 64bit
> > > > pref] pci
> > > > 0000:04:01.0: BAR 0: failed to assign [mem size 0x00010000 64bit
> > > > pref] pci
> > > > 0000:04:01.0: BAR 3: no space for [mem size 0x00004000 64bit pref]
> > > > pci
> > > > 0000:04:01.0: BAR 3: failed to assign [mem size 0x00004000 64bit
> > > > pref]
> > > >
> > > > Reboot the vm, everything turns back to normal and I can see the
> > > > new hotplugged nic by "ifconfig -a".
> > > >
> > > > Use the OVMF compiling from latest edk2 source code, the same
> > problem
> > > > arises.
> > > >
> > > > So, my questions are:
> > > > 1) the generic PCI bus driver in edk2 does not allocate IO and/or
> > > > MMIO for a bridge if there is no device behind the bridge that
> > > > consume that kind of resource?
> > > > 2) What's the purpose of this strategy?
> > > > 3) Why don't allocate resource to all bridges like seabios?
> > > > 4) Is there any switch for me to turn off this constraint so that
> > > > every pci-bridge including empty ones can be assigned IO and
> > > > memory
> > window?
> > > > Otherwise, each time I hotplug a device to empty pci-bridge, a
> > > > reboot operation should be implemented to use the device?
> > > >
> > > > Any help will be appreciated, Thanks!
> > > > _______________________________________________
> > > > edk2-devel mailing list
> > > > edk2-devel@lists.01.org
> > > > https://lists.01.org/mailman/listinfo/edk2-devel