SR-IOV setup in edk2

SR-IOV setup in edk2

[Jon Nettleton]

I am looking for some example code, or direction in how SR-IOV
functions are expected to be picked up by the platform pcie host
library so it can setup required bits like LUTs.  Currently the
HostLib is getting the gEfiPciIoProtocolGuid event and then setting
the controller up based on the BDF.  I see that PciScanBus is
detecting the VF's and calling PciAllocateBusNumber().

PCI-IOV ScanBus - SubBusNumber - 0x2
PciBus: Discovered PPB @ [00|00|00]

However my setup function is only ever triggered with the real device
BDF's...Is it up to me to check for the VFs if SR-IOV is enabled and
set them up manually?  Is there another event I should be listening to
that will be triggered for VF's vs PF's?

Any pointers would be very helpful.

-Jon

Re: [edk2-devel] SR-IOV setup in edk2

[Laszlo Ersek]

On 03/21/21 13:46, Jon Nettleton wrote:
> I am looking for some example code, or direction in how SR-IOV
> functions are expected to be picked up by the platform pcie host
> library so it can setup required bits like LUTs.  Currently the
> HostLib is getting the gEfiPciIoProtocolGuid event and then setting
> the controller up based on the BDF.  I see that PciScanBus is
> detecting the VF's and calling PciAllocateBusNumber().
> 
> PCI-IOV ScanBus - SubBusNumber - 0x2
> PciBus: Discovered PPB @ [00|00|00]
> 
> However my setup function is only ever triggered with the real device
> BDF's...

What do you mean by "setup function"?

If you have a UEFI driver that follows the UEFI driver model, i.e. it
installs at least one instance of the Driver Binding protocol, then it's
up to platform BDS to call ConnectController() on those devices that
should be connected per platform policy.

> Is it up to me to check for the VFs if SR-IOV is enabled and
> set them up manually?  Is there another event I should be listening to
> that will be triggered for VF's vs PF's?

My guess is that platform BDS does not try to connect the PciIo
instances that stand for VFs to any drivers.

In the UEFI shell, try running

  dh -d -v -p PciIo

Pick a handle ID that appears to stand for a VF, then run

  connect <that handle>

Thanks
Laszlo

> 
> Any pointers would be very helpful.
> 
> -Jon
> 
> 
> 
> 
> 

Re: [edk2-devel] SR-IOV setup in edk2

[Jon Nettleton]

On Mon, Mar 22, 2021 at 8:04 PM Laszlo Ersek <lersek@redhat.com> wrote:
>
> On 03/21/21 13:46, Jon Nettleton wrote:
> > I am looking for some example code, or direction in how SR-IOV
> > functions are expected to be picked up by the platform pcie host
> > library so it can setup required bits like LUTs.  Currently the
> > HostLib is getting the gEfiPciIoProtocolGuid event and then setting
> > the controller up based on the BDF.  I see that PciScanBus is
> > detecting the VF's and calling PciAllocateBusNumber().
> >
> > PCI-IOV ScanBus - SubBusNumber - 0x2
> > PciBus: Discovered PPB @ [00|00|00]
> >
> > However my setup function is only ever triggered with the real device
> > BDF's...
>
> What do you mean by "setup function"?

Well currently we followed the methodology of most the other
PCI Host Bus drivers and as I said above get the Io event and then
finish the completion of the device setup.  This includes setting up
the Luts, adding the device to the SMMU tables, and a few other
operations.

The VFs for SR-IOV are different as they are treated as hot
pluggable devices in both edk2 and linux, however there is still
setup that needs to be done to the host controller to support the
VFs on the bus.  The PciScanBus is obviously partially wired to
do some enumeration for PCI-IOV but I am unclear where the
platform code should hook into this for the additional proving
and setup.

I have now implemented a driver connected to gEfiPciPlatformProtocolGuid
and I can get events passed in for the platform setup hooks, is this where
we should be doing the bus specific setup, rather than catching
gEfiPciIoProtocolGuid?

>
> If you have a UEFI driver that follows the UEFI driver model, i.e. it
> installs at least one instance of the Driver Binding protocol, then it's
> up to platform BDS to call ConnectController() on those devices that
> should be connected per platform policy.
>
> > Is it up to me to check for the VFs if SR-IOV is enabled and
> > set them up manually?  Is there another event I should be listening to
> > that will be triggered for VF's vs PF's?
>
> My guess is that platform BDS does not try to connect the PciIo
> instances that stand for VFs to any drivers.
>
> In the UEFI shell, try running
>
>   dh -d -v -p PciIo
>
> Pick a handle ID that appears to stand for a VF, then run

No handles are created for the VFs as they aren't setup yet.

Thanks,
Jon

>
>   connect <that handle>
>
> Thanks
> Laszlo
>
> >
> > Any pointers would be very helpful.
> >
> > -Jon
> >
> >
> > 
> >
> >
>

Re: [edk2-devel] SR-IOV setup in edk2

[Laszlo Ersek]

Hi Jon.

On 03/25/21 09:02, Jon Nettleton wrote:

> I have now implemented a driver connected to gEfiPciPlatformProtocolGuid
> and I can get events passed in for the platform setup hooks, is this where
> we should be doing the bus specific setup, rather than catching
> gEfiPciIoProtocolGuid?

Sorry, the topic you are working on is over my head.

Catching gEfiPciIoProtocolGuid (with a protocol notify registration) is
definitely wrong, from a UEFI driver model perspective.

EFI_PCI_PLATFORM_PROTOCOL *could* be what you are afrer. I recommend
reading through Chapter 11, "PCI Platform", in Volume 5 of the Platform
Init spec (latest release is, to my knowledge, v1.7). Chapter 10 could
be relevant too ("PCI Host Bridge").

I think the PCI maintainers in edk2 could be of more help than I; please
see Hao and Ray in Maintainers.txt.

Thanks
Laszlo