PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

[Eric Song]

Hi, experts
         If there is 2 root bridges under 1 host bridge on the PCI hierarchy, whether EDK2 Pci bus driver can support it ? the answer is YES, but some confused things in it.

1)       On BDS phase, ConnectController is called for every root bridge, so Pci bus start() will be called for every root bridge.

  VisitAllInstancesOfProtocol (&gEfiPciRootBridgeIoProtocolGuid, ConnectRootBridge, NULL);

2)       In PCI bus driver, Pci start() will do enumeration on the entire host bridge

PciEnumerator()->PciHostBridgeEnumerator->”PciResAlloc->GetNextRootBridge()”

That means Do enumeration on entire host bridge for every one root bridge. So there are 2 enumerations on entire host bridge for my PCI hierarchy.
         What I confused is why do 2 enumerations on entire host bridge by every one root bridge. Since BDS connect controller by one root bridge , Pci bus driver should enumerate corresponding root bridge, But it do the entire host bridge which is the parent of the root bridge.
Or BDS should connect controller by one host bridge, then it will match the pci bus driver ‘s enumeration policy.
In a word, I think it is illogical for current EDKII code.

         Well, the current code execution will do 2 enumerations on host bridge for 2 root bridges under 1 host bridge. Even though 2rd enumeration would be not full enumeration, but do PciEnumeratorLight(), I think it is still not necessary. Whether there is any side effect ?  In another word, current PCI bus driver enumeration can’t be Suitable for 2 root bridges under 1 host bridge or not? It indeed match the PCI hierarchy of 2 host bridges and 1 root bridge under 1 host bridge.

Thanks
Eric



保密声明：
本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.

Re: PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

[Laszlo Ersek]

Adding Ray

On 08/17/17 06:50, Eric Song wrote:
> Hi, experts
> If there is 2 root bridges under 1 host bridge on the PCI hierarchy,
> whether EDK2 Pci bus driver can support it ? the answer is YES, but
> some confused things in it.
>
> 1)       On BDS phase, ConnectController is called for every root
> bridge, so Pci bus start() will be called for every root bridge.
>
>   VisitAllInstancesOfProtocol (&gEfiPciRootBridgeIoProtocolGuid, ConnectRootBridge, NULL);
>
> 2)       In PCI bus driver, Pci start() will do enumeration on the
> entire host bridge
>
> PciEnumerator()->PciHostBridgeEnumerator->\x1dPciResAlloc->GetNextRootBridge()
>
> That means Do enumeration on entire host bridge for every one root
> bridge. So there are 2 enumerations on entire host bridge for my PCI
> hierarchy.
> What I confused is why do 2 enumerations on entire host bridge by
> every one root bridge. Since BDS connect controller by one root bridge
> , Pci bus driver should enumerate corresponding root bridge, But it do
> the entire host bridge which is the parent of the root bridge. Or BDS
> should connect controller by one host bridge, then it will match the
> pci bus driver \x18s enumeration policy. In a word, I think it is
> illogical for current EDKII code.
>
> Well, the current code execution will do 2 enumerations on host bridge
> for 2 root bridges under 1 host bridge. Even though 2rd enumeration
> would be not full enumeration, but do PciEnumeratorLight(), I think it
> is still not necessary. Whether there is any side effect ? In another
> word, current PCI bus driver enumeration can\x19t be Suitable for 2
> root bridges under 1 host bridge or not? It indeed match the PCI
> hierarchy of 2 host bridges and 1 root bridge under 1 host bridge.

While I don't know the internals of this host bridge-level enumeration,
it is worth looking at:

(a) in what context PciEnumerator() is called, and

(b) how PciEnumerator() works across multiple calls.

Regarding (a): PciEnumerator() is called from
PciBusDriverBindingStart(), that is, when the PciBusDxe driver is
binding a root brirdge. This is the code:

  //
  // Enumerate the entire host bridge
  // After enumeration, a database that records all the device information will be created
  //
  //
  Status = PciEnumerator (Controller);

Regarding (b): the PciEnumerator() function depends on a global variable
called "gFullEnumeration". This is what we have at the beginning and at
the end of PciEnumerator(), respectively:

  //
  // If PCI bus has already done the full enumeration, never do it again
  //
  if (!gFullEnumeration) {
    return PciEnumeratorLight (Controller);
  }

...

  gFullEnumeration = FALSE;

  Status = gBS->InstallProtocolInterface (
                  &HostBridgeHandle,
                  &gEfiPciEnumerationCompleteProtocolGuid,
                  EFI_NATIVE_INTERFACE,
                  NULL
                  );

The above seems to imply that PciBusDxe needs host bridge-level
information to do its job. This information is apparently collected when
the first root bridge is bound. When binding further root bridges (under
the same host bridge), the same information collecting is not repeated;
PciEnumerator() branches to PciEnumeratorLight() instead.

Hope this helps,
Laszlo

答复: PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

[Eric Song]

Hi, Laszlo
        Many thanks for rapidly reply.
        Your analyze is very clear. But while the PciBusDxe driver is binding a root bridge, why the PciBusDxe driver do host bridge enumeration. It is illogical.
        Even though it maybe has no side effect, it is confused for users.

Thanks
Eric

-----邮件原件-----
发件人: Laszlo Ersek [mailto:lersek@redhat.com]
发送时间: 2017年8月17日 17:19
收件人: Eric Song; edk2-devel@lists.01.org
抄送: Tiger Liu; Ni, Ruiyu
主题: Re: [edk2] [EDK2] PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

Adding Ray

On 08/17/17 06:50, Eric Song wrote:
> Hi, experts
> If there is 2 root bridges under 1 host bridge on the PCI hierarchy,
> whether EDK2 Pci bus driver can support it ? the answer is YES, but
> some confused things in it.
>
> 1)       On BDS phase, ConnectController is called for every root
> bridge, so Pci bus start() will be called for every root bridge.
>
>   VisitAllInstancesOfProtocol (&gEfiPciRootBridgeIoProtocolGuid,
> ConnectRootBridge, NULL);
>
> 2)       In PCI bus driver, Pci start() will do enumeration on the
> entire host bridge
>
> PciEnumerator()->PciHostBridgeEnumerator->\x1dPciResAlloc->GetNextRootBri
> dge()
>
> That means Do enumeration on entire host bridge for every one root
> bridge. So there are 2 enumerations on entire host bridge for my PCI
> hierarchy.
> What I confused is why do 2 enumerations on entire host bridge by
> every one root bridge. Since BDS connect controller by one root bridge
> , Pci bus driver should enumerate corresponding root bridge, But it do
> the entire host bridge which is the parent of the root bridge. Or BDS
> should connect controller by one host bridge, then it will match the
> pci bus driver \x18s enumeration policy. In a word, I think it is
> illogical for current EDKII code.
>
> Well, the current code execution will do 2 enumerations on host bridge
> for 2 root bridges under 1 host bridge. Even though 2rd enumeration
> would be not full enumeration, but do PciEnumeratorLight(), I think it
> is still not necessary. Whether there is any side effect ? In another
> word, current PCI bus driver enumeration can\x19t be Suitable for 2 root
> bridges under 1 host bridge or not? It indeed match the PCI hierarchy
> of 2 host bridges and 1 root bridge under 1 host bridge.

While I don't know the internals of this host bridge-level enumeration, it is worth looking at:

(a) in what context PciEnumerator() is called, and

(b) how PciEnumerator() works across multiple calls.

Regarding (a): PciEnumerator() is called from PciBusDriverBindingStart(), that is, when the PciBusDxe driver is binding a root brirdge. This is the code:

  //
  // Enumerate the entire host bridge
  // After enumeration, a database that records all the device information will be created
  //
  //
  Status = PciEnumerator (Controller);

Regarding (b): the PciEnumerator() function depends on a global variable called "gFullEnumeration". This is what we have at the beginning and at the end of PciEnumerator(), respectively:

  //
  // If PCI bus has already done the full enumeration, never do it again
  //
  if (!gFullEnumeration) {
    return PciEnumeratorLight (Controller);
  }

...

  gFullEnumeration = FALSE;

  Status = gBS->InstallProtocolInterface (
                  &HostBridgeHandle,
                  &gEfiPciEnumerationCompleteProtocolGuid,
                  EFI_NATIVE_INTERFACE,
                  NULL
                  );

The above seems to imply that PciBusDxe needs host bridge-level information to do its job. This information is apparently collected when the first root bridge is bound. When binding further root bridges (under the same host bridge), the same information collecting is not repeated;
PciEnumerator() branches to PciEnumeratorLight() instead.

Hope this helps,
Laszlo


保密声明：
本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.

答复: PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

[Eric Song]

Hi, Ray
        Thank you very much for explanations.
        I agree with your summary, and I also have tested it.
        But I can't found Pci Io handle is created when a light enumeration.
        So the sequence is as follows:
        1) 1st root bridge handler will trigger Pci bus start() and do enumeration,
        PciEnumerator()->CreateRootBridge() and PciRootBridgeEnumerator , In CreateRootBridge(),it will initialize the PCI I/O instance structure for 1st root bridge.
                 -> CreateRootBridge() and PciRootBridgeEnumerator , In CreateRootBridge(),it will initialize the PCI I/O instance structure for 2rd root bridge.
    2) 2nd root bridge handler will trigger Pci bus start() and do enumeration,
        PciEnumerator()->PciEnumeratorLight().

        So, while 1), all things is done for 2 root bridges. And not populate PciIo for 2nd root bridge.
        Indeed, resource will be allocated based on the host bridge, so notify phase will do all root bridges allocation under host bridge.

Thanks
Eric


-----邮件原件-----
发件人: Ni, Ruiyu [mailto:ruiyu.ni@intel.com]
发送时间: 2017年8月17日 17:59
收件人: Eric Song; edk2-devel@lists.01.org
抄送: Tiger Liu
主题: RE: [edk2] [EDK2] PCI bus do 2 Enumeration when 2 root bridges under 1 host bridge

Eric,
Firstly, PciBus driver can support single-segment-multiple-rootbridge system or multiple-segment-multiple-rootbridge very well. (I agree the enumeration logic confuses you.)

Take a platform containing 2 root bridges for example, your understanding is correct. PciBus does the full enumeration (full means resource assignment is
done) for both root bridges.
But since PciBus driver is a driver model driver and it starts on a root bridge handle, so only the PciIo handles behind that root bridge is created.
The PciIo handles behind the other root bridge is not created until someone starts PciBus again using the other root bridge handle. And when this happens, only a light enumeration happens. Light means bus driver doesn't do the resource assignment but only populate the PciIo.

I guess the full enumeration + light enumeration confuses you.

You may have a question to ask: why not PciBus only does full enumeration on the specified root bridge passed into Start()? Instead of on all root bridges.

My understanding is it's a limitation of the PciHostBridge->NotifyPhase() defined in PI spec.
typedef
EFI_STATUS
(EFIAPI
*EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL_NOTIFY_PHASE)
(
IN EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL *This, IN EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PHASE Phase );

You can see that this interface doesn't carry the RootBridgeHandle so it implies that all root bridges should transfer the state to the next at the same moment.

That's why when even only one root bridge is asked to start, bus driver assigns resources for all root bridges.


In summary, we do have verified the code in single-segment-multiple-rootbridge system and multiple-segment-multiple-rootbridge system.

The driver should work as expected.

Thanks/Ray

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Eric Song
> Sent: Thursday, August 17, 2017 12:50 PM
> To: edk2-devel@lists.01.org
> Cc: Tiger Liu <TigerLiu@zhaoxin.com>
> Subject: [edk2] [EDK2] PCI bus do 2 Enumeration when 2 root bridges
> under
> 1 host bridge
>
> Hi, experts
>          If there is 2 root bridges under 1 host bridge on the PCI
> hierarchy, whether EDK2 Pci bus driver can support it ? the answer is
> YES, but some confused things in it.
>
> 1)       On BDS phase, ConnectController is called for every root bridge, so Pci
> bus start() will be called for every root bridge.
>
>   VisitAllInstancesOfProtocol (&gEfiPciRootBridgeIoProtocolGuid,
> ConnectRootBridge, NULL);
>
> 2)       In PCI bus driver, Pci start() will do enumeration on the entire host
> bridge
>
> PciEnumerator()->PciHostBridgeEnumerator->”PciResAlloc-
> >GetNextRootBridge()”
>
> That means Do enumeration on entire host bridge for every one root bridge.
> So there are 2 enumerations on entire host bridge for my PCI hierarchy.
>          What I confused is why do 2 enumerations on entire host
> bridge by every one root bridge. Since BDS connect controller by one
> root bridge , Pci bus driver should enumerate corresponding root
> bridge, But it do the entire host bridge which is the parent of the root bridge.
> Or BDS should connect controller by one host bridge, then it will
> match the pci bus driver ‘s enumeration policy.
> In a word, I think it is illogical for current EDKII code.
>
>          Well, the current code execution will do 2 enumerations on
> host bridge for 2 root bridges under 1 host bridge. Even though 2rd
> enumeration would be not full enumeration, but do
> PciEnumeratorLight(), I think it is still not necessary. Whether there
> is any side effect ?  In another word, current PCI bus driver
> enumeration can’t be Suitable for 2 root bridges under 1 host bridge
> or not? It indeed match the PCI hierarchy of 2 host bridges and 1 root bridge under 1 host bridge.
>
> Thanks
> Eric
>
>
>
> 保密声明：
> 本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其
> 内容做任何未经授权的查阅、使用、复制或转发。
> CONFIDENTIAL NOTE:
> This email contains confidential or legally privileged information and
> is for the sole use of its intended recipient. Any unauthorized
> review, use, copying or forwarding of this email or the content of this email is strictly prohibited.
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel


保密声明：
本邮件含有保密或专有信息，仅供指定收件人使用。严禁对本邮件或其内容做任何未经授权的查阅、使用、复制或转发。
CONFIDENTIAL NOTE:
This email contains confidential or legally privileged information and is for the sole use of its intended recipient. Any unauthorized review, use, copying or forwarding of this email or the content of this email is strictly prohibited.