* [RFC] MdeModulePkg/PciHostBridge: Add address translation support @ 2017-12-20 8:21 Heyi Guo 2017-12-20 9:13 ` Ard Biesheuvel 0 siblings, 1 reply; 16+ messages in thread From: Heyi Guo @ 2017-12-20 8:21 UTC (permalink / raw) To: linaro-uefi, edk2-devel Cc: Heyi Guo, Star Zeng, Eric Dong, Ruiyu Ni, Ard Biesheuvel, Jason Zhang PCIe on some ARM platforms requires address translation, not only for legacy IO access, but also for 32bit memory BAR access as well. There will be "Address Translation Unit" or something similar in PCI host bridges to translation CPU address to PCI address and vice versa. So we think it may be useful to add address translation support to the generic PCI host bridge driver. This RFC only contains one minor change to the definition of PciHostBridgeLib, and there certainly will be a lot of other changes to make it work, including: 1. Use CPU address for GCD space add and allocate operations, instead of PCI address; also IO space will be changed to memory space if translation exists. 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get translation of the corresponding aperture, add the translation to the input address, and then call CpuIo2 protocol; IO access will also be converted to memory access if IO translation exists. 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in the discriptor. If it makes sense, then I'll continue to prepare the formal patch. Any comments? Thanks, Gary (Heyi Guo) Cc: Star Zeng <star.zeng@intel.com> Cc: Eric Dong <eric.dong@intel.com> Cc: Ruiyu Ni <ruiyu.ni@intel.com> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> Cc: Jason Zhang <jason.zhang@linaro.org> --- MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + 1 file changed, 1 insertion(+) diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h index d42e9ec..b9e8c0f 100644 --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h @@ -22,6 +22,7 @@ typedef struct { UINT64 Base; UINT64 Limit; + UINT64 Translation; } PCI_ROOT_BRIDGE_APERTURE; typedef struct { -- 2.7.4 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-20 8:21 [RFC] MdeModulePkg/PciHostBridge: Add address translation support Heyi Guo @ 2017-12-20 9:13 ` Ard Biesheuvel 2017-12-20 15:17 ` gary guo 0 siblings, 1 reply; 16+ messages in thread From: Ard Biesheuvel @ 2017-12-20 9:13 UTC (permalink / raw) To: Heyi Guo Cc: linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Ruiyu Ni, Jason Zhang Hi Heyi, On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > PCIe on some ARM platforms requires address translation, not only for > legacy IO access, but also for 32bit memory BAR access as well. There > will be "Address Translation Unit" or something similar in PCI host > bridges to translation CPU address to PCI address and vice versa. So > we think it may be useful to add address translation support to the > generic PCI host bridge driver. > I agree. While unusual on a PC, it is quite common on other architectures to have more complex non 1:1 topologies, which currently require a forked PciHostBridgeDxe driver with local changes applied. > This RFC only contains one minor change to the definition of > PciHostBridgeLib, and there certainly will be a lot of other changes > to make it work, including: > > 1. Use CPU address for GCD space add and allocate operations, instead > of PCI address; also IO space will be changed to memory space if > translation exists. > For I/O space, the translation should simply be applied to the I/O range. I don't think it makes sense to use memory space here, given that it is specific to architectures that lack native port I/O. > 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get > translation of the corresponding aperture, add the translation to the > input address, and then call CpuIo2 protocol; IO access will also be > converted to memory access if IO translation exists. > Again, why is this necessary? A host bridge that implements a non 1:1 translation for port I/O ranges may be part of a system that has native port I/O, and so the translation should be based on that. > 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in > the discriptor. > Indeed. Note that this has been a source of much confusion lately, should we should discuss this carefully before spending time on an implementation. > If it makes sense, then I'll continue to prepare the formal patch. > > Any comments? > > Thanks, > > Gary (Heyi Guo) > > Cc: Star Zeng <star.zeng@intel.com> > Cc: Eric Dong <eric.dong@intel.com> > Cc: Ruiyu Ni <ruiyu.ni@intel.com> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > Cc: Jason Zhang <jason.zhang@linaro.org> > > --- > MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > index d42e9ec..b9e8c0f 100644 > --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h > +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > @@ -22,6 +22,7 @@ > typedef struct { > UINT64 Base; > UINT64 Limit; > + UINT64 Translation; > } PCI_ROOT_BRIDGE_APERTURE; > > typedef struct { > -- > 2.7.4 > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-20 9:13 ` Ard Biesheuvel @ 2017-12-20 15:17 ` gary guo 2017-12-20 15:26 ` Ard Biesheuvel 0 siblings, 1 reply; 16+ messages in thread From: gary guo @ 2017-12-20 15:17 UTC (permalink / raw) To: Ard Biesheuvel Cc: Heyi Guo, linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Ruiyu Ni, Jason Zhang On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: > Hi Heyi, > > On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > > PCIe on some ARM platforms requires address translation, not only for > > legacy IO access, but also for 32bit memory BAR access as well. There > > will be "Address Translation Unit" or something similar in PCI host > > bridges to translation CPU address to PCI address and vice versa. So > > we think it may be useful to add address translation support to the > > generic PCI host bridge driver. > > > > I agree. While unusual on a PC, it is quite common on other > architectures to have more complex non 1:1 topologies, which currently > require a forked PciHostBridgeDxe driver with local changes applied. > > > This RFC only contains one minor change to the definition of > > PciHostBridgeLib, and there certainly will be a lot of other changes > > to make it work, including: > > > > 1. Use CPU address for GCD space add and allocate operations, instead > > of PCI address; also IO space will be changed to memory space if > > translation exists. > > > > For I/O space, the translation should simply be applied to the I/O > range. I don't think it makes sense to use memory space here, given > that it is specific to architectures that lack native port I/O. > I made an assumption here that platforms supporting real port IO space do not need address translation, like IA32 and X64, and port IO translation implies the platform does not support real port IO space. Indeed the assumption is not so "generic", so I'll agree if you recommend to support IO to IO translation as well. But I still hope to have IO to memory translation support in PCI host bridge driver, rather than in CPU IO protocol, since the faked IO space might only be used for PCI host bridge and we may have overlapping IO ranges for each host bridge, which is compatible with PCIe specification and PCIe ACPI description. How about adding a flag to indicate whether port IO is translated to real port IO space or system memory space? Thanks and regards, Heyi > > 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get > > translation of the corresponding aperture, add the translation to the > > input address, and then call CpuIo2 protocol; IO access will also be > > converted to memory access if IO translation exists. > > > > Again, why is this necessary? A host bridge that implements a non 1:1 > translation for port I/O ranges may be part of a system that has > native port I/O, and so the translation should be based on that. > > > 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in > > the discriptor. > > > > Indeed. Note that this has been a source of much confusion lately, > should we should discuss this carefully before spending time on an > implementation. > > > If it makes sense, then I'll continue to prepare the formal patch. > > > > Any comments? > > > > Thanks, > > > > Gary (Heyi Guo) > > > > Cc: Star Zeng <star.zeng@intel.com> > > Cc: Eric Dong <eric.dong@intel.com> > > Cc: Ruiyu Ni <ruiyu.ni@intel.com> > > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > > Cc: Jason Zhang <jason.zhang@linaro.org> > > > > --- > > MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > > index d42e9ec..b9e8c0f 100644 > > --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h > > +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > > @@ -22,6 +22,7 @@ > > typedef struct { > > UINT64 Base; > > UINT64 Limit; > > + UINT64 Translation; > > } PCI_ROOT_BRIDGE_APERTURE; > > > > typedef struct { > > -- > > 2.7.4 > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-20 15:17 ` gary guo @ 2017-12-20 15:26 ` Ard Biesheuvel 2017-12-21 8:27 ` Guo Heyi 2017-12-21 9:43 ` Ni, Ruiyu 0 siblings, 2 replies; 16+ messages in thread From: Ard Biesheuvel @ 2017-12-20 15:26 UTC (permalink / raw) To: gary guo Cc: linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Ruiyu Ni, Jason Zhang On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: > On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >> Hi Heyi, >> >> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >> > PCIe on some ARM platforms requires address translation, not only for >> > legacy IO access, but also for 32bit memory BAR access as well. There >> > will be "Address Translation Unit" or something similar in PCI host >> > bridges to translation CPU address to PCI address and vice versa. So >> > we think it may be useful to add address translation support to the >> > generic PCI host bridge driver. >> > >> >> I agree. While unusual on a PC, it is quite common on other >> architectures to have more complex non 1:1 topologies, which currently >> require a forked PciHostBridgeDxe driver with local changes applied. >> >> > This RFC only contains one minor change to the definition of >> > PciHostBridgeLib, and there certainly will be a lot of other changes >> > to make it work, including: >> > >> > 1. Use CPU address for GCD space add and allocate operations, instead >> > of PCI address; also IO space will be changed to memory space if >> > translation exists. >> > >> >> For I/O space, the translation should simply be applied to the I/O >> range. I don't think it makes sense to use memory space here, given >> that it is specific to architectures that lack native port I/O. >> > > I made an assumption here that platforms supporting real port IO space > do not need address translation, like IA32 and X64, and port IO > translation implies the platform does not support real port IO space. > This may be a reasonable assumption. But I still think it is better not to encode any assumptions in the first place. > Indeed the assumption is not so "generic", so I'll agree if you > recommend to support IO to IO translation as well. But I still hope to > have IO to memory translation support in PCI host bridge driver, > rather than in CPU IO protocol, since the faked IO space might only be > used for PCI host bridge and we may have overlapping IO ranges for > each host bridge, which is compatible with PCIe specification and PCIe > ACPI description. > That is fine. Under UEFI, these will translate to non-overlapping I/O spaces in the CPU's view. Under the OS, this could be totally different. For example, RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff This is very similar to how MMIO translation works, and makes I/O devices behind the host bridges uniquely addressable for drivers. For our understanding, could you share the host bridge configuration that you are targetting? Thanks, Ard. > How about adding a flag to indicate whether port IO is translated to > real port IO space or system memory space? > > Thanks and regards, > > Heyi > >> > 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get >> > translation of the corresponding aperture, add the translation to the >> > input address, and then call CpuIo2 protocol; IO access will also be >> > converted to memory access if IO translation exists. >> > >> >> Again, why is this necessary? A host bridge that implements a non 1:1 >> translation for port I/O ranges may be part of a system that has >> native port I/O, and so the translation should be based on that. >> >> > 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in >> > the discriptor. >> > >> >> Indeed. Note that this has been a source of much confusion lately, >> should we should discuss this carefully before spending time on an >> implementation. >> >> > If it makes sense, then I'll continue to prepare the formal patch. >> > >> > Any comments? >> > >> > Thanks, >> > >> > Gary (Heyi Guo) >> > >> > Cc: Star Zeng <star.zeng@intel.com> >> > Cc: Eric Dong <eric.dong@intel.com> >> > Cc: Ruiyu Ni <ruiyu.ni@intel.com> >> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >> > Cc: Jason Zhang <jason.zhang@linaro.org> >> > >> > --- >> > MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + >> > 1 file changed, 1 insertion(+) >> > >> > diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >> > index d42e9ec..b9e8c0f 100644 >> > --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h >> > +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >> > @@ -22,6 +22,7 @@ >> > typedef struct { >> > UINT64 Base; >> > UINT64 Limit; >> > + UINT64 Translation; >> > } PCI_ROOT_BRIDGE_APERTURE; >> > >> > typedef struct { >> > -- >> > 2.7.4 >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-20 15:26 ` Ard Biesheuvel @ 2017-12-21 8:27 ` Guo Heyi 2017-12-21 8:32 ` Ard Biesheuvel 2017-12-21 9:43 ` Ni, Ruiyu 1 sibling, 1 reply; 16+ messages in thread From: Guo Heyi @ 2017-12-21 8:27 UTC (permalink / raw) To: Ard Biesheuvel Cc: gary guo, linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Ruiyu Ni, Jason Zhang On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: > On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: > > On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: > >> Hi Heyi, > >> > >> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > >> > PCIe on some ARM platforms requires address translation, not only for > >> > legacy IO access, but also for 32bit memory BAR access as well. There > >> > will be "Address Translation Unit" or something similar in PCI host > >> > bridges to translation CPU address to PCI address and vice versa. So > >> > we think it may be useful to add address translation support to the > >> > generic PCI host bridge driver. > >> > > >> > >> I agree. While unusual on a PC, it is quite common on other > >> architectures to have more complex non 1:1 topologies, which currently > >> require a forked PciHostBridgeDxe driver with local changes applied. > >> > >> > This RFC only contains one minor change to the definition of > >> > PciHostBridgeLib, and there certainly will be a lot of other changes > >> > to make it work, including: > >> > > >> > 1. Use CPU address for GCD space add and allocate operations, instead > >> > of PCI address; also IO space will be changed to memory space if > >> > translation exists. > >> > > >> > >> For I/O space, the translation should simply be applied to the I/O > >> range. I don't think it makes sense to use memory space here, given > >> that it is specific to architectures that lack native port I/O. > >> > > > > I made an assumption here that platforms supporting real port IO space > > do not need address translation, like IA32 and X64, and port IO > > translation implies the platform does not support real port IO space. > > > > This may be a reasonable assumption. But I still think it is better > not to encode any assumptions in the first place. > > > Indeed the assumption is not so "generic", so I'll agree if you > > recommend to support IO to IO translation as well. But I still hope to > > have IO to memory translation support in PCI host bridge driver, > > rather than in CPU IO protocol, since the faked IO space might only be > > used for PCI host bridge and we may have overlapping IO ranges for > > each host bridge, which is compatible with PCIe specification and PCIe > > ACPI description. > > > > That is fine. Under UEFI, these will translate to non-overlapping I/O > spaces in the CPU's view. Under the OS, this could be totally > different. > > > For example, > > RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff > RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff > > This is very similar to how MMIO translation works, and makes I/O > devices behind the host bridges uniquely addressable for drivers. > > For our understanding, could you share the host bridge configuration > that you are targetting? IO translation on one of our platforms is like below: PCI IO space CPU memory space 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff (The sizes are always 0x10000 so I will omit the limit for others) 0x0000 .. 0xffff -> 0x8abff0000 0x0000 .. 0xffff -> 0x8b7ff0000 ...... The translated addresses may be beyond 32bit address, will this violate IO space limitation? From EDK2 code, I didn't see such limitation for IO space. In my opinion, if we still treat the translated address as IO space in PCI host bridge driver while IO space is really translated to memory space, then we couldn't utilize the existing GCD manipulation code in PCI host bridge, including allocating memory space to avoid space colliding (we can't avoid colliding with other drivers which may also use the same memory address), setting MMU page tables. Anyway, this is not a blocking issue. Thanks and regards, Gary (Heyi Guo) > > Thanks, > Ard. > > > > How about adding a flag to indicate whether port IO is translated to > > real port IO space or system memory space? > > > > Thanks and regards, > > > > Heyi > > > >> > 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get > >> > translation of the corresponding aperture, add the translation to the > >> > input address, and then call CpuIo2 protocol; IO access will also be > >> > converted to memory access if IO translation exists. > >> > > >> > >> Again, why is this necessary? A host bridge that implements a non 1:1 > >> translation for port I/O ranges may be part of a system that has > >> native port I/O, and so the translation should be based on that. > >> > >> > 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in > >> > the discriptor. > >> > > >> > >> Indeed. Note that this has been a source of much confusion lately, > >> should we should discuss this carefully before spending time on an > >> implementation. > >> > >> > If it makes sense, then I'll continue to prepare the formal patch. > >> > > >> > Any comments? > >> > > >> > Thanks, > >> > > >> > Gary (Heyi Guo) > >> > > >> > Cc: Star Zeng <star.zeng@intel.com> > >> > Cc: Eric Dong <eric.dong@intel.com> > >> > Cc: Ruiyu Ni <ruiyu.ni@intel.com> > >> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > >> > Cc: Jason Zhang <jason.zhang@linaro.org> > >> > > >> > --- > >> > MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + > >> > 1 file changed, 1 insertion(+) > >> > > >> > diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >> > index d42e9ec..b9e8c0f 100644 > >> > --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >> > +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >> > @@ -22,6 +22,7 @@ > >> > typedef struct { > >> > UINT64 Base; > >> > UINT64 Limit; > >> > + UINT64 Translation; > >> > } PCI_ROOT_BRIDGE_APERTURE; > >> > > >> > typedef struct { > >> > -- > >> > 2.7.4 > >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 8:27 ` Guo Heyi @ 2017-12-21 8:32 ` Ard Biesheuvel 2017-12-21 9:14 ` Guo Heyi 0 siblings, 1 reply; 16+ messages in thread From: Ard Biesheuvel @ 2017-12-21 8:32 UTC (permalink / raw) To: Guo Heyi Cc: linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Ruiyu Ni, Jason Zhang On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: > On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >> > On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >> >> Hi Heyi, >> >> >> >> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >> >> > PCIe on some ARM platforms requires address translation, not only for >> >> > legacy IO access, but also for 32bit memory BAR access as well. There >> >> > will be "Address Translation Unit" or something similar in PCI host >> >> > bridges to translation CPU address to PCI address and vice versa. So >> >> > we think it may be useful to add address translation support to the >> >> > generic PCI host bridge driver. >> >> > >> >> >> >> I agree. While unusual on a PC, it is quite common on other >> >> architectures to have more complex non 1:1 topologies, which currently >> >> require a forked PciHostBridgeDxe driver with local changes applied. >> >> >> >> > This RFC only contains one minor change to the definition of >> >> > PciHostBridgeLib, and there certainly will be a lot of other changes >> >> > to make it work, including: >> >> > >> >> > 1. Use CPU address for GCD space add and allocate operations, instead >> >> > of PCI address; also IO space will be changed to memory space if >> >> > translation exists. >> >> > >> >> >> >> For I/O space, the translation should simply be applied to the I/O >> >> range. I don't think it makes sense to use memory space here, given >> >> that it is specific to architectures that lack native port I/O. >> >> >> > >> > I made an assumption here that platforms supporting real port IO space >> > do not need address translation, like IA32 and X64, and port IO >> > translation implies the platform does not support real port IO space. >> > >> >> This may be a reasonable assumption. But I still think it is better >> not to encode any assumptions in the first place. >> >> > Indeed the assumption is not so "generic", so I'll agree if you >> > recommend to support IO to IO translation as well. But I still hope to >> > have IO to memory translation support in PCI host bridge driver, >> > rather than in CPU IO protocol, since the faked IO space might only be >> > used for PCI host bridge and we may have overlapping IO ranges for >> > each host bridge, which is compatible with PCIe specification and PCIe >> > ACPI description. >> > >> >> That is fine. Under UEFI, these will translate to non-overlapping I/O >> spaces in the CPU's view. Under the OS, this could be totally >> different. >> >> >> For example, >> >> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >> >> This is very similar to how MMIO translation works, and makes I/O >> devices behind the host bridges uniquely addressable for drivers. >> >> For our understanding, could you share the host bridge configuration >> that you are targetting? > > IO translation on one of our platforms is like below: > > PCI IO space CPU memory space > 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff > (The sizes are always 0x10000 so I will omit the limit for others) > 0x0000 .. 0xffff -> 0x8abff0000 > 0x0000 .. 0xffff -> 0x8b7ff0000 > ...... > > The translated addresses may be beyond 32bit address, will this > violate IO space limitation? From EDK2 code, I didn't see such > limitation for IO space. > The MMIO address will not be used for I/O port addressing by the CPU. The MMIO to IO translation is an implementation detail of your CpuIo2 protocol implementation. So there will be two stacked translations, one for PCI I/O to CPU I/O, and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI host bridge driver. > In my opinion, if we still treat the translated address as IO space in > PCI host bridge driver while IO space is really translated to memory > space, then we couldn't utilize the existing GCD manipulation code in > PCI host bridge, including allocating memory space to avoid space > colliding (we can't avoid colliding with other drivers which may also > use the same memory address), setting MMU page tables. Anyway, this is > not a blocking issue. > > Thanks and regards, > > Gary (Heyi Guo) > >> >> Thanks, >> Ard. >> >> >> > How about adding a flag to indicate whether port IO is translated to >> > real port IO space or system memory space? >> > >> > Thanks and regards, >> > >> > Heyi >> > >> >> > 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get >> >> > translation of the corresponding aperture, add the translation to the >> >> > input address, and then call CpuIo2 protocol; IO access will also be >> >> > converted to memory access if IO translation exists. >> >> > >> >> >> >> Again, why is this necessary? A host bridge that implements a non 1:1 >> >> translation for port I/O ranges may be part of a system that has >> >> native port I/O, and so the translation should be based on that. >> >> >> >> > 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in >> >> > the discriptor. >> >> > >> >> >> >> Indeed. Note that this has been a source of much confusion lately, >> >> should we should discuss this carefully before spending time on an >> >> implementation. >> >> >> >> > If it makes sense, then I'll continue to prepare the formal patch. >> >> > >> >> > Any comments? >> >> > >> >> > Thanks, >> >> > >> >> > Gary (Heyi Guo) >> >> > >> >> > Cc: Star Zeng <star.zeng@intel.com> >> >> > Cc: Eric Dong <eric.dong@intel.com> >> >> > Cc: Ruiyu Ni <ruiyu.ni@intel.com> >> >> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >> >> > Cc: Jason Zhang <jason.zhang@linaro.org> >> >> > >> >> > --- >> >> > MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + >> >> > 1 file changed, 1 insertion(+) >> >> > >> >> > diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >> >> > index d42e9ec..b9e8c0f 100644 >> >> > --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h >> >> > +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >> >> > @@ -22,6 +22,7 @@ >> >> > typedef struct { >> >> > UINT64 Base; >> >> > UINT64 Limit; >> >> > + UINT64 Translation; >> >> > } PCI_ROOT_BRIDGE_APERTURE; >> >> > >> >> > typedef struct { >> >> > -- >> >> > 2.7.4 >> >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 8:32 ` Ard Biesheuvel @ 2017-12-21 9:14 ` Guo Heyi 2017-12-21 9:48 ` Ni, Ruiyu 0 siblings, 1 reply; 16+ messages in thread From: Guo Heyi @ 2017-12-21 9:14 UTC (permalink / raw) To: Ard Biesheuvel Cc: Guo Heyi, linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Ruiyu Ni, Jason Zhang On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: > On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: > > On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: > >> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: > >> > On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: > >> >> Hi Heyi, > >> >> > >> >> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > >> >> > PCIe on some ARM platforms requires address translation, not only for > >> >> > legacy IO access, but also for 32bit memory BAR access as well. There > >> >> > will be "Address Translation Unit" or something similar in PCI host > >> >> > bridges to translation CPU address to PCI address and vice versa. So > >> >> > we think it may be useful to add address translation support to the > >> >> > generic PCI host bridge driver. > >> >> > > >> >> > >> >> I agree. While unusual on a PC, it is quite common on other > >> >> architectures to have more complex non 1:1 topologies, which currently > >> >> require a forked PciHostBridgeDxe driver with local changes applied. > >> >> > >> >> > This RFC only contains one minor change to the definition of > >> >> > PciHostBridgeLib, and there certainly will be a lot of other changes > >> >> > to make it work, including: > >> >> > > >> >> > 1. Use CPU address for GCD space add and allocate operations, instead > >> >> > of PCI address; also IO space will be changed to memory space if > >> >> > translation exists. > >> >> > > >> >> > >> >> For I/O space, the translation should simply be applied to the I/O > >> >> range. I don't think it makes sense to use memory space here, given > >> >> that it is specific to architectures that lack native port I/O. > >> >> > >> > > >> > I made an assumption here that platforms supporting real port IO space > >> > do not need address translation, like IA32 and X64, and port IO > >> > translation implies the platform does not support real port IO space. > >> > > >> > >> This may be a reasonable assumption. But I still think it is better > >> not to encode any assumptions in the first place. > >> > >> > Indeed the assumption is not so "generic", so I'll agree if you > >> > recommend to support IO to IO translation as well. But I still hope to > >> > have IO to memory translation support in PCI host bridge driver, > >> > rather than in CPU IO protocol, since the faked IO space might only be > >> > used for PCI host bridge and we may have overlapping IO ranges for > >> > each host bridge, which is compatible with PCIe specification and PCIe > >> > ACPI description. > >> > > >> > >> That is fine. Under UEFI, these will translate to non-overlapping I/O > >> spaces in the CPU's view. Under the OS, this could be totally > >> different. > >> > >> > >> For example, > >> > >> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff > >> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff > >> > >> This is very similar to how MMIO translation works, and makes I/O > >> devices behind the host bridges uniquely addressable for drivers. > >> > >> For our understanding, could you share the host bridge configuration > >> that you are targetting? > > > > IO translation on one of our platforms is like below: > > > > PCI IO space CPU memory space > > 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff > > (The sizes are always 0x10000 so I will omit the limit for others) > > 0x0000 .. 0xffff -> 0x8abff0000 > > 0x0000 .. 0xffff -> 0x8b7ff0000 > > ...... > > > > The translated addresses may be beyond 32bit address, will this > > violate IO space limitation? From EDK2 code, I didn't see such > > limitation for IO space. > > > > The MMIO address will not be used for I/O port addressing by the CPU. > The MMIO to IO translation is an implementation detail of your CpuIo2 > protocol implementation. > > So there will be two stacked translations, one for PCI I/O to CPU I/O, > and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI > host bridge driver. Yes this should work. Hi Star, Eric and Ruiyu, Any comments on this RFC? Thanks, Gary > > > In my opinion, if we still treat the translated address as IO space in > > PCI host bridge driver while IO space is really translated to memory > > space, then we couldn't utilize the existing GCD manipulation code in > > PCI host bridge, including allocating memory space to avoid space > > colliding (we can't avoid colliding with other drivers which may also > > use the same memory address), setting MMU page tables. Anyway, this is > > not a blocking issue. > > > > Thanks and regards, > > > > Gary (Heyi Guo) > > > >> > >> Thanks, > >> Ard. > >> > >> > >> > How about adding a flag to indicate whether port IO is translated to > >> > real port IO space or system memory space? > >> > > >> > Thanks and regards, > >> > > >> > Heyi > >> > > >> >> > 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get > >> >> > translation of the corresponding aperture, add the translation to the > >> >> > input address, and then call CpuIo2 protocol; IO access will also be > >> >> > converted to memory access if IO translation exists. > >> >> > > >> >> > >> >> Again, why is this necessary? A host bridge that implements a non 1:1 > >> >> translation for port I/O ranges may be part of a system that has > >> >> native port I/O, and so the translation should be based on that. > >> >> > >> >> > 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in > >> >> > the discriptor. > >> >> > > >> >> > >> >> Indeed. Note that this has been a source of much confusion lately, > >> >> should we should discuss this carefully before spending time on an > >> >> implementation. > >> >> > >> >> > If it makes sense, then I'll continue to prepare the formal patch. > >> >> > > >> >> > Any comments? > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Gary (Heyi Guo) > >> >> > > >> >> > Cc: Star Zeng <star.zeng@intel.com> > >> >> > Cc: Eric Dong <eric.dong@intel.com> > >> >> > Cc: Ruiyu Ni <ruiyu.ni@intel.com> > >> >> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > >> >> > Cc: Jason Zhang <jason.zhang@linaro.org> > >> >> > > >> >> > --- > >> >> > MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + > >> >> > 1 file changed, 1 insertion(+) > >> >> > > >> >> > diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >> >> > index d42e9ec..b9e8c0f 100644 > >> >> > --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >> >> > +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >> >> > @@ -22,6 +22,7 @@ > >> >> > typedef struct { > >> >> > UINT64 Base; > >> >> > UINT64 Limit; > >> >> > + UINT64 Translation; > >> >> > } PCI_ROOT_BRIDGE_APERTURE; > >> >> > > >> >> > typedef struct { > >> >> > -- > >> >> > 2.7.4 > >> >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 9:14 ` Guo Heyi @ 2017-12-21 9:48 ` Ni, Ruiyu 2017-12-21 9:52 ` Ard Biesheuvel 0 siblings, 1 reply; 16+ messages in thread From: Ni, Ruiyu @ 2017-12-21 9:48 UTC (permalink / raw) To: Guo Heyi, Ard Biesheuvel Cc: linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Jason Zhang On 12/21/2017 5:14 PM, Guo Heyi wrote: > On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: >> On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: >>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >>>> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>>>>> Hi Heyi, >>>>>> >>>>>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >>>>>>> PCIe on some ARM platforms requires address translation, not only for >>>>>>> legacy IO access, but also for 32bit memory BAR access as well. There >>>>>>> will be "Address Translation Unit" or something similar in PCI host >>>>>>> bridges to translation CPU address to PCI address and vice versa. So >>>>>>> we think it may be useful to add address translation support to the >>>>>>> generic PCI host bridge driver. >>>>>>> >>>>>> >>>>>> I agree. While unusual on a PC, it is quite common on other >>>>>> architectures to have more complex non 1:1 topologies, which currently >>>>>> require a forked PciHostBridgeDxe driver with local changes applied. >>>>>> >>>>>>> This RFC only contains one minor change to the definition of >>>>>>> PciHostBridgeLib, and there certainly will be a lot of other changes >>>>>>> to make it work, including: >>>>>>> >>>>>>> 1. Use CPU address for GCD space add and allocate operations, instead >>>>>>> of PCI address; also IO space will be changed to memory space if >>>>>>> translation exists. >>>>>>> >>>>>> >>>>>> For I/O space, the translation should simply be applied to the I/O >>>>>> range. I don't think it makes sense to use memory space here, given >>>>>> that it is specific to architectures that lack native port I/O. >>>>>> >>>>> >>>>> I made an assumption here that platforms supporting real port IO space >>>>> do not need address translation, like IA32 and X64, and port IO >>>>> translation implies the platform does not support real port IO space. >>>>> >>>> >>>> This may be a reasonable assumption. But I still think it is better >>>> not to encode any assumptions in the first place. >>>> >>>>> Indeed the assumption is not so "generic", so I'll agree if you >>>>> recommend to support IO to IO translation as well. But I still hope to >>>>> have IO to memory translation support in PCI host bridge driver, >>>>> rather than in CPU IO protocol, since the faked IO space might only be >>>>> used for PCI host bridge and we may have overlapping IO ranges for >>>>> each host bridge, which is compatible with PCIe specification and PCIe >>>>> ACPI description. >>>>> >>>> >>>> That is fine. Under UEFI, these will translate to non-overlapping I/O >>>> spaces in the CPU's view. Under the OS, this could be totally >>>> different. >>>> >>>> >>>> For example, >>>> >>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >>>> >>>> This is very similar to how MMIO translation works, and makes I/O >>>> devices behind the host bridges uniquely addressable for drivers. >>>> >>>> For our understanding, could you share the host bridge configuration >>>> that you are targetting? >>> >>> IO translation on one of our platforms is like below: >>> >>> PCI IO space CPU memory space >>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>> (The sizes are always 0x10000 so I will omit the limit for others) >>> 0x0000 .. 0xffff -> 0x8abff0000 >>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>> ...... >>> >>> The translated addresses may be beyond 32bit address, will this >>> violate IO space limitation? From EDK2 code, I didn't see such >>> limitation for IO space. >>> >> >> The MMIO address will not be used for I/O port addressing by the CPU. >> The MMIO to IO translation is an implementation detail of your CpuIo2 >> protocol implementation. >> >> So there will be two stacked translations, one for PCI I/O to CPU I/O, >> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI >> host bridge driver. > > Yes this should work. > > Hi Star, Eric and Ruiyu, > > Any comments on this RFC? Let me confirm my understanding: The PciHostBridge core driver/library interface changes only take care of the MMIO translation. Heyi you will implement a special CpuIo driver in your platform code to take care of the IO to MMIO translation. But let me confirm, will you need to additional translate the MMIO (translated from IO) to another MMIO using an offset? If yes, will you handle that translation in your CpuIo driver? > > Thanks, > > Gary > >> >>> In my opinion, if we still treat the translated address as IO space in >>> PCI host bridge driver while IO space is really translated to memory >>> space, then we couldn't utilize the existing GCD manipulation code in >>> PCI host bridge, including allocating memory space to avoid space >>> colliding (we can't avoid colliding with other drivers which may also >>> use the same memory address), setting MMU page tables. Anyway, this is >>> not a blocking issue. >>> >>> Thanks and regards, >>> >>> Gary (Heyi Guo) >>> >>>> >>>> Thanks, >>>> Ard. >>>> >>>> >>>>> How about adding a flag to indicate whether port IO is translated to >>>>> real port IO space or system memory space? >>>>> >>>>> Thanks and regards, >>>>> >>>>> Heyi >>>>> >>>>>>> 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get >>>>>>> translation of the corresponding aperture, add the translation to the >>>>>>> input address, and then call CpuIo2 protocol; IO access will also be >>>>>>> converted to memory access if IO translation exists. >>>>>>> >>>>>> >>>>>> Again, why is this necessary? A host bridge that implements a non 1:1 >>>>>> translation for port I/O ranges may be part of a system that has >>>>>> native port I/O, and so the translation should be based on that. >>>>>> >>>>>>> 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in >>>>>>> the discriptor. >>>>>>> >>>>>> >>>>>> Indeed. Note that this has been a source of much confusion lately, >>>>>> should we should discuss this carefully before spending time on an >>>>>> implementation. >>>>>> >>>>>>> If it makes sense, then I'll continue to prepare the formal patch. >>>>>>> >>>>>>> Any comments? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Gary (Heyi Guo) >>>>>>> >>>>>>> Cc: Star Zeng <star.zeng@intel.com> >>>>>>> Cc: Eric Dong <eric.dong@intel.com> >>>>>>> Cc: Ruiyu Ni <ruiyu.ni@intel.com> >>>>>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >>>>>>> Cc: Jason Zhang <jason.zhang@linaro.org> >>>>>>> >>>>>>> --- >>>>>>> MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + >>>>>>> 1 file changed, 1 insertion(+) >>>>>>> >>>>>>> diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >>>>>>> index d42e9ec..b9e8c0f 100644 >>>>>>> --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h >>>>>>> +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >>>>>>> @@ -22,6 +22,7 @@ >>>>>>> typedef struct { >>>>>>> UINT64 Base; >>>>>>> UINT64 Limit; >>>>>>> + UINT64 Translation; >>>>>>> } PCI_ROOT_BRIDGE_APERTURE; >>>>>>> >>>>>>> typedef struct { >>>>>>> -- >>>>>>> 2.7.4 >>>>>>> -- Thanks, Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 9:48 ` Ni, Ruiyu @ 2017-12-21 9:52 ` Ard Biesheuvel 2017-12-21 9:59 ` Ni, Ruiyu 0 siblings, 1 reply; 16+ messages in thread From: Ard Biesheuvel @ 2017-12-21 9:52 UTC (permalink / raw) To: Ni, Ruiyu Cc: Guo Heyi, linaro-uefi, edk2-devel@lists.01.org, Star Zeng, Eric Dong, Jason Zhang On 21 December 2017 at 09:48, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: > On 12/21/2017 5:14 PM, Guo Heyi wrote: >> >> On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: >>> >>> On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: >>>> >>>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >>>>> >>>>> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >>>>>> >>>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>>>>>> >>>>>>> Hi Heyi, >>>>>>> >>>>>>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >>>>>>>> >>>>>>>> PCIe on some ARM platforms requires address translation, not only >>>>>>>> for >>>>>>>> legacy IO access, but also for 32bit memory BAR access as well. >>>>>>>> There >>>>>>>> will be "Address Translation Unit" or something similar in PCI host >>>>>>>> bridges to translation CPU address to PCI address and vice versa. So >>>>>>>> we think it may be useful to add address translation support to the >>>>>>>> generic PCI host bridge driver. >>>>>>>> >>>>>>> >>>>>>> I agree. While unusual on a PC, it is quite common on other >>>>>>> architectures to have more complex non 1:1 topologies, which >>>>>>> currently >>>>>>> require a forked PciHostBridgeDxe driver with local changes applied. >>>>>>> >>>>>>>> This RFC only contains one minor change to the definition of >>>>>>>> PciHostBridgeLib, and there certainly will be a lot of other changes >>>>>>>> to make it work, including: >>>>>>>> >>>>>>>> 1. Use CPU address for GCD space add and allocate operations, >>>>>>>> instead >>>>>>>> of PCI address; also IO space will be changed to memory space if >>>>>>>> translation exists. >>>>>>>> >>>>>>> >>>>>>> For I/O space, the translation should simply be applied to the I/O >>>>>>> range. I don't think it makes sense to use memory space here, given >>>>>>> that it is specific to architectures that lack native port I/O. >>>>>>> >>>>>> >>>>>> I made an assumption here that platforms supporting real port IO space >>>>>> do not need address translation, like IA32 and X64, and port IO >>>>>> translation implies the platform does not support real port IO space. >>>>>> >>>>> >>>>> This may be a reasonable assumption. But I still think it is better >>>>> not to encode any assumptions in the first place. >>>>> >>>>>> Indeed the assumption is not so "generic", so I'll agree if you >>>>>> recommend to support IO to IO translation as well. But I still hope to >>>>>> have IO to memory translation support in PCI host bridge driver, >>>>>> rather than in CPU IO protocol, since the faked IO space might only be >>>>>> used for PCI host bridge and we may have overlapping IO ranges for >>>>>> each host bridge, which is compatible with PCIe specification and PCIe >>>>>> ACPI description. >>>>>> >>>>> >>>>> That is fine. Under UEFI, these will translate to non-overlapping I/O >>>>> spaces in the CPU's view. Under the OS, this could be totally >>>>> different. >>>>> >>>>> >>>>> For example, >>>>> >>>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >>>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >>>>> >>>>> This is very similar to how MMIO translation works, and makes I/O >>>>> devices behind the host bridges uniquely addressable for drivers. >>>>> >>>>> For our understanding, could you share the host bridge configuration >>>>> that you are targetting? >>>> >>>> >>>> IO translation on one of our platforms is like below: >>>> >>>> PCI IO space CPU memory space >>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>>> ...... >>>> >>>> The translated addresses may be beyond 32bit address, will this >>>> violate IO space limitation? From EDK2 code, I didn't see such >>>> limitation for IO space. >>>> >>> >>> The MMIO address will not be used for I/O port addressing by the CPU. >>> The MMIO to IO translation is an implementation detail of your CpuIo2 >>> protocol implementation. >>> >>> So there will be two stacked translations, one for PCI I/O to CPU I/O, >>> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI >>> host bridge driver. >> >> >> Yes this should work. >> >> Hi Star, Eric and Ruiyu, >> >> Any comments on this RFC? > > > Let me confirm my understanding: > The PciHostBridge core driver/library interface changes only > take care of the MMIO translation. > > Heyi you will implement a special CpuIo driver in your > platform code to take care of the IO to MMIO translation. > > But let me confirm, will you need to additional translate > the MMIO (translated from IO) to another MMIO using an offset? > If yes, will you handle that translation in your CpuIo driver? > Hi Ray, The issue is that several PCIe root complexes have colliding I/O ranges: >>>> PCI IO space CPU memory space >>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>> 0x0000 .. 0xffff -> 0x8b7ff0000 So the CPU view is different from the PCI view, and to create a single CPU I/O space where all I/O port ranges behind all host bridges are accessible, we need I/O translation for the CPU. This will result in an intermediate representation >>>> PCI IO space CPU IO space >>>> 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff >>>> 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff >>>> 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff On top of that, given that ARM has no native port I/O instructions, we will need to implement MMIO to IO translation, but this can be implemented in the CpuIo2 protocol. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 9:52 ` Ard Biesheuvel @ 2017-12-21 9:59 ` Ni, Ruiyu 2017-12-21 10:07 ` Ard Biesheuvel 0 siblings, 1 reply; 16+ messages in thread From: Ni, Ruiyu @ 2017-12-21 9:59 UTC (permalink / raw) To: Ard Biesheuvel Cc: Eric Dong, Jason Zhang, edk2-devel@lists.01.org, Guo Heyi, linaro-uefi, Star Zeng On 12/21/2017 5:52 PM, Ard Biesheuvel wrote: > On 21 December 2017 at 09:48, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: >> On 12/21/2017 5:14 PM, Guo Heyi wrote: >>> >>> On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: >>>> >>>> On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: >>>>> >>>>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >>>>>> >>>>>> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >>>>>>> >>>>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>>>>>>> >>>>>>>> Hi Heyi, >>>>>>>> >>>>>>>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >>>>>>>>> >>>>>>>>> PCIe on some ARM platforms requires address translation, not only >>>>>>>>> for >>>>>>>>> legacy IO access, but also for 32bit memory BAR access as well. >>>>>>>>> There >>>>>>>>> will be "Address Translation Unit" or something similar in PCI host >>>>>>>>> bridges to translation CPU address to PCI address and vice versa. So >>>>>>>>> we think it may be useful to add address translation support to the >>>>>>>>> generic PCI host bridge driver. >>>>>>>>> >>>>>>>> >>>>>>>> I agree. While unusual on a PC, it is quite common on other >>>>>>>> architectures to have more complex non 1:1 topologies, which >>>>>>>> currently >>>>>>>> require a forked PciHostBridgeDxe driver with local changes applied. >>>>>>>> >>>>>>>>> This RFC only contains one minor change to the definition of >>>>>>>>> PciHostBridgeLib, and there certainly will be a lot of other changes >>>>>>>>> to make it work, including: >>>>>>>>> >>>>>>>>> 1. Use CPU address for GCD space add and allocate operations, >>>>>>>>> instead >>>>>>>>> of PCI address; also IO space will be changed to memory space if >>>>>>>>> translation exists. >>>>>>>>> >>>>>>>> >>>>>>>> For I/O space, the translation should simply be applied to the I/O >>>>>>>> range. I don't think it makes sense to use memory space here, given >>>>>>>> that it is specific to architectures that lack native port I/O. >>>>>>>> >>>>>>> >>>>>>> I made an assumption here that platforms supporting real port IO space >>>>>>> do not need address translation, like IA32 and X64, and port IO >>>>>>> translation implies the platform does not support real port IO space. >>>>>>> >>>>>> >>>>>> This may be a reasonable assumption. But I still think it is better >>>>>> not to encode any assumptions in the first place. >>>>>> >>>>>>> Indeed the assumption is not so "generic", so I'll agree if you >>>>>>> recommend to support IO to IO translation as well. But I still hope to >>>>>>> have IO to memory translation support in PCI host bridge driver, >>>>>>> rather than in CPU IO protocol, since the faked IO space might only be >>>>>>> used for PCI host bridge and we may have overlapping IO ranges for >>>>>>> each host bridge, which is compatible with PCIe specification and PCIe >>>>>>> ACPI description. >>>>>>> >>>>>> >>>>>> That is fine. Under UEFI, these will translate to non-overlapping I/O >>>>>> spaces in the CPU's view. Under the OS, this could be totally >>>>>> different. >>>>>> >>>>>> >>>>>> For example, >>>>>> >>>>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >>>>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >>>>>> >>>>>> This is very similar to how MMIO translation works, and makes I/O >>>>>> devices behind the host bridges uniquely addressable for drivers. >>>>>> >>>>>> For our understanding, could you share the host bridge configuration >>>>>> that you are targetting? >>>>> >>>>> >>>>> IO translation on one of our platforms is like below: >>>>> >>>>> PCI IO space CPU memory space >>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>>>> ...... >>>>> >>>>> The translated addresses may be beyond 32bit address, will this >>>>> violate IO space limitation? From EDK2 code, I didn't see such >>>>> limitation for IO space. >>>>> >>>> >>>> The MMIO address will not be used for I/O port addressing by the CPU. >>>> The MMIO to IO translation is an implementation detail of your CpuIo2 >>>> protocol implementation. >>>> >>>> So there will be two stacked translations, one for PCI I/O to CPU I/O, >>>> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI >>>> host bridge driver. >>> >>> >>> Yes this should work. >>> >>> Hi Star, Eric and Ruiyu, >>> >>> Any comments on this RFC? >> >> >> Let me confirm my understanding: >> The PciHostBridge core driver/library interface changes only >> take care of the MMIO translation. >> >> Heyi you will implement a special CpuIo driver in your >> platform code to take care of the IO to MMIO translation. >> >> But let me confirm, will you need to additional translate >> the MMIO (translated from IO) to another MMIO using an offset? >> If yes, will you handle that translation in your CpuIo driver? >> > > Hi Ray, > > The issue is that several PCIe root complexes have colliding I/O ranges: Ard, The IO-MMIO mapping needs CPU support. I am not sure whether IA32 or x64 supports. But I guess ARM supports. right? Will all the IO part be implemented in ARM CpuIo2 protocol? > >>>>> PCI IO space CPU memory space >>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 > > So the CPU view is different from the PCI view, and to create a single > CPU I/O space where all I/O port ranges behind all host bridges are > accessible, we need I/O translation for the CPU. This will result in > an intermediate representation > >>>>> PCI IO space CPU IO space >>>>> 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff >>>>> 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff >>>>> 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff > > On top of that, given that ARM has no native port I/O instructions, we > will need to implement MMIO to IO translation, but this can be > implemented in the CpuIo2 protocol. > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > -- Thanks, Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 9:59 ` Ni, Ruiyu @ 2017-12-21 10:07 ` Ard Biesheuvel 2017-12-26 6:50 ` Guo Heyi 0 siblings, 1 reply; 16+ messages in thread From: Ard Biesheuvel @ 2017-12-21 10:07 UTC (permalink / raw) To: Ni, Ruiyu Cc: Eric Dong, Jason Zhang, edk2-devel@lists.01.org, Guo Heyi, linaro-uefi, Star Zeng On 21 December 2017 at 09:59, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: > On 12/21/2017 5:52 PM, Ard Biesheuvel wrote: >> >> On 21 December 2017 at 09:48, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: >>> >>> On 12/21/2017 5:14 PM, Guo Heyi wrote: >>>> >>>> >>>> On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: >>>>> >>>>> >>>>> On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: >>>>>> >>>>>> >>>>>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >>>>>>> >>>>>>> >>>>>>> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Heyi, >>>>>>>>> >>>>>>>>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> PCIe on some ARM platforms requires address translation, not only >>>>>>>>>> for >>>>>>>>>> legacy IO access, but also for 32bit memory BAR access as well. >>>>>>>>>> There >>>>>>>>>> will be "Address Translation Unit" or something similar in PCI >>>>>>>>>> host >>>>>>>>>> bridges to translation CPU address to PCI address and vice versa. >>>>>>>>>> So >>>>>>>>>> we think it may be useful to add address translation support to >>>>>>>>>> the >>>>>>>>>> generic PCI host bridge driver. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I agree. While unusual on a PC, it is quite common on other >>>>>>>>> architectures to have more complex non 1:1 topologies, which >>>>>>>>> currently >>>>>>>>> require a forked PciHostBridgeDxe driver with local changes >>>>>>>>> applied. >>>>>>>>> >>>>>>>>>> This RFC only contains one minor change to the definition of >>>>>>>>>> PciHostBridgeLib, and there certainly will be a lot of other >>>>>>>>>> changes >>>>>>>>>> to make it work, including: >>>>>>>>>> >>>>>>>>>> 1. Use CPU address for GCD space add and allocate operations, >>>>>>>>>> instead >>>>>>>>>> of PCI address; also IO space will be changed to memory space if >>>>>>>>>> translation exists. >>>>>>>>>> >>>>>>>>> >>>>>>>>> For I/O space, the translation should simply be applied to the I/O >>>>>>>>> range. I don't think it makes sense to use memory space here, given >>>>>>>>> that it is specific to architectures that lack native port I/O. >>>>>>>>> >>>>>>>> >>>>>>>> I made an assumption here that platforms supporting real port IO >>>>>>>> space >>>>>>>> do not need address translation, like IA32 and X64, and port IO >>>>>>>> translation implies the platform does not support real port IO >>>>>>>> space. >>>>>>>> >>>>>>> >>>>>>> This may be a reasonable assumption. But I still think it is better >>>>>>> not to encode any assumptions in the first place. >>>>>>> >>>>>>>> Indeed the assumption is not so "generic", so I'll agree if you >>>>>>>> recommend to support IO to IO translation as well. But I still hope >>>>>>>> to >>>>>>>> have IO to memory translation support in PCI host bridge driver, >>>>>>>> rather than in CPU IO protocol, since the faked IO space might only >>>>>>>> be >>>>>>>> used for PCI host bridge and we may have overlapping IO ranges for >>>>>>>> each host bridge, which is compatible with PCIe specification and >>>>>>>> PCIe >>>>>>>> ACPI description. >>>>>>>> >>>>>>> >>>>>>> That is fine. Under UEFI, these will translate to non-overlapping I/O >>>>>>> spaces in the CPU's view. Under the OS, this could be totally >>>>>>> different. >>>>>>> >>>>>>> >>>>>>> For example, >>>>>>> >>>>>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >>>>>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >>>>>>> >>>>>>> This is very similar to how MMIO translation works, and makes I/O >>>>>>> devices behind the host bridges uniquely addressable for drivers. >>>>>>> >>>>>>> For our understanding, could you share the host bridge configuration >>>>>>> that you are targetting? >>>>>> >>>>>> >>>>>> >>>>>> IO translation on one of our platforms is like below: >>>>>> >>>>>> PCI IO space CPU memory space >>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>>>>> ...... >>>>>> >>>>>> The translated addresses may be beyond 32bit address, will this >>>>>> violate IO space limitation? From EDK2 code, I didn't see such >>>>>> limitation for IO space. >>>>>> >>>>> >>>>> The MMIO address will not be used for I/O port addressing by the CPU. >>>>> The MMIO to IO translation is an implementation detail of your CpuIo2 >>>>> protocol implementation. >>>>> >>>>> So there will be two stacked translations, one for PCI I/O to CPU I/O, >>>>> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI >>>>> host bridge driver. >>>> >>>> >>>> >>>> Yes this should work. >>>> >>>> Hi Star, Eric and Ruiyu, >>>> >>>> Any comments on this RFC? >>> >>> >>> >>> Let me confirm my understanding: >>> The PciHostBridge core driver/library interface changes only >>> take care of the MMIO translation. >>> >>> Heyi you will implement a special CpuIo driver in your >>> platform code to take care of the IO to MMIO translation. >>> >>> But let me confirm, will you need to additional translate >>> the MMIO (translated from IO) to another MMIO using an offset? >>> If yes, will you handle that translation in your CpuIo driver? >>> >> >> Hi Ray, >> >> The issue is that several PCIe root complexes have colliding I/O ranges: > > > Ard, > The IO-MMIO mapping needs CPU support. I am not sure whether IA32 or > x64 supports. > But I guess ARM supports. right? > Yes. Exposing PCI I/O ranges via MMIO translation is specific to the CpuIo2 implementations we have for ARM. > Will all the IO part be implemented in ARM CpuIo2 protocol? > No. The CPU to PCI I/O translation needs to be in PciHostBridgeDxe, because it is in charge of allocating the GCD space, and without translation, those allocations will collide. Also, while perhaps non-existent in reality, it is imaginable that a host bridge could translate port I/O addresses between the two sides of the bridge, similar to how non 1:1 mapped PCI MMIO is handled. So just like MMIO, the port I/O address used by the CPU, and the port I/O address programmed into the device BAR could be subject to translation. This is not solvable in the CpuIo2 protocol, because without translation at the host bridge driver level, a CPU port I/O address is ambiguous: e.g., address 0x1000 may apply to each of the RCs. >> >>>>>> PCI IO space CPU memory space >>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >> >> >> So the CPU view is different from the PCI view, and to create a single >> CPU I/O space where all I/O port ranges behind all host bridges are >> accessible, we need I/O translation for the CPU. This will result in >> an intermediate representation >> >>>>>> PCI IO space CPU IO space >>>>>> 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff >>>>>> 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff >>>>>> 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff >> >> >> On top of that, given that ARM has no native port I/O instructions, we >> will need to implement MMIO to IO translation, but this can be >> implemented in the CpuIo2 protocol. >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel >> > > > -- > Thanks, > Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 10:07 ` Ard Biesheuvel @ 2017-12-26 6:50 ` Guo Heyi 2018-01-02 7:56 ` Ni, Ruiyu 0 siblings, 1 reply; 16+ messages in thread From: Guo Heyi @ 2017-12-26 6:50 UTC (permalink / raw) To: Ard Biesheuvel Cc: Ni, Ruiyu, Eric Dong, Jason Zhang, edk2-devel@lists.01.org, Guo Heyi, linaro-uefi, Star Zeng Hi Ard, Ray, Have we come to the final conclusion? Or are we still waiting for more comments on this? Thanks, Gary On Thu, Dec 21, 2017 at 10:07:51AM +0000, Ard Biesheuvel wrote: > On 21 December 2017 at 09:59, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: > > On 12/21/2017 5:52 PM, Ard Biesheuvel wrote: > >> > >> On 21 December 2017 at 09:48, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: > >>> > >>> On 12/21/2017 5:14 PM, Guo Heyi wrote: > >>>> > >>>> > >>>> On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: > >>>>> > >>>>> > >>>>> On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: > >>>>>> > >>>>>> > >>>>>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: > >>>>>>> > >>>>>>> > >>>>>>> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Hi Heyi, > >>>>>>>>> > >>>>>>>>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> PCIe on some ARM platforms requires address translation, not only > >>>>>>>>>> for > >>>>>>>>>> legacy IO access, but also for 32bit memory BAR access as well. > >>>>>>>>>> There > >>>>>>>>>> will be "Address Translation Unit" or something similar in PCI > >>>>>>>>>> host > >>>>>>>>>> bridges to translation CPU address to PCI address and vice versa. > >>>>>>>>>> So > >>>>>>>>>> we think it may be useful to add address translation support to > >>>>>>>>>> the > >>>>>>>>>> generic PCI host bridge driver. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> I agree. While unusual on a PC, it is quite common on other > >>>>>>>>> architectures to have more complex non 1:1 topologies, which > >>>>>>>>> currently > >>>>>>>>> require a forked PciHostBridgeDxe driver with local changes > >>>>>>>>> applied. > >>>>>>>>> > >>>>>>>>>> This RFC only contains one minor change to the definition of > >>>>>>>>>> PciHostBridgeLib, and there certainly will be a lot of other > >>>>>>>>>> changes > >>>>>>>>>> to make it work, including: > >>>>>>>>>> > >>>>>>>>>> 1. Use CPU address for GCD space add and allocate operations, > >>>>>>>>>> instead > >>>>>>>>>> of PCI address; also IO space will be changed to memory space if > >>>>>>>>>> translation exists. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> For I/O space, the translation should simply be applied to the I/O > >>>>>>>>> range. I don't think it makes sense to use memory space here, given > >>>>>>>>> that it is specific to architectures that lack native port I/O. > >>>>>>>>> > >>>>>>>> > >>>>>>>> I made an assumption here that platforms supporting real port IO > >>>>>>>> space > >>>>>>>> do not need address translation, like IA32 and X64, and port IO > >>>>>>>> translation implies the platform does not support real port IO > >>>>>>>> space. > >>>>>>>> > >>>>>>> > >>>>>>> This may be a reasonable assumption. But I still think it is better > >>>>>>> not to encode any assumptions in the first place. > >>>>>>> > >>>>>>>> Indeed the assumption is not so "generic", so I'll agree if you > >>>>>>>> recommend to support IO to IO translation as well. But I still hope > >>>>>>>> to > >>>>>>>> have IO to memory translation support in PCI host bridge driver, > >>>>>>>> rather than in CPU IO protocol, since the faked IO space might only > >>>>>>>> be > >>>>>>>> used for PCI host bridge and we may have overlapping IO ranges for > >>>>>>>> each host bridge, which is compatible with PCIe specification and > >>>>>>>> PCIe > >>>>>>>> ACPI description. > >>>>>>>> > >>>>>>> > >>>>>>> That is fine. Under UEFI, these will translate to non-overlapping I/O > >>>>>>> spaces in the CPU's view. Under the OS, this could be totally > >>>>>>> different. > >>>>>>> > >>>>>>> > >>>>>>> For example, > >>>>>>> > >>>>>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff > >>>>>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff > >>>>>>> > >>>>>>> This is very similar to how MMIO translation works, and makes I/O > >>>>>>> devices behind the host bridges uniquely addressable for drivers. > >>>>>>> > >>>>>>> For our understanding, could you share the host bridge configuration > >>>>>>> that you are targetting? > >>>>>> > >>>>>> > >>>>>> > >>>>>> IO translation on one of our platforms is like below: > >>>>>> > >>>>>> PCI IO space CPU memory space > >>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff > >>>>>> (The sizes are always 0x10000 so I will omit the limit for others) > >>>>>> 0x0000 .. 0xffff -> 0x8abff0000 > >>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 > >>>>>> ...... > >>>>>> > >>>>>> The translated addresses may be beyond 32bit address, will this > >>>>>> violate IO space limitation? From EDK2 code, I didn't see such > >>>>>> limitation for IO space. > >>>>>> > >>>>> > >>>>> The MMIO address will not be used for I/O port addressing by the CPU. > >>>>> The MMIO to IO translation is an implementation detail of your CpuIo2 > >>>>> protocol implementation. > >>>>> > >>>>> So there will be two stacked translations, one for PCI I/O to CPU I/O, > >>>>> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI > >>>>> host bridge driver. > >>>> > >>>> > >>>> > >>>> Yes this should work. > >>>> > >>>> Hi Star, Eric and Ruiyu, > >>>> > >>>> Any comments on this RFC? > >>> > >>> > >>> > >>> Let me confirm my understanding: > >>> The PciHostBridge core driver/library interface changes only > >>> take care of the MMIO translation. > >>> > >>> Heyi you will implement a special CpuIo driver in your > >>> platform code to take care of the IO to MMIO translation. > >>> > >>> But let me confirm, will you need to additional translate > >>> the MMIO (translated from IO) to another MMIO using an offset? > >>> If yes, will you handle that translation in your CpuIo driver? > >>> > >> > >> Hi Ray, > >> > >> The issue is that several PCIe root complexes have colliding I/O ranges: > > > > > > Ard, > > The IO-MMIO mapping needs CPU support. I am not sure whether IA32 or > > x64 supports. > > But I guess ARM supports. right? > > > > Yes. Exposing PCI I/O ranges via MMIO translation is specific to the > CpuIo2 implementations we have for ARM. > > > Will all the IO part be implemented in ARM CpuIo2 protocol? > > > > No. The CPU to PCI I/O translation needs to be in PciHostBridgeDxe, > because it is in charge of allocating the GCD space, and without > translation, those allocations will collide. Also, while perhaps > non-existent in reality, it is imaginable that a host bridge could > translate port I/O addresses between the two sides of the bridge, > similar to how non 1:1 mapped PCI MMIO is handled. > > So just like MMIO, the port I/O address used by the CPU, and the port > I/O address programmed into the device BAR could be subject to > translation. > > This is not solvable in the CpuIo2 protocol, because without > translation at the host bridge driver level, a CPU port I/O address is > ambiguous: e.g., address 0x1000 may apply to each of the RCs. > > > >> > >>>>>> PCI IO space CPU memory space > >>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff > >>>>>> (The sizes are always 0x10000 so I will omit the limit for others) > >>>>>> 0x0000 .. 0xffff -> 0x8abff0000 > >>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 > >> > >> > >> So the CPU view is different from the PCI view, and to create a single > >> CPU I/O space where all I/O port ranges behind all host bridges are > >> accessible, we need I/O translation for the CPU. This will result in > >> an intermediate representation > >> > >>>>>> PCI IO space CPU IO space > >>>>>> 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff > >>>>>> 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff > >>>>>> 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff > >> > >> > >> On top of that, given that ARM has no native port I/O instructions, we > >> will need to implement MMIO to IO translation, but this can be > >> implemented in the CpuIo2 protocol. > >> _______________________________________________ > >> edk2-devel mailing list > >> edk2-devel@lists.01.org > >> https://lists.01.org/mailman/listinfo/edk2-devel > >> > > > > > > -- > > Thanks, > > Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-26 6:50 ` Guo Heyi @ 2018-01-02 7:56 ` Ni, Ruiyu 2018-01-02 15:09 ` gary guo 0 siblings, 1 reply; 16+ messages in thread From: Ni, Ruiyu @ 2018-01-02 7:56 UTC (permalink / raw) To: Guo Heyi, Ard Biesheuvel Cc: Eric Dong, Jason Zhang, edk2-devel@lists.01.org, linaro-uefi, Star Zeng On 12/26/2017 2:50 PM, Guo Heyi wrote: > Hi Ard, Ray, > > Have we come to the final conclusion? Or are we still waiting for more comments on this? Heyi, I think you can send out a draft version of changes for better understanding. > > Thanks, > > Gary > > On Thu, Dec 21, 2017 at 10:07:51AM +0000, Ard Biesheuvel wrote: >> On 21 December 2017 at 09:59, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: >>> On 12/21/2017 5:52 PM, Ard Biesheuvel wrote: >>>> >>>> On 21 December 2017 at 09:48, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: >>>>> >>>>> On 12/21/2017 5:14 PM, Guo Heyi wrote: >>>>>> >>>>>> >>>>>> On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: >>>>>>> >>>>>>> >>>>>>> On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Heyi, >>>>>>>>>>> >>>>>>>>>>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> PCIe on some ARM platforms requires address translation, not only >>>>>>>>>>>> for >>>>>>>>>>>> legacy IO access, but also for 32bit memory BAR access as well. >>>>>>>>>>>> There >>>>>>>>>>>> will be "Address Translation Unit" or something similar in PCI >>>>>>>>>>>> host >>>>>>>>>>>> bridges to translation CPU address to PCI address and vice versa. >>>>>>>>>>>> So >>>>>>>>>>>> we think it may be useful to add address translation support to >>>>>>>>>>>> the >>>>>>>>>>>> generic PCI host bridge driver. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I agree. While unusual on a PC, it is quite common on other >>>>>>>>>>> architectures to have more complex non 1:1 topologies, which >>>>>>>>>>> currently >>>>>>>>>>> require a forked PciHostBridgeDxe driver with local changes >>>>>>>>>>> applied. >>>>>>>>>>> >>>>>>>>>>>> This RFC only contains one minor change to the definition of >>>>>>>>>>>> PciHostBridgeLib, and there certainly will be a lot of other >>>>>>>>>>>> changes >>>>>>>>>>>> to make it work, including: >>>>>>>>>>>> >>>>>>>>>>>> 1. Use CPU address for GCD space add and allocate operations, >>>>>>>>>>>> instead >>>>>>>>>>>> of PCI address; also IO space will be changed to memory space if >>>>>>>>>>>> translation exists. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> For I/O space, the translation should simply be applied to the I/O >>>>>>>>>>> range. I don't think it makes sense to use memory space here, given >>>>>>>>>>> that it is specific to architectures that lack native port I/O. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I made an assumption here that platforms supporting real port IO >>>>>>>>>> space >>>>>>>>>> do not need address translation, like IA32 and X64, and port IO >>>>>>>>>> translation implies the platform does not support real port IO >>>>>>>>>> space. >>>>>>>>>> >>>>>>>>> >>>>>>>>> This may be a reasonable assumption. But I still think it is better >>>>>>>>> not to encode any assumptions in the first place. >>>>>>>>> >>>>>>>>>> Indeed the assumption is not so "generic", so I'll agree if you >>>>>>>>>> recommend to support IO to IO translation as well. But I still hope >>>>>>>>>> to >>>>>>>>>> have IO to memory translation support in PCI host bridge driver, >>>>>>>>>> rather than in CPU IO protocol, since the faked IO space might only >>>>>>>>>> be >>>>>>>>>> used for PCI host bridge and we may have overlapping IO ranges for >>>>>>>>>> each host bridge, which is compatible with PCIe specification and >>>>>>>>>> PCIe >>>>>>>>>> ACPI description. >>>>>>>>>> >>>>>>>>> >>>>>>>>> That is fine. Under UEFI, these will translate to non-overlapping I/O >>>>>>>>> spaces in the CPU's view. Under the OS, this could be totally >>>>>>>>> different. >>>>>>>>> >>>>>>>>> >>>>>>>>> For example, >>>>>>>>> >>>>>>>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >>>>>>>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >>>>>>>>> >>>>>>>>> This is very similar to how MMIO translation works, and makes I/O >>>>>>>>> devices behind the host bridges uniquely addressable for drivers. >>>>>>>>> >>>>>>>>> For our understanding, could you share the host bridge configuration >>>>>>>>> that you are targetting? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> IO translation on one of our platforms is like below: >>>>>>>> >>>>>>>> PCI IO space CPU memory space >>>>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>>>>>>> ...... >>>>>>>> >>>>>>>> The translated addresses may be beyond 32bit address, will this >>>>>>>> violate IO space limitation? From EDK2 code, I didn't see such >>>>>>>> limitation for IO space. >>>>>>>> >>>>>>> >>>>>>> The MMIO address will not be used for I/O port addressing by the CPU. >>>>>>> The MMIO to IO translation is an implementation detail of your CpuIo2 >>>>>>> protocol implementation. >>>>>>> >>>>>>> So there will be two stacked translations, one for PCI I/O to CPU I/O, >>>>>>> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI >>>>>>> host bridge driver. >>>>>> >>>>>> >>>>>> >>>>>> Yes this should work. >>>>>> >>>>>> Hi Star, Eric and Ruiyu, >>>>>> >>>>>> Any comments on this RFC? >>>>> >>>>> >>>>> >>>>> Let me confirm my understanding: >>>>> The PciHostBridge core driver/library interface changes only >>>>> take care of the MMIO translation. >>>>> >>>>> Heyi you will implement a special CpuIo driver in your >>>>> platform code to take care of the IO to MMIO translation. >>>>> >>>>> But let me confirm, will you need to additional translate >>>>> the MMIO (translated from IO) to another MMIO using an offset? >>>>> If yes, will you handle that translation in your CpuIo driver? >>>>> >>>> >>>> Hi Ray, >>>> >>>> The issue is that several PCIe root complexes have colliding I/O ranges: >>> >>> >>> Ard, >>> The IO-MMIO mapping needs CPU support. I am not sure whether IA32 or >>> x64 supports. >>> But I guess ARM supports. right? >>> >> >> Yes. Exposing PCI I/O ranges via MMIO translation is specific to the >> CpuIo2 implementations we have for ARM. >> >>> Will all the IO part be implemented in ARM CpuIo2 protocol? >>> >> >> No. The CPU to PCI I/O translation needs to be in PciHostBridgeDxe, >> because it is in charge of allocating the GCD space, and without >> translation, those allocations will collide. Also, while perhaps >> non-existent in reality, it is imaginable that a host bridge could >> translate port I/O addresses between the two sides of the bridge, >> similar to how non 1:1 mapped PCI MMIO is handled. >> >> So just like MMIO, the port I/O address used by the CPU, and the port >> I/O address programmed into the device BAR could be subject to >> translation. >> >> This is not solvable in the CpuIo2 protocol, because without >> translation at the host bridge driver level, a CPU port I/O address is >> ambiguous: e.g., address 0x1000 may apply to each of the RCs. >> >> >>>> >>>>>>>> PCI IO space CPU memory space >>>>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>>> >>>> >>>> So the CPU view is different from the PCI view, and to create a single >>>> CPU I/O space where all I/O port ranges behind all host bridges are >>>> accessible, we need I/O translation for the CPU. This will result in >>>> an intermediate representation >>>> >>>>>>>> PCI IO space CPU IO space >>>>>>>> 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff >>>>>>>> 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff >>>>>>>> 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff >>>> >>>> >>>> On top of that, given that ARM has no native port I/O instructions, we >>>> will need to implement MMIO to IO translation, but this can be >>>> implemented in the CpuIo2 protocol. >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >>>> >>> >>> >>> -- >>> Thanks, >>> Ray > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > -- Thanks, Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2018-01-02 7:56 ` Ni, Ruiyu @ 2018-01-02 15:09 ` gary guo 0 siblings, 0 replies; 16+ messages in thread From: gary guo @ 2018-01-02 15:09 UTC (permalink / raw) To: Ni, Ruiyu Cc: Guo Heyi, Ard Biesheuvel, Eric Dong, Jason Zhang, edk2-devel@lists.01.org, linaro-uefi, Star Zeng On Tue, Jan 02, 2018 at 03:56:14PM +0800, Ni, Ruiyu wrote: > On 12/26/2017 2:50 PM, Guo Heyi wrote: > > Hi Ard, Ray, > > > > Have we come to the final conclusion? Or are we still waiting for more comments on this? > > Heyi, > I think you can send out a draft version of changes for better > understanding. Sure, we can do that. Thanks, Gary > > > > > Thanks, > > > > Gary > > > > On Thu, Dec 21, 2017 at 10:07:51AM +0000, Ard Biesheuvel wrote: > > > On 21 December 2017 at 09:59, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: > > > > On 12/21/2017 5:52 PM, Ard Biesheuvel wrote: > > > > > > > > > > On 21 December 2017 at 09:48, Ni, Ruiyu <ruiyu.ni@intel.com> wrote: > > > > > > > > > > > > On 12/21/2017 5:14 PM, Guo Heyi wrote: > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 21 December 2017 at 08:27, Guo Heyi <heyi.guo@linaro.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Heyi, > > > > > > > > > > > > > > > > > > > > > > > > On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PCIe on some ARM platforms requires address translation, not only > > > > > > > > > > > > > for > > > > > > > > > > > > > legacy IO access, but also for 32bit memory BAR access as well. > > > > > > > > > > > > > There > > > > > > > > > > > > > will be "Address Translation Unit" or something similar in PCI > > > > > > > > > > > > > host > > > > > > > > > > > > > bridges to translation CPU address to PCI address and vice versa. > > > > > > > > > > > > > So > > > > > > > > > > > > > we think it may be useful to add address translation support to > > > > > > > > > > > > > the > > > > > > > > > > > > > generic PCI host bridge driver. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree. While unusual on a PC, it is quite common on other > > > > > > > > > > > > architectures to have more complex non 1:1 topologies, which > > > > > > > > > > > > currently > > > > > > > > > > > > require a forked PciHostBridgeDxe driver with local changes > > > > > > > > > > > > applied. > > > > > > > > > > > > > > > > > > > > > > > > > This RFC only contains one minor change to the definition of > > > > > > > > > > > > > PciHostBridgeLib, and there certainly will be a lot of other > > > > > > > > > > > > > changes > > > > > > > > > > > > > to make it work, including: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Use CPU address for GCD space add and allocate operations, > > > > > > > > > > > > > instead > > > > > > > > > > > > > of PCI address; also IO space will be changed to memory space if > > > > > > > > > > > > > translation exists. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For I/O space, the translation should simply be applied to the I/O > > > > > > > > > > > > range. I don't think it makes sense to use memory space here, given > > > > > > > > > > > > that it is specific to architectures that lack native port I/O. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I made an assumption here that platforms supporting real port IO > > > > > > > > > > > space > > > > > > > > > > > do not need address translation, like IA32 and X64, and port IO > > > > > > > > > > > translation implies the platform does not support real port IO > > > > > > > > > > > space. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This may be a reasonable assumption. But I still think it is better > > > > > > > > > > not to encode any assumptions in the first place. > > > > > > > > > > > > > > > > > > > > > Indeed the assumption is not so "generic", so I'll agree if you > > > > > > > > > > > recommend to support IO to IO translation as well. But I still hope > > > > > > > > > > > to > > > > > > > > > > > have IO to memory translation support in PCI host bridge driver, > > > > > > > > > > > rather than in CPU IO protocol, since the faked IO space might only > > > > > > > > > > > be > > > > > > > > > > > used for PCI host bridge and we may have overlapping IO ranges for > > > > > > > > > > > each host bridge, which is compatible with PCIe specification and > > > > > > > > > > > PCIe > > > > > > > > > > > ACPI description. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That is fine. Under UEFI, these will translate to non-overlapping I/O > > > > > > > > > > spaces in the CPU's view. Under the OS, this could be totally > > > > > > > > > > different. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, > > > > > > > > > > > > > > > > > > > > RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff > > > > > > > > > > RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff > > > > > > > > > > > > > > > > > > > > This is very similar to how MMIO translation works, and makes I/O > > > > > > > > > > devices behind the host bridges uniquely addressable for drivers. > > > > > > > > > > > > > > > > > > > > For our understanding, could you share the host bridge configuration > > > > > > > > > > that you are targetting? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IO translation on one of our platforms is like below: > > > > > > > > > > > > > > > > > > PCI IO space CPU memory space > > > > > > > > > 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff > > > > > > > > > (The sizes are always 0x10000 so I will omit the limit for others) > > > > > > > > > 0x0000 .. 0xffff -> 0x8abff0000 > > > > > > > > > 0x0000 .. 0xffff -> 0x8b7ff0000 > > > > > > > > > ...... > > > > > > > > > > > > > > > > > > The translated addresses may be beyond 32bit address, will this > > > > > > > > > violate IO space limitation? From EDK2 code, I didn't see such > > > > > > > > > limitation for IO space. > > > > > > > > > > > > > > > > > > > > > > > > > The MMIO address will not be used for I/O port addressing by the CPU. > > > > > > > > The MMIO to IO translation is an implementation detail of your CpuIo2 > > > > > > > > protocol implementation. > > > > > > > > > > > > > > > > So there will be two stacked translations, one for PCI I/O to CPU I/O, > > > > > > > > and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI > > > > > > > > host bridge driver. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes this should work. > > > > > > > > > > > > > > Hi Star, Eric and Ruiyu, > > > > > > > > > > > > > > Any comments on this RFC? > > > > > > > > > > > > > > > > > > > > > > > > Let me confirm my understanding: > > > > > > The PciHostBridge core driver/library interface changes only > > > > > > take care of the MMIO translation. > > > > > > > > > > > > Heyi you will implement a special CpuIo driver in your > > > > > > platform code to take care of the IO to MMIO translation. > > > > > > > > > > > > But let me confirm, will you need to additional translate > > > > > > the MMIO (translated from IO) to another MMIO using an offset? > > > > > > If yes, will you handle that translation in your CpuIo driver? > > > > > > > > > > > > > > > > Hi Ray, > > > > > > > > > > The issue is that several PCIe root complexes have colliding I/O ranges: > > > > > > > > > > > > Ard, > > > > The IO-MMIO mapping needs CPU support. I am not sure whether IA32 or > > > > x64 supports. > > > > But I guess ARM supports. right? > > > > > > > > > > Yes. Exposing PCI I/O ranges via MMIO translation is specific to the > > > CpuIo2 implementations we have for ARM. > > > > > > > Will all the IO part be implemented in ARM CpuIo2 protocol? > > > > > > > > > > No. The CPU to PCI I/O translation needs to be in PciHostBridgeDxe, > > > because it is in charge of allocating the GCD space, and without > > > translation, those allocations will collide. Also, while perhaps > > > non-existent in reality, it is imaginable that a host bridge could > > > translate port I/O addresses between the two sides of the bridge, > > > similar to how non 1:1 mapped PCI MMIO is handled. > > > > > > So just like MMIO, the port I/O address used by the CPU, and the port > > > I/O address programmed into the device BAR could be subject to > > > translation. > > > > > > This is not solvable in the CpuIo2 protocol, because without > > > translation at the host bridge driver level, a CPU port I/O address is > > > ambiguous: e.g., address 0x1000 may apply to each of the RCs. > > > > > > > > > > > > > > > > > > > > PCI IO space CPU memory space > > > > > > > > > 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff > > > > > > > > > (The sizes are always 0x10000 so I will omit the limit for others) > > > > > > > > > 0x0000 .. 0xffff -> 0x8abff0000 > > > > > > > > > 0x0000 .. 0xffff -> 0x8b7ff0000 > > > > > > > > > > > > > > > So the CPU view is different from the PCI view, and to create a single > > > > > CPU I/O space where all I/O port ranges behind all host bridges are > > > > > accessible, we need I/O translation for the CPU. This will result in > > > > > an intermediate representation > > > > > > > > > > > > > > PCI IO space CPU IO space > > > > > > > > > 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff > > > > > > > > > 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff > > > > > > > > > 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff > > > > > > > > > > > > > > > On top of that, given that ARM has no native port I/O instructions, we > > > > > will need to implement MMIO to IO translation, but this can be > > > > > implemented in the CpuIo2 protocol. > > > > > _______________________________________________ > > > > > edk2-devel mailing list > > > > > edk2-devel@lists.01.org > > > > > https://lists.01.org/mailman/listinfo/edk2-devel > > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Ray > > _______________________________________________ > > edk2-devel mailing list > > edk2-devel@lists.01.org > > https://lists.01.org/mailman/listinfo/edk2-devel > > > > > -- > Thanks, > Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-20 15:26 ` Ard Biesheuvel 2017-12-21 8:27 ` Guo Heyi @ 2017-12-21 9:43 ` Ni, Ruiyu 2017-12-21 11:32 ` Guo Heyi 1 sibling, 1 reply; 16+ messages in thread From: Ni, Ruiyu @ 2017-12-21 9:43 UTC (permalink / raw) To: Ard Biesheuvel, gary guo Cc: linaro-uefi, edk2-devel@lists.01.org, Zeng, Star, Dong, Eric, Jason Zhang On 12/20/2017 11:26 PM, Ard Biesheuvel wrote: > On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: >> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>> Hi Heyi, >>> >>> On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: >>>> PCIe on some ARM platforms requires address translation, not only for >>>> legacy IO access, but also for 32bit memory BAR access as well. There >>>> will be "Address Translation Unit" or something similar in PCI host >>>> bridges to translation CPU address to PCI address and vice versa. So >>>> we think it may be useful to add address translation support to the >>>> generic PCI host bridge driver. >>>> >>> >>> I agree. While unusual on a PC, it is quite common on other >>> architectures to have more complex non 1:1 topologies, which currently >>> require a forked PciHostBridgeDxe driver with local changes applied. >>> >>>> This RFC only contains one minor change to the definition of >>>> PciHostBridgeLib, and there certainly will be a lot of other changes >>>> to make it work, including: >>>> >>>> 1. Use CPU address for GCD space add and allocate operations, instead >>>> of PCI address; also IO space will be changed to memory space if >>>> translation exists. >>>> >>> >>> For I/O space, the translation should simply be applied to the I/O >>> range. I don't think it makes sense to use memory space here, given >>> that it is specific to architectures that lack native port I/O. >>> >> >> I made an assumption here that platforms supporting real port IO space >> do not need address translation, like IA32 and X64, and port IO >> translation implies the platform does not support real port IO space. >> > > This may be a reasonable assumption. But I still think it is better > not to encode any assumptions in the first place. > >> Indeed the assumption is not so "generic", so I'll agree if you >> recommend to support IO to IO translation as well. But I still hope to >> have IO to memory translation support in PCI host bridge driver, >> rather than in CPU IO protocol, since the faked IO space might only be >> used for PCI host bridge and we may have overlapping IO ranges for >> each host bridge, which is compatible with PCIe specification and PCIe >> ACPI description. >> > > That is fine. Under UEFI, these will translate to non-overlapping I/O > spaces in the CPU's view. Under the OS, this could be totally > different. > I fully agree with the PciRootBridgeLib interface change to support MMIO offset. I guess that you will propose to add a AddrTranslationOffset to PCI_ROOT_BRIDGE structure. But I don't quite understand the needs of IO offset support. And further more, there was a discussion in the mailing list regarding how to utilize the Offset. There was 2 points about the meaning of Offset and we cannot agree with each other. Spec is not quite clear. 1. Offset = Host - Device 2. Offset = Device - Host > > For example, > > RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff > RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff > > This is very similar to how MMIO translation works, and makes I/O > devices behind the host bridges uniquely addressable for drivers. > > For our understanding, could you share the host bridge configuration > that you are targetting? > > Thanks, > Ard. > > >> How about adding a flag to indicate whether port IO is translated to >> real port IO space or system memory space? >> >> Thanks and regards, >> >> Heyi >> >>>> 2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get >>>> translation of the corresponding aperture, add the translation to the >>>> input address, and then call CpuIo2 protocol; IO access will also be >>>> converted to memory access if IO translation exists. >>>> >>> >>> Again, why is this necessary? A host bridge that implements a non 1:1 >>> translation for port I/O ranges may be part of a system that has >>> native port I/O, and so the translation should be based on that. >>> >>>> 3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in >>>> the discriptor. >>>> >>> >>> Indeed. Note that this has been a source of much confusion lately, >>> should we should discuss this carefully before spending time on an >>> implementation. >>> >>>> If it makes sense, then I'll continue to prepare the formal patch. >>>> >>>> Any comments? >>>> >>>> Thanks, >>>> >>>> Gary (Heyi Guo) >>>> >>>> Cc: Star Zeng <star.zeng@intel.com> >>>> Cc: Eric Dong <eric.dong@intel.com> >>>> Cc: Ruiyu Ni <ruiyu.ni@intel.com> >>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >>>> Cc: Jason Zhang <jason.zhang@linaro.org> >>>> >>>> --- >>>> MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + >>>> 1 file changed, 1 insertion(+) >>>> >>>> diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >>>> index d42e9ec..b9e8c0f 100644 >>>> --- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h >>>> +++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h >>>> @@ -22,6 +22,7 @@ >>>> typedef struct { >>>> UINT64 Base; >>>> UINT64 Limit; >>>> + UINT64 Translation; >>>> } PCI_ROOT_BRIDGE_APERTURE; >>>> >>>> typedef struct { >>>> -- >>>> 2.7.4 >>>> -- Thanks, Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support 2017-12-21 9:43 ` Ni, Ruiyu @ 2017-12-21 11:32 ` Guo Heyi 0 siblings, 0 replies; 16+ messages in thread From: Guo Heyi @ 2017-12-21 11:32 UTC (permalink / raw) To: Ni, Ruiyu Cc: Ard Biesheuvel, gary guo, linaro-uefi, edk2-devel@lists.01.org, Zeng, Star, Dong, Eric, Jason Zhang On Thu, Dec 21, 2017 at 05:43:17PM +0800, Ni, Ruiyu wrote: > On 12/20/2017 11:26 PM, Ard Biesheuvel wrote: > >On 20 December 2017 at 15:17, gary guo <heyi.guo@linaro.org> wrote: > >>On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: > >>>Hi Heyi, > >>> > >>>On 20 December 2017 at 08:21, Heyi Guo <heyi.guo@linaro.org> wrote: > >>>>PCIe on some ARM platforms requires address translation, not only for > >>>>legacy IO access, but also for 32bit memory BAR access as well. There > >>>>will be "Address Translation Unit" or something similar in PCI host > >>>>bridges to translation CPU address to PCI address and vice versa. So > >>>>we think it may be useful to add address translation support to the > >>>>generic PCI host bridge driver. > >>>> > >>> > >>>I agree. While unusual on a PC, it is quite common on other > >>>architectures to have more complex non 1:1 topologies, which currently > >>>require a forked PciHostBridgeDxe driver with local changes applied. > >>> > >>>>This RFC only contains one minor change to the definition of > >>>>PciHostBridgeLib, and there certainly will be a lot of other changes > >>>>to make it work, including: > >>>> > >>>>1. Use CPU address for GCD space add and allocate operations, instead > >>>>of PCI address; also IO space will be changed to memory space if > >>>>translation exists. > >>>> > >>> > >>>For I/O space, the translation should simply be applied to the I/O > >>>range. I don't think it makes sense to use memory space here, given > >>>that it is specific to architectures that lack native port I/O. > >>> > >> > >>I made an assumption here that platforms supporting real port IO space > >>do not need address translation, like IA32 and X64, and port IO > >>translation implies the platform does not support real port IO space. > >> > > > >This may be a reasonable assumption. But I still think it is better > >not to encode any assumptions in the first place. > > > >>Indeed the assumption is not so "generic", so I'll agree if you > >>recommend to support IO to IO translation as well. But I still hope to > >>have IO to memory translation support in PCI host bridge driver, > >>rather than in CPU IO protocol, since the faked IO space might only be > >>used for PCI host bridge and we may have overlapping IO ranges for > >>each host bridge, which is compatible with PCIe specification and PCIe > >>ACPI description. > >> > > > >That is fine. Under UEFI, these will translate to non-overlapping I/O > >spaces in the CPU's view. Under the OS, this could be totally > >different. > > > > I fully agree with the PciRootBridgeLib interface change to support > MMIO offset. I guess that you will propose to add a > AddrTranslationOffset to PCI_ROOT_BRIDGE structure. > > But I don't quite understand the needs of IO offset support. > > And further more, there was a discussion in the mailing list > regarding how to utilize the Offset. There was 2 points > about the meaning of Offset and we cannot agree with each > other. Spec is not quite clear. > 1. Offset = Host - Device > 2. Offset = Device - Host On our platforms, we use the 1st one and it seems Linux works well with it. And I think normally we will translate high CPU address into low PCI address (less than 4G) to make it compatible with some old PCIe devices, and in this scenario Offset will always be positive. Thanks, Gary > > > > > >For example, > > > >RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff > >RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff > > > >This is very similar to how MMIO translation works, and makes I/O > >devices behind the host bridges uniquely addressable for drivers. > > > >For our understanding, could you share the host bridge configuration > >that you are targetting? > > > >Thanks, > >Ard. > > > > > >>How about adding a flag to indicate whether port IO is translated to > >>real port IO space or system memory space? > >> > >>Thanks and regards, > >> > >>Heyi > >> > >>>>2. RootBridgeIoMemRead/Write, RootBridgeIoRead/Write need to get > >>>>translation of the corresponding aperture, add the translation to the > >>>>input address, and then call CpuIo2 protocol; IO access will also be > >>>>converted to memory access if IO translation exists. > >>>> > >>> > >>>Again, why is this necessary? A host bridge that implements a non 1:1 > >>>translation for port I/O ranges may be part of a system that has > >>>native port I/O, and so the translation should be based on that. > >>> > >>>>3. RootBridgeIoConfiguration needs to fill AddrTranslationOffset in > >>>>the discriptor. > >>>> > >>> > >>>Indeed. Note that this has been a source of much confusion lately, > >>>should we should discuss this carefully before spending time on an > >>>implementation. > >>> > >>>>If it makes sense, then I'll continue to prepare the formal patch. > >>>> > >>>>Any comments? > >>>> > >>>>Thanks, > >>>> > >>>>Gary (Heyi Guo) > >>>> > >>>>Cc: Star Zeng <star.zeng@intel.com> > >>>>Cc: Eric Dong <eric.dong@intel.com> > >>>>Cc: Ruiyu Ni <ruiyu.ni@intel.com> > >>>>Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > >>>>Cc: Jason Zhang <jason.zhang@linaro.org> > >>>> > >>>>--- > >>>> MdeModulePkg/Include/Library/PciHostBridgeLib.h | 1 + > >>>> 1 file changed, 1 insertion(+) > >>>> > >>>>diff --git a/MdeModulePkg/Include/Library/PciHostBridgeLib.h b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >>>>index d42e9ec..b9e8c0f 100644 > >>>>--- a/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >>>>+++ b/MdeModulePkg/Include/Library/PciHostBridgeLib.h > >>>>@@ -22,6 +22,7 @@ > >>>> typedef struct { > >>>> UINT64 Base; > >>>> UINT64 Limit; > >>>>+ UINT64 Translation; > >>>> } PCI_ROOT_BRIDGE_APERTURE; > >>>> > >>>> typedef struct { > >>>>-- > >>>>2.7.4 > >>>> > > > -- > Thanks, > Ray ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2018-01-02 15:04 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-12-20 8:21 [RFC] MdeModulePkg/PciHostBridge: Add address translation support Heyi Guo 2017-12-20 9:13 ` Ard Biesheuvel 2017-12-20 15:17 ` gary guo 2017-12-20 15:26 ` Ard Biesheuvel 2017-12-21 8:27 ` Guo Heyi 2017-12-21 8:32 ` Ard Biesheuvel 2017-12-21 9:14 ` Guo Heyi 2017-12-21 9:48 ` Ni, Ruiyu 2017-12-21 9:52 ` Ard Biesheuvel 2017-12-21 9:59 ` Ni, Ruiyu 2017-12-21 10:07 ` Ard Biesheuvel 2017-12-26 6:50 ` Guo Heyi 2018-01-02 7:56 ` Ni, Ruiyu 2018-01-02 15:09 ` gary guo 2017-12-21 9:43 ` Ni, Ruiyu 2017-12-21 11:32 ` Guo Heyi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox