Re: [edk2-devel] [RFC] Ordering of Arm PCI ECAM and MMIO operations

["Ard Biesheuvel" <ardb@kernel.org>]

On Wed, 1 Nov 2023 at 13:25, Michael Brown <mcb30@ipxe.org> wrote:
>
> On 01/11/2023 09:56, Ard Biesheuvel wrote:
> > On Wed, 1 Nov 2023 at 03:09, Pedro Falcato <pedro.falcato@gmail.com> wrote:
> >> On Wed, Nov 1, 2023 at 12:40 AM Joe L <jlotwo@gmail.com> wrote:
> >>> Our CMN TRM showcases an example where ECAM and MMIO are two different regions in the HN-I SAM. The implication is that we would expect a DSB between the ECAM write and MMIO read. I'm asking our Open Source Software group to confirm that standard PCIe software is generally expected to be aware of the need for a DSB--but my impression from talking to some of our hardware engineers is that that is indeed the expectation.
> >
> > <snip>
> >
> > 1) as per the architecture (as interpreted by the ARM architects), a
> > DSB is required to ensure that the side effects of enabling a MMIO BAR
> > in the PCI config space are sufficiently observable to memory accesses
> > to that BAR that appear after the PCI config space access in the
> > program.
>
> It's possibly worth mentioning what the PCIe specification requires in
> terms of ECAM ordering:
>
>    "As an example, software may wish to configure a device Function’s
>     Base Address register by writing to the device using the ECAM,
>     and then read a location in the memory-mapped range described by
>     this Base Address register. If the software were to issue the
>     memory-mapped read before the ECAM write was completed, it would
>     be possible for the memory-mapped read to be re-ordered and arrive
>     at the device before the Configuration Write Request, thus causing
>     unpredictable results.
>
>     To avoid this problem, processor and host bridge implementations must
>     ensure that a method exists for the software to determine when the
>     write using the ECAM is completed by the completer."
>
> By my reading, the PCIe specification seems to therefore require
> something stronger than an ordering guarantee: it requires the ability
> for software to make a standalone determination that the write has
> *completed*, independent of the existence of any subsequent I/O operations.
>

indeed, thanks for bringing this up.

> As a practical example of when this might be relevant: software could be
> writing to device configuration space to disable bus mastering as part
> of a reset or shutdown sequence, in order to guarantee that the device
> will initiate no further DMA operations and that any DMA buffers
> allocated to the device can be freed and reused.  In this situation,
> there may be no subsequent MMIO read or write to the device, and so
> there is no way to rely upon an ordering guarantee to satisfy the
> requirement.
>
> Any solution involving ordering guarantees can therefore mask the
> problem in some situations, but cannot solve it.
>
> The PCIe specification does not mandate that any particular mechanism be
> used, but it does require that the processor and/or host bridge provides
> *some* mechanism for software to determine that the ECAM write has
> completed.
>
> What mechanism does ARM (or the host bridge) provide to determine
> completion of an ECAM write?
>

A MMIO read of the same location should ensure that the MMIO write has
completed by the time the read returns. Not sure whether or not there
are any other requirements (e.g., wrt.the size of the read vs the size
of the write).


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#110478): https://edk2.groups.io/g/devel/message/110478
Mute This Topic: https://groups.io/mt/102310377/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-