Re: [edk2-devel] [RFC] Ordering of Arm PCI ECAM and MMIO operations

["Michael Brown" <mcb30@ipxe.org>]

On 01/11/2023 09:56, Ard Biesheuvel wrote:
> On Wed, 1 Nov 2023 at 03:09, Pedro Falcato <pedro.falcato@gmail.com> wrote:
>> On Wed, Nov 1, 2023 at 12:40 AM Joe L <jlotwo@gmail.com> wrote:
>>> Our CMN TRM showcases an example where ECAM and MMIO are two different regions in the HN-I SAM. The implication is that we would expect a DSB between the ECAM write and MMIO read. I'm asking our Open Source Software group to confirm that standard PCIe software is generally expected to be aware of the need for a DSB--but my impression from talking to some of our hardware engineers is that that is indeed the expectation.
> 
> <snip>
> 
> 1) as per the architecture (as interpreted by the ARM architects), a
> DSB is required to ensure that the side effects of enabling a MMIO BAR
> in the PCI config space are sufficiently observable to memory accesses
> to that BAR that appear after the PCI config space access in the
> program.

It's possibly worth mentioning what the PCIe specification requires in 
terms of ECAM ordering:

   "As an example, software may wish to configure a device Function’s
    Base Address register by writing to the device using the ECAM,
    and then read a location in the memory-mapped range described by
    this Base Address register. If the software were to issue the
    memory-mapped read before the ECAM write was completed, it would
    be possible for the memory-mapped read to be re-ordered and arrive
    at the device before the Configuration Write Request, thus causing
    unpredictable results.

    To avoid this problem, processor and host bridge implementations must
    ensure that a method exists for the software to determine when the
    write using the ECAM is completed by the completer."

By my reading, the PCIe specification seems to therefore require 
something stronger than an ordering guarantee: it requires the ability 
for software to make a standalone determination that the write has 
*completed*, independent of the existence of any subsequent I/O operations.

As a practical example of when this might be relevant: software could be 
writing to device configuration space to disable bus mastering as part 
of a reset or shutdown sequence, in order to guarantee that the device 
will initiate no further DMA operations and that any DMA buffers 
allocated to the device can be freed and reused.  In this situation, 
there may be no subsequent MMIO read or write to the device, and so 
there is no way to rely upon an ordering guarantee to satisfy the 
requirement.

Any solution involving ordering guarantees can therefore mask the 
problem in some situations, but cannot solve it.

The PCIe specification does not mandate that any particular mechanism be 
used, but it does require that the processor and/or host bridge provides 
*some* mechanism for software to determine that the ECAM write has 
completed.

What mechanism does ARM (or the host bridge) provide to determine 
completion of an ECAM write?

Thanks,

Michael



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#110477): https://edk2.groups.io/g/devel/message/110477
Mute This Topic: https://groups.io/mt/102310377/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-