Re: [edk2-devel] [PATCH] MdeModulePkg/PciHostBridgeDxe: Add readback after final Cfg-Write

["Michael Brown" <mcb30@ipxe.org>]

On 03/11/2023 16:03, Kinney, Michael D wrote:
> PCI Configuration read/write operations are non-posted, so the PCI
> Configuration write operation should complete without the need for
> an additional transaction.  If you are seeing an issue, then it may
> be in the implementation of the PciLib/PciSegmentLib that is not
> guaranteeing this non-posted behavior.

 From the PCIe spec:

   "The ECAM converts memory transactions from the host CPU into
    Configuration Requests on the PCI Express fabric. This conversion
    potentially creates ordering problems for the software, because
    writes to memory addresses are typically posted transactions but
    writes to Configuration Space are not posted on the PCI Express
    fabric."

My understanding from the above is that the posted write occurs at the 
level of the CPU performing a memory write to the ECAM window.  By the 
time this write reaches the ECAM and becomes a non-posted write 
transaction within the downstream PCIe fabric, the CPU has already moved 
on and is not waiting for completion of the ECAM memory write.

Reading back from the PCI configuration register will cause the CPU to 
wait until the read from the ECAM window has completed, which cannot 
happen until the corresponding downstream PCI configuration read has 
completed.  Since the CPU will (with appropriate barrier operations) not 
reorder the memory read ahead of the preceding memory write, the overall 
effect is to guarantee that the memory write has reached the ECAM, and 
that the memory write reached the ECAM before the subsequent memory read 
from the same location.

There is an implicit assumption in the above that the ECAM and PCIe 
fabric will themselves not reorder the PCI configuration read ahead of 
the PCI configuration write.  Unfortunately, on a closer reading of the 
spec, this may not be a valid assumption: other parts of the PCIe spec 
state that non-posted transactions (e.g. configuration reads and writes) 
may be freely reordered.

It seems slightly insane that PCIe bridges would be allowed to 
arbitrarily reorder configuration cycles to downstream devices, but 
that's what the spec seems to state.  The upshot seems to be that:

a) software is entirely responsible for ensuring that PCI configuration 
writes via ECAM have completed, and

b) software has no available mechanism for ensuring that PCI 
configuration writes via ECAM have completed.

Ard: any alternative suggestions on ways we can wait for completion, 
given that even reading back from the PCI configuration register is 
apparently not guaranteed?

Thanks,

Michael



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#110646): https://edk2.groups.io/g/devel/message/110646
Mute This Topic: https://groups.io/mt/102354842/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-