From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 31011D8050D for ; Wed, 1 Nov 2023 12:52:12 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=jsq24Ek7dulcMMLglhYaGPX/oVHyVUaQ9e7VFoCM36o=; c=relaxed/simple; d=groups.io; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:To:Cc:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type:Content-Transfer-Encoding; s=20140610; t=1698843130; v=1; b=Bnm62PPOJM7SNQ4L2osgytPh3ll3oT0bFyggLu9Q4aJgeC2+YOfVipMbG519kNfGFtVVuLdb NzMnYzdqn7JmGfpXIFQX59OYkDsaEhqM9FwJIsj+YpTQaI6b/M3pb6wKdAlZc8OAUNiR+h9BqPL GHE9HVRDIuz/J2XKzXGAxT7E= X-Received: by 127.0.0.2 with SMTP id WUjHYY7687511xR1WlCHvFLF; Wed, 01 Nov 2023 05:52:10 -0700 X-Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mx.groups.io with SMTP id smtpd.web10.6391.1698843130199793953 for ; Wed, 01 Nov 2023 05:52:10 -0700 X-Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id B1F4461215 for ; Wed, 1 Nov 2023 12:52:09 +0000 (UTC) X-Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83EF7C433C7 for ; Wed, 1 Nov 2023 12:52:09 +0000 (UTC) X-Received: by mail-lf1-f44.google.com with SMTP id 2adb3069b0e04-50797cf5b69so9251600e87.2 for ; Wed, 01 Nov 2023 05:52:09 -0700 (PDT) X-Gm-Message-State: AspHy2mhev5EIRRCbJu0gXopx7686176AA= X-Google-Smtp-Source: AGHT+IFkUP7it0BHQsupAVdz80a2A7UMiFnzfVq8pjgrZqluDoUtWvU8BCdZQM/4hgKT1h1iN5liRnxWFRM4TFQI7Tc= X-Received: by 2002:ac2:5185:0:b0:503:7dd:7ebd with SMTP id u5-20020ac25185000000b0050307dd7ebdmr10142379lfi.24.1698843127708; Wed, 01 Nov 2023 05:52:07 -0700 (PDT) MIME-Version: 1.0 References: <0102018b8ad8c2e7-187ec6ab-01ca-4b47-8c19-aa2748065b06-000000@eu-west-1.amazonses.com> In-Reply-To: <0102018b8ad8c2e7-187ec6ab-01ca-4b47-8c19-aa2748065b06-000000@eu-west-1.amazonses.com> From: "Ard Biesheuvel" Date: Wed, 1 Nov 2023 13:51:56 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [edk2-devel] [RFC] Ordering of Arm PCI ECAM and MMIO operations To: Michael Brown Cc: devel@edk2.groups.io, pedro.falcato@gmail.com, jlotwo@gmail.com, Leif Lindholm , Sami Mujawar , Ray Ni Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,ardb@kernel.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=Bnm62PPO; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) On Wed, 1 Nov 2023 at 13:25, Michael Brown wrote: > > On 01/11/2023 09:56, Ard Biesheuvel wrote: > > On Wed, 1 Nov 2023 at 03:09, Pedro Falcato wr= ote: > >> On Wed, Nov 1, 2023 at 12:40=E2=80=AFAM Joe L wrote= : > >>> Our CMN TRM showcases an example where ECAM and MMIO are two differen= t regions in the HN-I SAM. The implication is that we would expect a DSB be= tween the ECAM write and MMIO read. I'm asking our Open Source Software gro= up to confirm that standard PCIe software is generally expected to be aware= of the need for a DSB--but my impression from talking to some of our hardw= are engineers is that that is indeed the expectation. > > > > > > > > 1) as per the architecture (as interpreted by the ARM architects), a > > DSB is required to ensure that the side effects of enabling a MMIO BAR > > in the PCI config space are sufficiently observable to memory accesses > > to that BAR that appear after the PCI config space access in the > > program. > > It's possibly worth mentioning what the PCIe specification requires in > terms of ECAM ordering: > > "As an example, software may wish to configure a device Function=E2=80= =99s > Base Address register by writing to the device using the ECAM, > and then read a location in the memory-mapped range described by > this Base Address register. If the software were to issue the > memory-mapped read before the ECAM write was completed, it would > be possible for the memory-mapped read to be re-ordered and arrive > at the device before the Configuration Write Request, thus causing > unpredictable results. > > To avoid this problem, processor and host bridge implementations must > ensure that a method exists for the software to determine when the > write using the ECAM is completed by the completer." > > By my reading, the PCIe specification seems to therefore require > something stronger than an ordering guarantee: it requires the ability > for software to make a standalone determination that the write has > *completed*, independent of the existence of any subsequent I/O operation= s. > indeed, thanks for bringing this up. > As a practical example of when this might be relevant: software could be > writing to device configuration space to disable bus mastering as part > of a reset or shutdown sequence, in order to guarantee that the device > will initiate no further DMA operations and that any DMA buffers > allocated to the device can be freed and reused. In this situation, > there may be no subsequent MMIO read or write to the device, and so > there is no way to rely upon an ordering guarantee to satisfy the > requirement. > > Any solution involving ordering guarantees can therefore mask the > problem in some situations, but cannot solve it. > > The PCIe specification does not mandate that any particular mechanism be > used, but it does require that the processor and/or host bridge provides > *some* mechanism for software to determine that the ECAM write has > completed. > > What mechanism does ARM (or the host bridge) provide to determine > completion of an ECAM write? > A MMIO read of the same location should ensure that the MMIO write has completed by the time the read returns. Not sure whether or not there are any other requirements (e.g., wrt.the size of the read vs the size of the write). -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#110478): https://edk2.groups.io/g/devel/message/110478 Mute This Topic: https://groups.io/mt/102310377/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-