From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id A37117803D1 for ; Wed, 1 Nov 2023 16:41:22 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=n46smZjWyxcBzdl1rwgsIIuVCwGaYURJjtueLh/Kkko=; c=relaxed/simple; d=groups.io; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:To:Cc:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type; s=20140610; t=1698856881; v=1; b=AY6nXpQNHc3vR03Wy9PyLyHDFQNaq1IoHqlaBbYnTvZJzK2I6RIbORKy4vel9M/w84fiA2nQ X0D0jsbIeWSWWrSSz9Mjq25qCbGblLLOgb9Jn9v708wzBzSRxChnXeS7xcarBcKfY/2PdtnjL72 8gKP5VTHBG5Dsk1ErQcR3wSI= X-Received: by 127.0.0.2 with SMTP id HlQhYY7687511xekGhB5aPHD; Wed, 01 Nov 2023 09:41:21 -0700 X-Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mx.groups.io with SMTP id smtpd.web10.12524.1698856880720189296 for ; Wed, 01 Nov 2023 09:41:20 -0700 X-Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1305461252 for ; Wed, 1 Nov 2023 16:41:20 +0000 (UTC) X-Received: by smtp.kernel.org (Postfix) with ESMTPSA id B1B33C433CB for ; Wed, 1 Nov 2023 16:41:19 +0000 (UTC) X-Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-2c4fe37f166so95435921fa.1 for ; Wed, 01 Nov 2023 09:41:19 -0700 (PDT) X-Gm-Message-State: GSgh4qbCofGdTfkVdf86TYwJx7686176AA= X-Google-Smtp-Source: AGHT+IHB281FLisxHRNU9aGuiQTCRbrb83Zf/UGIQpcV4h0/tMX6aQxZ2+bONUrt2NOL1aOv75o5PuCkm/t/DWYbiII= X-Received: by 2002:a2e:97ce:0:b0:2c6:edfd:658e with SMTP id m14-20020a2e97ce000000b002c6edfd658emr92642ljj.52.1698856877848; Wed, 01 Nov 2023 09:41:17 -0700 (PDT) MIME-Version: 1.0 References: <0102018b8ad8c2e7-187ec6ab-01ca-4b47-8c19-aa2748065b06-000000@eu-west-1.amazonses.com> <0102018b8b0e1ca8-ce15d29c-2c0c-4982-9c45-be42ae53a1d6-000000@eu-west-1.amazonses.com> In-Reply-To: <0102018b8b0e1ca8-ce15d29c-2c0c-4982-9c45-be42ae53a1d6-000000@eu-west-1.amazonses.com> From: "Ard Biesheuvel" Date: Wed, 1 Nov 2023 17:41:06 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [edk2-devel] [RFC] Ordering of Arm PCI ECAM and MMIO operations To: Michael Brown Cc: devel@edk2.groups.io, pedro.falcato@gmail.com, jlotwo@gmail.com, Leif Lindholm , Sami Mujawar , Ray Ni Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,ardb@kernel.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: Content-Type: text/plain; charset="UTF-8" X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=AY6nXpQN; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io On Wed, 1 Nov 2023 at 14:24, Michael Brown wrote: > > On 01/11/2023 12:51, Ard Biesheuvel wrote: > > On Wed, 1 Nov 2023 at 13:25, Michael Brown wrote: > >> By my reading, the PCIe specification seems to therefore require > >> something stronger than an ordering guarantee: it requires the ability > >> for software to make a standalone determination that the write has > >> *completed*, independent of the existence of any subsequent I/O operations. > > > > indeed, thanks for bringing this up. > > > >> The PCIe specification does not mandate that any particular mechanism be > >> used, but it does require that the processor and/or host bridge provides > >> *some* mechanism for software to determine that the ECAM write has > >> completed. > >> > >> What mechanism does ARM (or the host bridge) provide to determine > >> completion of an ECAM write? > > > > A MMIO read of the same location should ensure that the MMIO write has > > completed by the time the read returns. Not sure whether or not there > > are any other requirements (e.g., wrt.the size of the read vs the size > > of the write). > > That seems to suggest that a logical PCIe configuration space write > operation using ECAM should probably always comprise: > > 1. ECAM write > 2. ECAM read from the same location (using the same size) > > If reads are not allowed to have side effects (e.g. read-clear > registers) then this seems safe. The PCIe specification "Configuration > Register Types" list comprises (in version 3.0, at least): > > HwInit - read-only, no read side effects > > RO - read-only, no read side effects > > RW - read-write, no read side effects > > RW1C - write 1 to clear bits, no read side effects > > ROS - read-only, no read side effects > > RWS - read-write, no read side effects > > RW1CS - write 1 to clear bits, no read side effects > > RsvdP - read-write, no read side effects > > RsvdZ - read-write, no read side effects > > So, unless newer versions of the PCIe specification have allowed for the > existence of configuration register types with read side effects, then > the approach of always reading back from ECAM seems to be safe for any > conforming PCIe device. > > I would therefore suggest that all ECAM driver implementation functions > in EDK2 (e.g. PciExpressWrite32(), PciExpressOr32(), > PciSegmentWrite32(), etc) be updated to add the relevant ECAM read > following the write operation. > > PCI configuration space writes are never fast-path operations (in any > sane hardware), and so the delay introduced by the read should not be > significant. > > Does this seem like a sensible solution? > I think that would be reasonable, but it needs to be implemented at the correct level of abstraction. We have some plumbing to iterate over ranges to do what basically comes down to memcpy and memset on MMIO ranges, and I don't think we want the readback on every store in a sequence of multiple. RootBridgeIoPciAccess() looks like a reasonable location to insert a readback of the last access if it was a write. -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#110486): https://edk2.groups.io/g/devel/message/110486 Mute This Topic: https://groups.io/mt/102310377/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-