From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 1E60D941835 for ; Wed, 1 Nov 2023 13:24:03 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=HJ2Ed8BSRhdRzFFA2fix8Ey9LkGiTMx35ePn4kzMtv4=; c=relaxed/simple; d=groups.io; h=Message-ID:Date:MIME-Version:User-Agent:To:Cc:References:From:Subject:In-Reply-To:Feedback-ID:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1698845042; v=1; b=UcpX7zTMvOF0jcA9/ZkFSAzDYwOOb8Cr7tbcH3wkA88JH6pA5IWLM7MehLLi1beA5WuU5fs7 fEIDN3sv9QPi8JTu/QJvz6KzkPZuurFCqGuyiMG+h4tXRyxdHVXEgpRIsNz5cSpI7BCEHk280qH Z4RcT1qKsfGWVUpUjKgg0qOc= X-Received: by 127.0.0.2 with SMTP id 7gv9YY7687511xyU5zZ1GkIQ; Wed, 01 Nov 2023 06:24:02 -0700 X-Received: from a7-18.smtp-out.eu-west-1.amazonses.com (a7-18.smtp-out.eu-west-1.amazonses.com [54.240.7.18]) by mx.groups.io with SMTP id smtpd.web11.7039.1698845041791169753 for ; Wed, 01 Nov 2023 06:24:02 -0700 Message-ID: <0102018b8b0e1ca8-ce15d29c-2c0c-4982-9c45-be42ae53a1d6-000000@eu-west-1.amazonses.com> Date: Wed, 1 Nov 2023 13:23:59 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: devel@edk2.groups.io, ardb@kernel.org Cc: pedro.falcato@gmail.com, jlotwo@gmail.com, Leif Lindholm , Sami Mujawar , Ray Ni References: <0102018b8ad8c2e7-187ec6ab-01ca-4b47-8c19-aa2748065b06-000000@eu-west-1.amazonses.com> From: "Michael Brown" Subject: Re: [edk2-devel] [RFC] Ordering of Arm PCI ECAM and MMIO operations In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, URIBL_DBL_BLOCKED_OPENDNS,URIBL_ZEN_BLOCKED_OPENDNS autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on blyat.fensystems.co.uk Feedback-ID: 1.eu-west-1.fspj4M/5bzJ9NLRzJP0PaxRwxrpZqiDQJ1IF94CF2TA=:AmazonSES X-SES-Outgoing: 2023.11.01-54.240.7.18 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mcb30@ipxe.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: nZAvHQCylW85yhcUmVj079pEx7686176AA= Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=UcpX7zTM; dmarc=none; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io On 01/11/2023 12:51, Ard Biesheuvel wrote: > On Wed, 1 Nov 2023 at 13:25, Michael Brown wrote: >> By my reading, the PCIe specification seems to therefore require >> something stronger than an ordering guarantee: it requires the ability >> for software to make a standalone determination that the write has >> *completed*, independent of the existence of any subsequent I/O operatio= ns. >=20 > indeed, thanks for bringing this up. >=20 >> The PCIe specification does not mandate that any particular mechanism be >> used, but it does require that the processor and/or host bridge provides >> *some* mechanism for software to determine that the ECAM write has >> completed. >> >> What mechanism does ARM (or the host bridge) provide to determine >> completion of an ECAM write? >=20 > A MMIO read of the same location should ensure that the MMIO write has > completed by the time the read returns. Not sure whether or not there > are any other requirements (e.g., wrt.the size of the read vs the size > of the write). That seems to suggest that a logical PCIe configuration space write=20 operation using ECAM should probably always comprise: 1. ECAM write 2. ECAM read from the same location (using the same size) If reads are not allowed to have side effects (e.g. read-clear=20 registers) then this seems safe. The PCIe specification "Configuration=20 Register Types" list comprises (in version 3.0, at least): HwInit - read-only, no read side effects RO - read-only, no read side effects RW - read-write, no read side effects RW1C - write 1 to clear bits, no read side effects ROS - read-only, no read side effects RWS - read-write, no read side effects RW1CS - write 1 to clear bits, no read side effects RsvdP - read-write, no read side effects RsvdZ - read-write, no read side effects So, unless newer versions of the PCIe specification have allowed for the=20 existence of configuration register types with read side effects, then=20 the approach of always reading back from ECAM seems to be safe for any=20 conforming PCIe device. I would therefore suggest that all ECAM driver implementation functions=20 in EDK2 (e.g. PciExpressWrite32(), PciExpressOr32(),=20 PciSegmentWrite32(), etc) be updated to add the relevant ECAM read=20 following the write operation. PCI configuration space writes are never fast-path operations (in any=20 sane hardware), and so the delay introduced by the read should not be=20 significant. Does this seem like a sensible solution? Thanks, Michael -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#110480): https://edk2.groups.io/g/devel/message/110480 Mute This Topic: https://groups.io/mt/102310377/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-