From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web12.514.1614280986602926549 for ; Thu, 25 Feb 2021 11:23:06 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=V1ZsBEhw; spf=pass (domain: redhat.com, ip: 170.10.133.124, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1614280985; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4x2og+YR6A8kBNgvVXWa9JMR+RfAHeI20fbfhheriCM=; b=V1ZsBEhwL+opE9z3nAezglOYsxXQz0GZyGqi6/V2ifeTsXlizyIw0l2ZMKWxRbyehJBAAJ Gq7nBKKq4bCOoXLNMdr72XrTbL0YRvB4cYTGmZUIe6O2WI1QOpJqgA4H5bOSQv0Vicl6pZ jep0WWiB1u1g8cV2LGlYCCh02iEBfO0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-40-lQabpmLJOcar4_bAGmAdNg-1; Thu, 25 Feb 2021 14:23:03 -0500 X-MC-Unique: lQabpmLJOcar4_bAGmAdNg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7C157C73A4; Thu, 25 Feb 2021 19:23:01 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-115-188.ams2.redhat.com [10.36.115.188]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7744319D9C; Thu, 25 Feb 2021 19:22:59 +0000 (UTC) Subject: Re: [edk2-devel] [PATCH v8 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject To: Ankur Arora , devel@edk2.groups.io Cc: imammedo@redhat.com, boris.ostrovsky@oracle.com, Jordan Justen , Ard Biesheuvel , Aaron Young References: <20210222071928.1401820-1-ankur.a.arora@oracle.com> <20210222071928.1401820-10-ankur.a.arora@oracle.com> <0f373648-0d88-6151-f2f6-aa185041b576@redhat.com> From: "Laszlo Ersek" Message-ID: Date: Thu, 25 Feb 2021 20:22:58 +0100 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=lersek@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit On 02/24/21 04:44, Ankur Arora wrote: > On 2021-02-23 1:39 p.m., Laszlo Ersek wrote: >> On 02/22/21 08:19, Ankur Arora wrote: >>> +    UINT32 Idx; >>> + >>> +    for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) { >>> +      UINT64 QemuSelector; >>> + >>> +      QemuSelector = mCpuHotEjectData->QemuSelectorMap[Idx]; >>> + >>> +      if (QemuSelector != CPU_EJECT_QEMU_SELECTOR_INVALID) { >>> +        // >>> +        // This to-be-ejected-CPU has already received the BSP's SMI >>> exit >>> +        // signal and, will execute SmmCpuFeaturesRendezvousExit() >>> +        // followed by this callback or is already waiting in the >>> +        // CpuSleep() loop below. >>> +        // >>> +        // Tell QEMU to context-switch it out. >>> +        // >>> +        QemuCpuhpWriteCpuSelector (mMmCpuIo, (UINT32) QemuSelector); >>> +        QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECT); >>> + >>> +        // >>> +        // We need a compiler barrier here to ensure that the compiler >>> +        // does not reorder the CpuStatus and QemuSelectorMap[Idx] >>> stores. >>> +        // >>> +        // A store fence is not strictly necessary on x86 which has >>> +        // TSO; however, both of these stores are in different >>> address spaces >>> +        // so also add a Store Fence here. >>> +        // >>> +        MemoryFence (); >> >> (6) I wonder if this compiler barrier + comment block are helpful. >> Paraphrasing your (ex-)colleague Liran, if MMIO and IO Port accessors >> didn't contain built-in fences, all hell would break lose. We're using >> EFI_MM_CPU_IO_PROTOCOL for IO Port accesses. I think we should be safe >> ordering-wise, even without an explicit compiler barrier here. >> >> To me personally, this particular fence only muddies the picture -- >> where we already have an acquire memory fence and a store memory fence >> to couple with each other. >> >> I'd recommend removing this. (If you disagree, I'm willing to listen to >> arguments, of course!) > > You are right that we don't need a memory fence here -- given that there > is an implicit fence due to the MMIO. > > As for the compiler fence, I'm just now re-looking at handlers in > EFI_MM_CPU_IO_PROTOCOL and they do seem to include a compiler barrier. > > So I agree with you that we have all the fences that we need. However, > I do think it's a good idea to document both of these here. OK. >>> diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c >>> b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c >>> index 99988285b6a2..ddfef05ee6cf 100644 >>> --- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c >>> +++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c >>> @@ -472,6 +472,37 @@ SmmCpuFeaturesRendezvousExit ( >>>     // (PcdCpuMaxLogicalProcessorNumber > 1), and hot-eject is needed >>>     // in this SMI exit (otherwise mCpuHotEjectData->Handler is not >>> armed.) >>>     // >>> +  // mCpuHotEjectData itself is stable once setup so it can be >>> +  // dereferenced without needing any synchronization, >>> +  // but, mCpuHotEjectData->Handler is updated on the BSP in the >>> +  // ongoing SMI iteration at two places: >>> +  // >>> +  // - UnplugCpus() where the BSP determines if a CPU is under ejection >>> +  //   or not. As the comment where mCpuHotEjectData->Handler is set-up >>> +  //   describes any such updates are guaranteed to be >>> ordered-before the >>> +  //   dereference below. >>> +  // >>> +  // - EjectCpu() (which is called via the Handler below), on the BSP >>> +  //   updates mCpuHotEjectData->Handler once it is done with all >>> ejections. >>> +  // >>> +  //   The CPU under ejection: might be executing anywhere between the >>> +  //   "AllCpusInSync" exit loop in SmiRendezvous() to about to >>> +  //   dereference the Handler field. >>> +  //   Given that the BSP ensures that this store only happens after >>> all >>> +  //   CPUs under ejection have been ejected, this CPU would never see >>> +  //   the after value. >>> +  //   (Note that any CPU that is already executing the CpuSleep() loop >>> +  //   below never raced any updates and always saw the before value.) >>> +  // >>> +  //   CPUs not-under ejection: might see either value of the Handler >>> +  //   which is fine, because the Handler is a NOP for CPUs not-under >>> +  //   ejection. >>> +  // >>> +  //   Lastly, note that we are also guaranteed that any dereferencing >>> +  //   CPU only sees the before or after value and not an intermediate >>> +  //   value. This is because mCpuHotEjectData->Handler is aligned at a >>> +  //   natural boundary. >>> +  // >>>       if (mCpuHotEjectData != NULL) { >>>       CPU_HOT_EJECT_HANDLER Handler; >>> >> >> (8) I can't really put my finger on it, I just feel that repeating >> (open-coding) this wall of text here is not really productive. > > Part of the reason I wanted to document this here was to get your > opinion on it and figure out how much of it is useful and how > much might be overkill. > >> >> Do you think that, after you add the "acquire memory fence" comment in >> patch #7, we could avoid most of the text here? I think we should only >> point out (in patch #7) the "release fence" that the logic here pairs >> with.> If you really want to present it all from both perspectives, I >> guess I'm >> OK with that, but then I believe we should drop the last paragraph at >> least (see point (4)). > > Rereading it after a gap of a few days and given that most of this is > just a repeat, I'm also tending towards overkill. I think a comment > talking about acquire/release pairing is useful. Rest of it can probably > be met with just a pointer towards the comment in EjectCpus(). Does that > make sense? Yes, absolutely. Short comment + pointer to the "other half" (which has the large comment too) seem best. Thanks Laszlo