From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 876AAD801E7 for ; Tue, 30 Jul 2024 16:38:44 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=PAaz910zT8ZZAy2XSTi/C9v6GqpFJtiASvpv5eST5C0=; c=relaxed/simple; d=groups.io; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Resent-Date:Resent-From:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20240206; t=1722357524; v=1; b=tujAPO7BevUYAa7XllAg0VG99Wzs/maJs/p5jpr72W5HWZgs2qjzqouiky2+cURJjUEmdjCk todiVnM15x4Qa0eH2J8SQ4633GSQv5MBoReOo+VRqwrxaMbN9VQz5LyCcKgj2g9XndfHyKGHOyt zL022IHUURmcXMzFaSKAT/l8T8vifvjzzpSPQeoPSehc0NdrJxv8Uu73G+W/hnnUloFIfBiuvKz TpOyLA5MLVV4wbOXSoklsBFkt0gHsN0zaYYoeTjXatQCVQoDQWF4Do0JCnCBN/n82lE7889hKf1 CRKwreCqv7cuiXFsR68hyJJCfL13pkAiKLxfbM/6vzqqA== X-Received: by 127.0.0.2 with SMTP id 2chlYY7687511xGZwvvsnZjD; Tue, 30 Jul 2024 09:38:42 -0700 X-Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by mx.groups.io with SMTP id smtpd.web11.64843.1722009498993552827 for ; Fri, 26 Jul 2024 08:58:19 -0700 X-Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1fd9e70b592so6902405ad.3 for ; Fri, 26 Jul 2024 08:58:18 -0700 (PDT) X-Gm-Message-State: tCDJ7Law698Z1XGPXgtHN51wx7686176AA= X-Google-Smtp-Source: AGHT+IEzWgWgmQ4u6+GI9owAQy8KN7DHLGC31q+Pn771f206o2WSk9BdF0R6DLvorr77BcqfpQzGLA== X-Received: by 2002:a17:903:1245:b0:1fd:66fd:4dae with SMTP id d9443c01a7336-1ff04780cc8mr1179895ad.3.1722009497731; Fri, 26 Jul 2024 08:58:17 -0700 (PDT) X-Received: from localhost.localdomain ([118.242.3.34]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fed7ee1234sm34366855ad.165.2024.07.26.08.58.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Jul 2024 08:58:17 -0700 (PDT) From: "Wencheng Yang" To: devel@edk2.groups.io Cc: yuanhao.xie@intel.com, brijesh.singh@amd.com, min.m.xu@intel.com, michael.kubacki@microsoft.com, Wencheng Yang Subject: [edk2-devel] [PATCH v1] UefiCpuPkg/Library/MpInitLib: fix SEV-ES AP bootinng failure Date: Fri, 26 Jul 2024 23:58:04 +0800 Message-Id: <20240726155804.27989-1-east.moutain.yang@gmail.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Resent-Date: Tue, 30 Jul 2024 09:38:42 -0700 Resent-From: east.moutain.yang@gmail.com Reply-To: devel@edk2.groups.io,east.moutain.yang@gmail.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240206 header.b=tujAPO7B; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=gmail.com (policy=none) According to SEV-ES Guest-Hypervisor Communication Block Standardization section 4.3 SMP Booting, the subsequent reset requires the AP enters Reset Hold state either by AP Reset Hold NAE event or AP Reset Hold Request MSR Protocol. If the AP is not in AP Reset Hold state, it may miss subsequent INIT-SIPI, as the INIT-SIPI process depends on GHCB page in kernel, which is only mapped in VMGEXIT interception. To ensure all APs are in AP Reset Hold state, we add a var in AP specific data structure, set the var before the AP going to AP Reset Hold state. Subsequent INIT-SIPI should check the var of each AP before issuing INIT-SIPI signal. Cc: Yuanhao Xie Cc: Brijesh Singh Cc: Min Xu Cc: Michael Kubacki Signed-off-by: Wencheng Yang --- UefiCpuPkg/Library/MpInitLib/AmdSev.c | 11 ++++++- UefiCpuPkg/Library/MpInitLib/MpHandOff.h | 1 + UefiCpuPkg/Library/MpInitLib/MpLib.c | 39 ++++++++++++++++++++++-- UefiCpuPkg/Library/MpInitLib/MpLib.h | 4 ++- UefiCpuPkg/Library/MpInitLib/PeiMpLib.c | 2 ++ 5 files changed, 52 insertions(+), 5 deletions(-) diff --git a/UefiCpuPkg/Library/MpInitLib/AmdSev.c b/UefiCpuPkg/Library/MpI= nitLib/AmdSev.c index d34f9513e0..e5d5ecb181 100644 --- a/UefiCpuPkg/Library/MpInitLib/AmdSev.c +++ b/UefiCpuPkg/Library/MpInitLib/AmdSev.c @@ -194,16 +194,19 @@ SetSevEsJumpTable ( **/=0D VOID=0D SevEsPlaceApHlt (=0D - CPU_MP_DATA *CpuMpData=0D + CPU_MP_DATA *CpuMpData,=0D + UINT32 ProcessorNumber=0D )=0D {=0D MSR_SEV_ES_GHCB_REGISTER Msr;=0D GHCB *Ghcb;=0D UINT64 Status;=0D BOOLEAN DoDecrement;=0D + BOOLEAN EnterHltLoop;=0D BOOLEAN InterruptState;=0D =0D DoDecrement =3D (BOOLEAN)(CpuMpData->InitFlag =3D=3D ApInitConfig);=0D + EnterHltLoop =3D FALSE;=0D =0D while (TRUE) {=0D Msr.GhcbPhysicalAddress =3D AsmReadMsr64 (MSR_SEV_ES_GHCB);=0D @@ -221,7 +224,13 @@ SevEsPlaceApHlt ( InterlockedDecrement ((UINT32 *)&CpuMpData->MpCpuExchangeInfo->NumAp= sExecuting);=0D }=0D =0D + if (!EnterHltLoop) {=0D + EnterHltLoop =3D TRUE;=0D + CpuMpData->CpuData[ProcessorNumber].SevEsApEnterHltLoopAfterWakeup = =3D 1;=0D + }=0D +=0D Status =3D CcExitVmgExit (Ghcb, SVM_EXIT_AP_RESET_HOLD, 0, 0);=0D +=0D if ((Status =3D=3D 0) && (Ghcb->SaveArea.SwExitInfo2 !=3D 0)) {=0D CcExitVmgDone (Ghcb, InterruptState);=0D break;=0D diff --git a/UefiCpuPkg/Library/MpInitLib/MpHandOff.h b/UefiCpuPkg/Library/= MpInitLib/MpHandOff.h index ae93b7e3d7..50e290f5b3 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpHandOff.h +++ b/UefiCpuPkg/Library/MpInitLib/MpHandOff.h @@ -37,6 +37,7 @@ extern EFI_GUID mMpHandOffConfigGuid; typedef struct {=0D UINT32 ApicId;=0D UINT32 Health;=0D + UINT32 SevEsApEnterHltLoopAfterWakeup;=0D UINT64 StartupSignalAddress;=0D UINT64 StartupProcedureAddress;=0D } PROCESSOR_HAND_OFF;=0D diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.c b/UefiCpuPkg/Library/MpIn= itLib/MpLib.c index 1951922912..59f0d87f9d 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpLib.c +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.c @@ -660,13 +660,14 @@ InitializeApData ( **/=0D VOID=0D PlaceAPInHltLoop (=0D - IN CPU_MP_DATA *CpuMpData=0D + IN CPU_MP_DATA *CpuMpData,=0D + IN UINTN ProcessorNumber=0D )=0D {=0D while (TRUE) {=0D DisableInterrupts ();=0D if (CpuMpData->UseSevEsAPMethod) {=0D - SevEsPlaceApHlt (CpuMpData);=0D + SevEsPlaceApHlt (CpuMpData, ProcessorNumber);=0D } else {=0D CpuSleep ();=0D }=0D @@ -762,6 +763,9 @@ ApWakeupFunction ( while (TRUE) {=0D if (CpuMpData->InitFlag =3D=3D ApInitConfig) {=0D ProcessorNumber =3D ApIndex;=0D + if (CpuMpData->UseSevEsAPMethod) {=0D + CpuMpData->CpuData[ProcessorNumber].SevEsApEnterHltLoopAfterWakeup= =3D 0;=0D + }=0D //=0D // This is first time AP wakeup, get BIST information from AP stack= =0D //=0D @@ -782,6 +786,9 @@ ApWakeupFunction ( // Execute AP function if AP is ready=0D //=0D GetProcessorNumber (CpuMpData, &ProcessorNumber);=0D + if (CpuMpData->UseSevEsAPMethod) {=0D + CpuMpData->CpuData[ProcessorNumber].SevEsApEnterHltLoopAfterWakeup= =3D 0;=0D + }=0D //=0D // Clear AP start-up signal when AP waken up=0D //=0D @@ -903,7 +910,7 @@ ApWakeupFunction ( // Place AP is specified loop mode=0D //=0D if (CpuMpData->ApLoopMode =3D=3D ApInHltLoop) {=0D - PlaceAPInHltLoop (CpuMpData);=0D + PlaceAPInHltLoop (CpuMpData, ProcessorNumber);=0D //=0D // Never run here=0D //=0D @@ -993,6 +1000,20 @@ GetApResetVectorSize ( *SizeAbove1Mb =3D AddressMap->RendezvousFunnelSize - AddressMap->ModeT= ransitionOffset;=0D }=0D }=0D +/**=0D + Wait for SEV-ES AP enter in HLT-LOOP.^M=0D +=0D + @param[in] SevEsApInHltLoop Pointer to SevEsApInHltLoop^M=0D +**/=0D +VOID=0D +WaitSevEsApEnterHltLoopAfterWakeup (=0D + IN volatile UINT32 *SevEsApEnterHltLoop=0D + )=0D +{=0D + while (*(UINT32*)SevEsApEnterHltLoop =3D=3D 0) {=0D + CpuPause ();=0D + }=0D +}=0D =0D /**=0D This function will fill the exchange info structure.=0D @@ -1324,6 +1345,17 @@ WakeUpAP ( //=0D SendStartupIpiAllExcludingSelf ((UINT32)ExchangeInfo->BufferStar= t);=0D } else {=0D + //=0D + // Subsequent INIT-SIPI-SIPI=0D + //=0D + if (CpuMpData->SevEsIsEnabled && (CpuMpData->InitFlag !=3D ApIni= tConfig)) {=0D + for (Index =3D 0; Index < CpuMpData->CpuCount; Index++) {=0D + CpuData =3D &CpuMpData->CpuData[Index];=0D + if (Index !=3D CpuMpData->BspNumber) {=0D + WaitSevEsApEnterHltLoopAfterWakeup(&CpuData->SevEsApEnterH= ltLoopAfterWakeup);=0D + }=0D + }=0D + }=0D SendInitSipiSipiAllExcludingSelf ((UINT32)ExchangeInfo->BufferSt= art);=0D }=0D }=0D @@ -2270,6 +2302,7 @@ MpInitLibInitialize ( InitializeSpinLock (&CpuMpData->CpuData[Index].ApLock);=0D CpuMpData->CpuData[Index].CpuHealthy =3D (MpHandOff->Info[HobIndex= ].Health =3D=3D 0) ? TRUE : FALSE;=0D CpuMpData->CpuData[Index].ApFunction =3D 0;=0D + CpuMpData->CpuData[Index].SevEsApEnterHltLoopAfterWakeup =3D MpHan= dOff->Info[HobIndex].SevEsApEnterHltLoopAfterWakeup;=0D CpuInfoInHob[Index].InitialApicId =3D MpHandOff->Info[HobIndex]= .ApicId;=0D CpuInfoInHob[Index].ApTopOfStack =3D CpuMpData->Buffer + (Inde= x + 1) * CpuMpData->CpuApStackSize;=0D CpuInfoInHob[Index].ApicId =3D MpHandOff->Info[HobIndex]= .ApicId;=0D diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.h b/UefiCpuPkg/Library/MpIn= itLib/MpLib.h index 88b31fecca..c33a7bf658 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpLib.h +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.h @@ -146,6 +146,7 @@ typedef enum { typedef struct {=0D SPIN_LOCK ApLock;=0D volatile UINT32 *StartupApSignal;=0D + volatile UINT32 SevEsApEnterHltLoopAfterWakeup;=0D volatile UINTN ApFunction;=0D volatile UINTN ApFunctionArgument;=0D BOOLEAN CpuHealthy;=0D @@ -862,7 +863,8 @@ SetSevEsJumpTable ( **/=0D VOID=0D SevEsPlaceApHlt (=0D - CPU_MP_DATA *CpuMpData=0D + CPU_MP_DATA *CpuMpData,=0D + UINT32 ProcessorNumber=0D );=0D =0D /**=0D diff --git a/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c b/UefiCpuPkg/Library/M= pInitLib/PeiMpLib.c index 16a858d542..b393ce00be 100644 --- a/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c +++ b/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c @@ -164,6 +164,8 @@ SaveCpuMpData ( if (CpuMpData->ApLoopMode !=3D ApInHltLoop) {=0D MpHandOff->Info[Index-HobBase].StartupSignalAddress =3D (UINT64)(= UINTN)CpuMpData->CpuData[Index].StartupApSignal;=0D MpHandOff->Info[Index-HobBase].StartupProcedureAddress =3D (UINT64)(= UINTN)&CpuMpData->CpuData[Index].ApFunction;=0D + } else {=0D + MpHandOff->Info[Index-HobBase].SevEsApEnterHltLoopAfterWakeup =3D Cp= uMpData->CpuData[Index].SevEsApEnterHltLoopAfterWakeup;=0D }=0D }=0D =0D --=20 2.39.0 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#120127): https://edk2.groups.io/g/devel/message/120127 Mute This Topic: https://groups.io/mt/107630732/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-