From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mx.groups.io with SMTP id smtpd.web09.3894.1576139048664619078 for ; Thu, 12 Dec 2019 00:24:08 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.31, mailfrom: ray.ni@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Dec 2019 00:24:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,305,1571727600"; d="scan'208";a="265141982" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by FMSMGA003.fm.intel.com with ESMTP; 12 Dec 2019 00:24:07 -0800 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 12 Dec 2019 00:24:07 -0800 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 12 Dec 2019 00:24:06 -0800 Received: from shsmsx153.ccr.corp.intel.com (10.239.6.53) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 12 Dec 2019 00:24:05 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.90]) by SHSMSX153.ccr.corp.intel.com ([169.254.12.195]) with mapi id 14.03.0439.000; Thu, 12 Dec 2019 16:24:03 +0800 From: "Ni, Ray" To: "Lendacky, Thomas" , "devel@edk2.groups.io" CC: "Justen, Jordan L" , Laszlo Ersek , Ard Biesheuvel , "Kinney, Michael D" , "Gao, Liming" , "Dong, Eric" , "Singh, Brijesh" Subject: Re: [RFC PATCH v2 42/44] UefiCpuPkg/MpInitLib: Prepare SEV-ES guest APs for OS use Thread-Topic: [RFC PATCH v2 42/44] UefiCpuPkg/MpInitLib: Prepare SEV-ES guest APs for OS use Thread-Index: AQHVbyP4YQ9v4CWpzkSEePbu6C8gOqe2qzQw Date: Thu, 12 Dec 2019 08:24:03 +0000 Message-ID: <734D49CCEBEEF84792F5B80ED585239D5C399C13@SHSMSX104.ccr.corp.intel.com> References: <00047308110ff3380000f6eb140e815c01499e3a.1568922729.git.thomas.lendacky@amd.com> In-Reply-To: <00047308110ff3380000f6eb140e815c01499e3a.1568922729.git.thomas.lendacky@amd.com> Accept-Language: en-US, zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZDEyYTRjYzEtZDY2NC00NzczLWJlZDEtNTU3Yzk2YzVlMDE0IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoieFhvZjZSQnBvV3R4QWQ5dEdmXC9FS0w2WlkrTGZubnpWcDBZQUJBMXpyVnJRVTVWUmV4STZaVGJCUkhkUHpvVkMifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: ray.ni@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Tom, why this cannot be done as today's code in ExitBS callback? Because it should be done after interrupt is disabled? > -----Original Message----- > From: Lendacky, Thomas > Sent: Friday, September 20, 2019 3:53 AM > To: devel@edk2.groups.io > Cc: Justen, Jordan L ; Laszlo Ersek ; Ard Biesheuvel > ; Kinney, Michael D ; Gao, Liming ; Dong, > Eric ; Ni, Ray ; Singh, Brijesh > Subject: [RFC PATCH v2 42/44] UefiCpuPkg/MpInitLib: Prepare SEV-ES guest = APs for OS use >=20 > From: Tom Lendacky >=20 > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2198 >=20 > Before UEFI transfers control to the OS, it must park the AP. This is > done using the AsmRelocateApLoop function to transition into 32-bit > non-paging mode. For an SEV-ES guest, a few additional things must be > done: > - AsmRelocateApLoop must be updated to support SEV-ES. This means > performing a VMGEXIT AP Reset Hold instead of an MWAIT or HLT loop. > - Since the AP must transition to real mode, a small routine is copied > to the WakeupBuffer area. Since the WakeupBuffer will be used by > the AP during OS booting, it must be placed in reserved memory. > Additionally, the AP stack must be located where it can be accessed > in real mode. > - Once the AP is in real mode it will transfer control to the > destination specified by the OS in the SEV-ES AP Jump Table. The > SEV-ES AP Jump Table address is communicated to the OS using the BSP > GHCB MSR. >=20 > Cc: Eric Dong > Cc: Ray Ni > Cc: Laszlo Ersek > Signed-off-by: Tom Lendacky > --- > UefiCpuPkg/Library/MpInitLib/MpLib.h | 8 +- > UefiCpuPkg/Library/MpInitLib/DxeMpLib.c | 54 ++++++- > UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm | 134 ++++++++++++++++-- > 3 files changed, 176 insertions(+), 20 deletions(-) >=20 > diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.h b/UefiCpuPkg/Library/Mp= InitLib/MpLib.h > index 5966510d4a1b..2d38a0e85a40 100644 > --- a/UefiCpuPkg/Library/MpInitLib/MpLib.h > +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.h > @@ -277,7 +277,8 @@ struct _CPU_MP_DATA { > UINT64 GhcbBase; > }; >=20 > -#define AP_RESET_STACK_SIZE 64 > +#define AP_SAFE_STACK_SIZE 128 > +#define AP_RESET_STACK_SIZE AP_SAFE_STACK_SIZE >=20 > typedef union { > struct { > @@ -331,8 +332,11 @@ VOID > IN BOOLEAN MwaitSupport, > IN UINTN ApTargetCState, > IN UINTN PmCodeSegment, > + IN UINTN Pm16CodeSegment, > IN UINTN TopOfApStack, > - IN UINTN NumberToFinish > + IN UINTN NumberToFinish, > + IN UINTN SevEsAPJumpTable, > + IN UINTN WakeupBuffer > ); >=20 > /** > diff --git a/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c b/UefiCpuPkg/Library= /MpInitLib/DxeMpLib.c > index 16603ef3f20e..cf53b5026aa4 100644 > --- a/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c > +++ b/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c > @@ -18,7 +18,6 @@ > #include >=20 > #define AP_CHECK_INTERVAL (EFI_TIMER_PERIOD_MILLISECONDS (100)) > -#define AP_SAFE_STACK_SIZE 128 >=20 > CPU_MP_DATA *mCpuMpData =3D NULL; > EFI_EVENT mCheckAllApsEvent =3D NULL; > @@ -86,6 +85,13 @@ GetWakeupBuffer ( > { > EFI_STATUS Status; > EFI_PHYSICAL_ADDRESS StartAddress; > + EFI_MEMORY_TYPE MemoryType; > + > + if (PcdGetBool (PcdSevEsActive)) { > + MemoryType =3D EfiReservedMemoryType; > + } else { > + MemoryType =3D EfiBootServicesData; > + } >=20 > // > // Try to allocate buffer below 1M for waking vector. > @@ -98,7 +104,7 @@ GetWakeupBuffer ( > StartAddress =3D 0x88000; > Status =3D gBS->AllocatePages ( > AllocateMaxAddress, > - EfiBootServicesData, > + MemoryType, > EFI_SIZE_TO_PAGES (WakeupBufferSize), > &StartAddress > ); > @@ -331,17 +337,26 @@ RelocateApLoop ( > BOOLEAN MwaitSupport; > ASM_RELOCATE_AP_LOOP AsmRelocateApLoopFunc; > UINTN ProcessorNumber; > + UINTN StackStart; >=20 > MpInitLibWhoAmI (&ProcessorNumber); > CpuMpData =3D GetCpuMpData (); > MwaitSupport =3D IsMwaitSupport (); > + if (CpuMpData->SevEsActive) { > + StackStart =3D CpuMpData->SevEsAPResetStackStart; > + } else { > + StackStart =3D mReservedTopOfApStack; > + } > AsmRelocateApLoopFunc =3D (ASM_RELOCATE_AP_LOOP) (UINTN) mReservedApLo= opFunc; > AsmRelocateApLoopFunc ( > MwaitSupport, > CpuMpData->ApTargetCState, > CpuMpData->PmCodeSegment, > - mReservedTopOfApStack - ProcessorNumber * AP_SAFE_STACK_SIZE, > - (UINTN) &mNumberToFinish > + CpuMpData->Pm16CodeSegment, > + StackStart - ProcessorNumber * AP_SAFE_STACK_SIZE, > + (UINTN) &mNumberToFinish, > + CpuMpData->SevEsAPBuffer, > + CpuMpData->WakeupBuffer > ); > // > // It should never reach here > @@ -895,9 +910,34 @@ MpFinalize ( > IN CPU_MP_DATA *CpuMpData > ) > { > - // > - // DXE phase will do this transition, but just return EFI_SUCCESS for = now. > - // > + if (CpuMpData->SevEsActive) { > + // > + // Perform SEV-ES specific finalization > + // > + if (CpuMpData->WakeupBuffer =3D=3D (UINTN) -1) { > + // > + // No APs parked in UEFI, clear the GHCB > + // > + AsmWriteMsr64 (MSR_SEV_ES_GHCB, 0); > + } else { > + // > + // Re-use reserved memory area below 1MB from WakeupBuffer > + // > + CopyMem ( > + (VOID *) CpuMpData->WakeupBuffer, > + (VOID *) CpuMpData->AddressMap.RendezvousFunnelAddress + > + CpuMpData->AddressMap.SwitchToRealPM16ModeOffset, > + CpuMpData->AddressMap.SwitchToRealPM16ModeSize > + ); > + > + // > + // Point the GHCB at the AP jump table to communicate the address = to > + // the booting system. > + // > + AsmWriteMsr64 (MSR_SEV_ES_GHCB, (CpuMpData->SevEsAPBuffer) | 0x03)= ; > + } > + } > + > return EFI_SUCCESS; > } >=20 > diff --git a/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm b/UefiCpuPkg/L= ibrary/MpInitLib/X64/MpFuncs.nasm > index bbc7432740ff..3cb0cd5bb306 100644 > --- a/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm > +++ b/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm > @@ -465,6 +465,10 @@ BITS 16 > ; - IP for Real Mode (two bytes) > ; - CS for Real Mode (two bytes) > ; > + ; This label is also used with AsmRelocateApLoop. During MP finaliza= tion, > + ; the code from PM16Mode to SwitchToRealProcEnd is copied to the sta= rt of > + ; the WakeupBuffer, allowing a parked AP to be booted by an OS. > + ; > PM16Mode: > mov eax, cr0 ; Read CR0 > btr eax, 0 ; Set PE=3D0 > @@ -487,32 +491,95 @@ PM16Mode: > SwitchToRealProcEnd: >=20 > ;-----------------------------------------------------------------------= -------------- > -; AsmRelocateApLoop (MwaitSupport, ApTargetCState, PmCodeSegment, TopOf= ApStack, CountTofinish); > +; AsmRelocateApLoop (MwaitSupport, ApTargetCState, PmCodeSegment, Pm16C= odeSegment, TopOfApStack, > CountTofinish, SevEsAPJumpTable, WakeupBuffer); > ;-----------------------------------------------------------------------= -------------- > global ASM_PFX(AsmRelocateApLoop) > ASM_PFX(AsmRelocateApLoop): > AsmRelocateApLoopStart: > BITS 64 > + cmp qword [rsp + 56], 0 > + je NoSevEs > + > + ; > + ; Perform some SEV-ES related setup before leaving 64-bit mode > + ; > + push rcx > + push rdx > + > + ; > + ; Get the RDX reset value using CPUID > + ; > + mov rax, 1 > + cpuid > + mov rsi, rax ; Save off the reset value for RDX > + > + ; > + ; Prepare the GHCB for the AP_HLT_LOOP VMGEXIT call > + ; - Must be done while in 64-bit long mode so that writes to > + ; the GHCB memory will be unencrypted. > + ; - No NAE events can be generated once this is set otherwise > + ; the AP_RESET_HOLD SW_EXITCODE will be overwritten. > + ; > + mov rcx, 0xc0010130 > + rdmsr ; Retrieve current GHCB address > + shl rdx, 32 > + or rdx, rax > + > + mov rdi, rdx > + xor rax, rax > + mov rcx, 0x800 > + shr rcx, 3 > + rep stosq ; Clear the GHCB > + > + mov rax, 0x80000004 ; VMGEXIT AP_RESET_HOLD > + mov [rdx + 0x390], rax > + > + pop rdx > + pop rcx > + > +NoSevEs: > cli ; Disable interrupt before switching to= 32-bit mode > - mov rax, [rsp + 40] ; CountTofinish > + mov rax, [rsp + 48] ; CountTofinish > lock dec dword [rax] ; (*CountTofinish)-- > - mov rsp, r9 > - push rcx > - push rdx >=20 > - lea rsi, [PmEntry] ; rsi <- The start address of transitio= n code > + mov rax, [rsp + 56] ; SevEsAPJumpTable > + mov rbx, [rsp + 64] ; WakeupBuffer > + mov rsp, [rsp + 40] ; TopOfApStack > + > + push rax ; Save SevEsAPJumpTable > + push rbx ; Save WakeupBuffer > + push r9 ; Save Pm16CodeSegment > + push rcx ; Save MwaitSupport > + push rdx ; Save ApTargetCState > + > + lea rax, [PmEntry] ; rax <- The start address of transitio= n code >=20 > push r8 > - push rsi > - DB 0x48 > - retf > + push rax > + > + ; > + ; Clear R8 - R15, for reset, before going into 32-bit mode > + ; > + xor r8, r8 > + xor r9, r9 > + xor r10, r10 > + xor r11, r11 > + xor r12, r12 > + xor r13, r13 > + xor r14, r14 > + xor r15, r15 > + > + ; > + ; Far return into 32-bit mode > + ; > +o64 retf > + > BITS 32 > PmEntry: > mov eax, cr0 > btr eax, 31 ; Clear CR0.PG > mov cr0, eax ; Disable paging and caches >=20 > - mov ebx, edx ; Save EntryPoint to rbx, for rdmsr wil= l overwrite rdx > mov ecx, 0xc0000080 > rdmsr > and ah, ~ 1 ; Clear LME > @@ -525,6 +592,8 @@ PmEntry: > add esp, 4 > pop ecx, > add esp, 4 > + > +MwaitCheck: > cmp cl, 1 ; Check mwait-monitor support > jnz HltLoop > mov ebx, edx ; Save C-State to ebx > @@ -538,10 +607,53 @@ MwaitLoop: > shl eax, 4 > mwait > jmp MwaitLoop > + > HltLoop: > + pop edx ; PM16CodeSegment > + add esp, 4 > + pop ebx ; WakeupBuffer > + add esp, 4 > + pop eax ; SevEsAPJumpTable > + add esp, 4 > + cmp eax, 0 ; Check for SEV-ES > + je DoHlt > + > + cli > + ; > + ; SEV-ES is active, use VMGEXIT (GHCB information already > + ; set by caller) > + ; > + ; VMGEXIT is rep vmmcall > + ; > + db 0xf3 > + db 0x0f > + db 0x01 > + db 0xd9 > + > + ; > + ; Back from VMGEXIT AP_HLT_LOOP > + ; Push the FLAGS/CS/IP values to use > + ; > + push word 0x0002 ; EFLAGS > + xor ecx, ecx > + mov cx, [eax + 2] ; CS > + push cx > + mov cx, [eax] ; IP > + push cx > + push word 0x0000 ; For alignment, will be discarded > + > + push edx > + push ebx > + > + mov edx, esi ; Restore RDX reset value > + > + retf > + > +DoHlt: > cli > hlt > - jmp HltLoop > + jmp DoHlt > + > BITS 64 > AsmRelocateApLoopEnd: >=20 > -- > 2.17.1