From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.65; helo=mga03.intel.com; envelope-from=eric.dong@intel.com; receiver=edk2-devel@lists.01.org Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 0730F2034A8A6 for ; Wed, 17 Jan 2018 22:48:35 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jan 2018 22:53:56 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,376,1511856000"; d="scan'208";a="24302374" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga001.jf.intel.com with ESMTP; 17 Jan 2018 22:53:56 -0800 Received: from fmsmsx152.amr.corp.intel.com (10.18.125.5) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 17 Jan 2018 22:53:56 -0800 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by FMSMSX152.amr.corp.intel.com (10.18.125.5) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 17 Jan 2018 22:53:55 -0800 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.189]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.93]) with mapi id 14.03.0319.002; Thu, 18 Jan 2018 14:53:49 +0800 From: "Dong, Eric" To: "Wang, Jian J" , "edk2-devel@lists.01.org" CC: "Yao, Jiewen" , "Ni, Ruiyu" , Laszlo Ersek Thread-Topic: [PATCH 1/6] UefiCpuPkg/MpInitLib: split wake up buffer into two parts Thread-Index: AQHTjd6IYnLzW0bOGU+WtiOV7XfeYaN5NleQ Date: Thu, 18 Jan 2018 06:53:49 +0000 Message-ID: References: <20180115085433.25008-1-jian.j.wang@intel.com> <20180115085433.25008-2-jian.j.wang@intel.com> In-Reply-To: <20180115085433.25008-2-jian.j.wang@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [PATCH 1/6] UefiCpuPkg/MpInitLib: split wake up buffer into two parts X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jan 2018 06:48:36 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Eric Dong -----Original Message----- From: Wang, Jian J=20 Sent: Monday, January 15, 2018 4:54 PM To: edk2-devel@lists.01.org Cc: Yao, Jiewen ; Ni, Ruiyu ; Don= g, Eric ; Laszlo Ersek Subject: [PATCH 1/6] UefiCpuPkg/MpInitLib: split wake up buffer into two pa= rts If PcdDxeNxMemoryProtectionPolicy is set to enable protection for memory of= EfiBootServicesCode, EfiConventionalMemory, the BIOS will hang at a page f= ault exception during MP initialization. The root cause is that the AP wake up buffer, which is below 1MB and used t= o hold both AP init code and data, is type of EfiConventionalMemory (not re= ally allocated because of potential conflict with legacy code), and is mark= ed as non-executable. During the transition from real address mode to long = mode, the AP init code has to enable paging which will then cause itself a = page fault exception because it's just running in non-executable memory. The solution is splitting AP wake up buffer into two part: lower part is st= ill below 1MB and shared with legacy system, higher part is really allocate= d memory of BootServicesCode type. The init code in the memory below 1MB wi= ll not enable paging but just switch to protected mode and jump to higher m= emory, in which the init code will enable paging and switch to long mode. Cc: Jiewen Yao Cc: Ruiyu Ni Cc: Eric Dong Cc: Laszlo Ersek Contributed-under: TianoCore Contribution Agreement 1.1 Signed-off-by: Jian J Wang --- UefiCpuPkg/Library/MpInitLib/DxeMpLib.c | 34 ++++++++++ UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc | 5 ++ UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm | 32 +++++----- UefiCpuPkg/Library/MpInitLib/MpLib.c | 45 +++++++++++++ UefiCpuPkg/Library/MpInitLib/MpLib.h | 22 +++++++ UefiCpuPkg/Library/MpInitLib/PeiMpLib.c | 23 +++++++ UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc | 5 +- UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm | 87 ++++++++++++++++------= ---- 8 files changed, 204 insertions(+), 49 deletions(-) diff --git a/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c b/UefiCpuPkg/Library/M= pInitLib/DxeMpLib.c index d2bcef53d6..fd2317924f 100644 --- a/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c +++ b/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c @@ -113,6 +113,40 @@ GetWakeupBuffer ( return (UINTN) StartAddress; } =20 +/** + Get available EfiBootServicesCode memory below 4GB by specified size. + + This buffer is required to safely transfer AP from real address mode=20 + to protected mode or long mode, due to the fact that the buffer=20 + returned by + GetWakeupBuffer() may be marked as non-executable. + + @param[in] BufferSize Wakeup transition buffer size. + + @retval other Return wakeup transition buffer address below 4GB. + @retval 0 Cannot find free memory below 4GB. +**/ +UINTN +GetModeTransitionBuffer ( + IN UINTN BufferSize + ) +{ + EFI_STATUS Status; + EFI_PHYSICAL_ADDRESS StartAddress; + + StartAddress =3D BASE_4GB - 1; + Status =3D gBS->AllocatePages ( + AllocateMaxAddress, + EfiBootServicesCode, + EFI_SIZE_TO_PAGES (BufferSize), + &StartAddress + ); + if (EFI_ERROR (Status)) { + StartAddress =3D 0; + } + + return (UINTN)StartAddress; +} + /** Checks APs status and updates APs status if needed. =20 diff --git a/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc b/UefiCpuPkg/Libra= ry/MpInitLib/Ia32/MpEqu.inc index bdfe0d33cc..1648f2c4b0 100644 --- a/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc +++ b/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc @@ -41,4 +41,9 @@ Cr3Location equ LockLocation + 3= 4h InitFlagLocation equ LockLocation + 38h CpuInfoLocation equ LockLocation + 3Ch NumApsExecutingLocation equ LockLocation + 40h +InitializeFloatingPointUnitsAddress equ LockLocation + 48h +ModeTransitionMemoryLocation equ LockLocation + 4Ch +ModeTransitionSegmentLocation equ LockLocation + 50h +ModeHighMemoryLocation equ LockLocation + 52h +ModeHighSegmentLocation equ LockLocation + 56h =20 diff --git a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm b/UefiCpuPkg/Li= brary/MpInitLib/Ia32/MpFuncs.nasm index 2b6c27d4ec..bd79be0f5e 100644 --- a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm +++ b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm @@ -48,34 +48,35 @@ BITS 16 mov si, BufferStartLocation mov ebx, [si] =20 - mov si, ModeOffsetLocation - mov eax, [si] - mov si, CodeSegmentLocation - mov edx, [si] - mov di, ax - sub di, 02h - mov [di], dx - sub di, 04h - add eax, ebx - mov [di],eax - mov si, DataSegmentLocation mov edx, [si] =20 + ; + ; Get start address of 32-bit code in low memory (<1MB) + ; + mov edi, ModeTransitionMemoryLocation + mov si, GdtrLocation o32 lgdt [cs:si] =20 mov si, IdtrLocation o32 lidt [cs:si] =20 - xor ax, ax - mov ds, ax - + ; + ; Switch to protected mode + ; mov eax, cr0 ; Get control register 0 or eax, 000000003h ; Set PE bit (bit #0) & MP mov cr0, eax =20 - jmp 0:strict dword 0 ; far jump to protected mod= e + ; Switch to 32-bit code in executable memory (>1MB) +o32 jmp far [cs:di] + +; +; Following code may be copied to memory with type of EfiBootServicesCode. +; This is required at DXE phase if NX is enabled for=20 +EfiBootServicesCode of ; memory. +; BITS 32 Flat32Start: ; protected mode entry poin= t mov ds, dx @@ -266,6 +267,7 @@ ASM_PFX(AsmGetAddressMap): mov dword [ebx + 8h], RendezvousFunnelProcEnd - RendezvousFunn= elProcStart mov dword [ebx + 0Ch], AsmRelocateApLoopStart mov dword [ebx + 10h], AsmRelocateApLoopEnd - AsmRelocateApLoop= Start + mov dword [ebx + 14h], Flat32Start - RendezvousFunnelProcStart =20 popad ret diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.c b/UefiCpuPkg/Library/MpIn= itLib/MpLib.c index cdc03113e5..fbcbcc6cc9 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpLib.c +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.c @@ -772,6 +772,8 @@ FillExchangeInfoData ( ) { volatile MP_CPU_EXCHANGE_INFO *ExchangeInfo; + UINTN Size; + IA32_SEGMENT_DESCRIPTOR *Selector; =20 ExchangeInfo =3D CpuMpData->MpCpuExchangeInfo; ExchangeInfo->Lock =3D 0; @@ -801,6 +803,44 @@ FillExchangeInfoData ( // AsmReadGdtr ((IA32_DESCRIPTOR *) &ExchangeInfo->GdtrProfile); AsmReadIdtr ((IA32_DESCRIPTOR *) &ExchangeInfo->IdtrProfile); + + // + // Find a 32-bit code segment + // + Selector =3D (IA32_SEGMENT_DESCRIPTOR *)ExchangeInfo->GdtrProfile.Base; = =20 + Size =3D ExchangeInfo->GdtrProfile.Limit + 1; while (Size > 0) { + if (Selector->Bits.L =3D=3D 0 && Selector->Bits.Type >=3D 8) { + ExchangeInfo->ModeTransitionSegment =3D + (UINT16)((UINTN)Selector - ExchangeInfo->GdtrProfile.Base); + break; + } + Selector +=3D 1; + Size -=3D sizeof (IA32_SEGMENT_DESCRIPTOR); } + + // + // Copy all 32-bit code and 64-bit code into memory with type of //=20 + EfiBootServicesCode to avoid page fault if NX memory protection is enable= d. + // + if (ExchangeInfo->ModeTransitionMemory !=3D 0) { + Size =3D CpuMpData->AddressMap.RendezvousFunnelSize - + CpuMpData->AddressMap.ModeTransitionOffset; + CopyMem ( + (VOID *)(UINTN)ExchangeInfo->ModeTransitionMemory, + CpuMpData->AddressMap.RendezvousFunnelAddress + + CpuMpData->AddressMap.ModeTransitionOffset, + Size + ); + + ExchangeInfo->ModeHighMemory =3D ExchangeInfo->ModeTransitionMemory; + ExchangeInfo->ModeHighMemory +=3D (UINT32)ExchangeInfo->ModeOffset - + (UINT32)CpuMpData->AddressMap.ModeTransitionOffset; + ExchangeInfo->ModeHighSegment =3D (UINT16)ExchangeInfo->CodeSegment; + } else { + ExchangeInfo->ModeTransitionMemory =3D (UINT32) + (ExchangeInfo->BufferStart +=20 + CpuMpData->AddressMap.ModeTransitionOffset); + } } =20 /** @@ -876,6 +916,11 @@ AllocateResetVector ( CpuMpData->WakeupBuffer =3D GetWakeupBuffer (ApResetVectorSize); CpuMpData->MpCpuExchangeInfo =3D (MP_CPU_EXCHANGE_INFO *) (UINTN) (CpuMpData->WakeupBuffer + CpuMpData->AddressMap.Rende= zvousFunnelSize); + CpuMpData->MpCpuExchangeInfo->ModeTransitionMemory =3D (UINT32) + GetModeTransitionBuffer ( + CpuMpData->AddressMap.RendezvousFunnelSize - + CpuMpData->AddressMap.ModeTransitionOffset + ); } BackupAndPrepareWakeupBuffer (CpuMpData); } diff --git a/UefiCpuPkg/Lib= rary/MpInitLib/MpLib.h b/UefiCpuPkg/Library/MpInitLib/MpLib.h index 685e96cbac..0232fe896a 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpLib.h +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.h @@ -152,6 +152,7 @@ typedef struct { UINTN RendezvousFunnelSize; UINT8 *RelocateApLoopFuncAddress; UINTN RelocateApLoopFuncSize; + UINTN ModeTransitionOffset; } MP_ASSEMBLY_ADDRESS_MAP; =20 typedef struct _CPU_MP_DATA CPU_MP_DATA; @@ -182,6 +183,10 @@ typedef str= uct { UINTN NumApsExecuting; CPU_MP_DATA *CpuMpData; UINTN InitializeFloatingPointUnitsAddress; + UINT32 ModeTransitionMemory; + UINT16 ModeTransitionSegment; + UINT32 ModeHighMemory; + UINT16 ModeHighSegment; } MP_CPU_EXCHANGE_INFO; =20 #pragma pack() @@ -329,6 +334,23 @@ GetWakeupBuffer ( IN UINTN WakeupBufferSize ); =20 +/** + Get available EfiBootServicesCode memory below 4GB by specified size. + + This buffer is required to safely transfer AP from real address mode=20 + to protected mode or long mode, due to the fact that the buffer=20 + returned by + GetWakeupBuffer() may be marked as non-executable. + + @param[in] BufferSize Wakeup transition buffer size. + + @retval other Return wakeup transition buffer address below 4GB. + @retval 0 Cannot find free memory below 4GB. +**/ +UINTN +GetModeTransitionBuffer ( + IN UINTN BufferSize + ); + /** This function will be called by BSP to wakeup AP. =20 diff --git a/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c b/UefiCpuPkg/Library/M= pInitLib/PeiMpLib.c index 70c2bc7323..ad43bd33f5 100644 --- a/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c +++ b/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c @@ -187,6 +187,29 @@ GetWakeupBuffer ( return (UINTN) -1; } =20 +/** + Get available EfiBootServicesCode memory below 4GB by specified size. + + This buffer is required to safely transfer AP from real address mode=20 + to protected mode or long mode, due to the fact that the buffer=20 + returned by + GetWakeupBuffer() may be marked as non-executable. + + @param[in] BufferSize Wakeup transition buffer size. + + @retval other Return wakeup transition buffer address below 4GB. + @retval 0 Cannot find free memory below 4GB. +**/ +UINTN +GetModeTransitionBuffer ( + IN UINTN BufferSize + ) +{ + // + // PEI phase doesn't need to do such transition. So simply return 0. + // + return 0; +} + /** Checks APs status and updates APs status if needed. =20 diff --git a/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc b/UefiCpuPkg/Librar= y/MpInitLib/X64/MpEqu.inc index d255ca5e1b..b5e09c6bc1 100644 --- a/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc +++ b/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc @@ -42,4 +42,7 @@ InitFlagLocation equ LockLocation + 6= Ch CpuInfoLocation equ LockLocation + 74h NumApsExecutingLocation equ LockLocation + 7Ch InitializeFloatingPointUnitsAddress equ LockLocation + 8Ch - +ModeTransitionMemoryLocation equ LockLocation + 94h +ModeTransitionSegmentLocation equ LockLocation + 98h +ModeHighMemoryLocation equ LockLocation + 9Ah +ModeHighSegmentLocation equ LockLocation + 9Eh diff --git a/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm b/UefiCpuPkg/Lib= rary/MpInitLib/X64/MpFuncs.nasm index 21d278600d..7595988884 100644 --- a/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm +++ b/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm @@ -52,16 +52,13 @@ BITS 16 mov si, BufferStartLocation mov ebx, [si] =20 - mov di, ModeOffsetLocation - mov eax, [di] - mov di, CodeSegmentLocation - mov edx, [di] - mov di, ax - sub di, 02h - mov [di],dx ; Patch long mode CS - sub di, 04h - add eax, ebx - mov [di],eax ; Patch address + mov si, DataSegmentLocation + mov edx, [si] + + ; + ; Get start address of 32-bit code in low memory (<1MB) + ; + mov edi, ModeTransitionMemoryLocation =20 mov si, GdtrLocation o32 lgdt [cs:si] @@ -69,56 +66,79 @@ o32 lgdt [cs:si] mov si, IdtrLocation o32 lidt [cs:si] =20 - mov si, EnableExecuteDisableLocation - cmp byte [si], 0 - jz SkipEnableExecuteDisableBit + ; + ; Switch to protected mode + ; + mov eax, cr0 ; Get control register 0 + or eax, 000000003h ; Set PE bit (bit #0) & MP + mov cr0, eax + + ; Switch to 32-bit code (>1MB) +o32 jmp far [cs:di] + +; +; Following code must be copied to memory with type of EfiBootServicesCode= . +; This is required if NX is enabled for EfiBootServicesCode of memory. +; +BITS 32 +Flat32Start: ; protected mode entry poin= t + mov ds, dx + mov es, dx + mov fs, dx + mov gs, dx + mov ss, dx =20 ; ; Enable execute disable bit ; + mov esi, EnableExecuteDisableLocation + cmp byte [ebx + esi], 0 + jz SkipEnableExecuteDisableBit + mov ecx, 0c0000080h ; EFER MSR number rdmsr ; Read EFER bts eax, 11 ; Enable Execute Disable Bit wrmsr ; Write EFER =20 SkipEnableExecuteDisableBit: - - mov di, DataSegmentLocation - mov edi, [di] ; Save long mode DS in edi - - mov si, Cr3Location ; Save CR3 in ecx - mov ecx, [si] - - xor ax, ax - mov ds, ax ; Clear data segment - - mov eax, cr0 ; Get control register 0 - or eax, 000000003h ; Set PE bit (bit #0) & MP - mov cr0, eax - + ; + ; Enable PAE + ; mov eax, cr4 bts eax, 5 mov cr4, eax =20 + ; + ; Load page table + ; + mov esi, Cr3Location ; Save CR3 in ecx + mov ecx, [ebx + esi] mov cr3, ecx ; Load CR3 =20 + ; + ; Enable long mode + ; mov ecx, 0c0000080h ; EFER MSR number rdmsr ; Read EFER bts eax, 8 ; Set LME=3D1 wrmsr ; Write EFER =20 + ; + ; Enable paging + ; mov eax, cr0 ; Read CR0 bts eax, 31 ; Set PG=3D1 mov cr0, eax ; Write CR0 =20 - jmp 0:strict dword 0 ; far jump to long mode + ; + ; Far jump to 64-bit code + ; + mov edi, ModeHighMemoryLocation + add edi, ebx + jmp far [edi] + BITS 64 LongModeStart: - mov eax, edi - mov ds, ax - mov es, ax - mov ss, ax - mov esi, ebx lea edi, [esi + InitFlagLocation] cmp qword [edi], 1 ; ApInitConfig @@ -295,6 +315,7 @@ ASM_PFX(AsmGetAddressMap): lea rax, [ASM_PFX(AsmRelocateApLoop)] mov qword [rcx + 18h], rax mov qword [rcx + 20h], AsmRelocateApLoopEnd - AsmRelocateApLoop= Start + mov qword [rcx + 28h], Flat32Start - RendezvousFunnelProcStart ret =20 ;-------------------------------------------------------------------------= ------------ -- 2.15.1.windows.2