* [PATCH 1/6] OvmfPkg: Add Tdx BFV/CFV PCDs and PcdOvmfImageSizeInKb
2021-07-12 1:19 [PATCH 0/6] Add Intel TDX support in OvmfPkg/ResetVector Min Xu
@ 2021-07-12 1:19 ` Min Xu
2021-07-12 1:19 ` [PATCH 2/6] OvmfPkg: Add Tdx metadata Min Xu
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Min Xu @ 2021-07-12 1:19 UTC (permalink / raw)
To: devel
Cc: Min Xu, Brijesh Singh, Erdem Aktas, James Bottomley, Jiewen Yao,
Laszlo Ersek, Tom Lendacky
RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429
Tdx Virtual Firmware (TDVF) includes one Firmware Volume (FV) known
as the Boot Firmware Volume (BFV). The FV format is defined in the
UEFI Platform Initialization (PI) spec. BFV includes all TDVF components
required during boot.
TDVF also include a configuration firmware volume (CFV) that is separated
from the BFV. The reason is because the CFV is measured in RTMR, while
the BFV is measured in MRTD.
In practice BFV is the code part of Ovmf image. CFV is the vars part of
Ovmf image (exclude the SPARE part).
PcdOvmfImageSizeInKb is added which is used to calculate the offset
of TdxMetadata in ResetVectorVtf0.asm.
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
OvmfPkg/OvmfPkg.dec | 13 +++++++++++++
OvmfPkg/OvmfPkgDefines.fdf.inc | 12 +++++++++++-
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 6ae733f6e39f..6d9bb91e9274 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -321,6 +321,19 @@
gUefiOvmfPkgTokenSpaceGuid.PcdSevLaunchSecretBase|0x0|UINT32|0x42
gUefiOvmfPkgTokenSpaceGuid.PcdSevLaunchSecretSize|0x0|UINT32|0x43
+ ## The base address and size of the TDX Cfv base and size.
+ gUefiOvmfPkgTokenSpaceGuid.PcdCfvBase|0|UINT32|0x47
+ gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataOffset|0|UINT32|0x48
+ gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataSize|0|UINT32|0x49
+
+ ## The base address and size of the TDX Bfv base and size.
+ gUefiOvmfPkgTokenSpaceGuid.PcdBfvBase|0|UINT32|0x4a
+ gUefiOvmfPkgTokenSpaceGuid.PcdBfvRawDataOffset|0|UINT32|0x4b
+ gUefiOvmfPkgTokenSpaceGuid.PcdBfvRawDataSize|0|UINT32|0x4c
+
+ ## Size of the Ovmf image in KB
+ gUefiOvmfPkgTokenSpaceGuid.PcdOvmfImageSizeInKb|0|UINT32|0x4d
+
[PcdsDynamic, PcdsDynamicEx]
gUefiOvmfPkgTokenSpaceGuid.PcdEmuVariableEvent|0|UINT64|2
gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashVariablesEnable|FALSE|BOOLEAN|0x10
diff --git a/OvmfPkg/OvmfPkgDefines.fdf.inc b/OvmfPkg/OvmfPkgDefines.fdf.inc
index 35fd454b97ab..401e491e4cbe 100644
--- a/OvmfPkg/OvmfPkgDefines.fdf.inc
+++ b/OvmfPkg/OvmfPkgDefines.fdf.inc
@@ -2,13 +2,14 @@
# FDF include file that defines the main macros and sets the dependent PCDs.
#
# Copyright (C) 2014, Red Hat, Inc.
-# Copyright (c) 2006 - 2013, Intel Corporation. All rights reserved.<BR>
+# Copyright (c) 2006 - 2021, Intel Corporation. All rights reserved.<BR>
#
# SPDX-License-Identifier: BSD-2-Clause-Patent
#
##
DEFINE BLOCK_SIZE = 0x1000
+DEFINE VARS_OFFSET = 0
#
# A firmware binary built with FD_SIZE_IN_KB=1024, and a firmware binary built
@@ -66,6 +67,7 @@ DEFINE SECFV_OFFSET = 0x003CC000
DEFINE SECFV_SIZE = 0x34000
!endif
+SET gUefiOvmfPkgTokenSpaceGuid.PcdOvmfImageSizeInKb = $(FD_SIZE_IN_KB)
SET gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFdBaseAddress = $(FW_BASE_ADDRESS)
SET gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFirmwareFdSize = $(FW_SIZE)
SET gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFirmwareBlockSize = $(BLOCK_SIZE)
@@ -82,6 +84,14 @@ SET gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwWorkingSize = $(BLOCK_SIZ
SET gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashNvStorageFtwSpareBase = gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashNvStorageFtwWorkingBase + gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwWorkingSize
SET gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareSize = $(VARS_SPARE_SIZE)
+SET gUefiOvmfPkgTokenSpaceGuid.PcdCfvBase = $(FW_BASE_ADDRESS)
+SET gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataOffset = $(VARS_OFFSET)
+SET gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataSize = $(VARS_LIVE_SIZE)
+
+SET gUefiOvmfPkgTokenSpaceGuid.PcdBfvBase = $(CODE_BASE_ADDRESS)
+SET gUefiOvmfPkgTokenSpaceGuid.PcdBfvRawDataOffset = $(VARS_SIZE)
+SET gUefiOvmfPkgTokenSpaceGuid.PcdBfvRawDataSize = $(CODE_SIZE)
+
!if $(SMM_REQUIRE) == TRUE
SET gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageVariableBase64 = gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashNvStorageVariableBase
SET gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwWorkingBase = gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashNvStorageFtwWorkingBase
--
2.29.2.windows.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 2/6] OvmfPkg: Add Tdx metadata
2021-07-12 1:19 [PATCH 0/6] Add Intel TDX support in OvmfPkg/ResetVector Min Xu
2021-07-12 1:19 ` [PATCH 1/6] OvmfPkg: Add Tdx BFV/CFV PCDs and PcdOvmfImageSizeInKb Min Xu
@ 2021-07-12 1:19 ` Min Xu
2021-07-12 1:19 ` [PATCH 3/6] UefiCpuPkg/ResetVector: Add InitTdx in UefiCpuPkg Min Xu
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Min Xu @ 2021-07-12 1:19 UTC (permalink / raw)
To: devel
Cc: Min Xu, Brijesh Singh, Erdem Aktas, James Bottomley, Jiewen Yao,
Laszlo Ersek, Tom Lendacky
RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429
Tdx Metadata describes the information about the image for VMM use.
For example, the base address and length of the TdHob, TdMailbox, etc.
Its offset is put in a GUID-ed structure which is appended in the GUID-ed
chain from a fixed GPA (0xffffffd0). Below are the items in TdxMetadata:
_Bfv:
Boot Firmware Volume
_Cfv:
Configuration Firmware Volume
_Stack:
Initial stack
_Heap:
Initial heap
_MailBox:
TDVF reserves the memory region so each AP can receive the message
sent by the guest OS.
_TdHob:
VMM pass the resource information in TdHob to TDVF.
_TdxPageTable:
If 5-level page table is supported (GPAW is 52), a top level page
directory pointers (1 * 256TB entry) is generated in this page.
_OvmfPageTable:
Initial page table for standard Ovmf.
TDVF indicate above chunk of temporary initialized memory region (_Stack/
_Heap/_MailBox/_TdHob/_TdxPageTables/OvmfPageTable) to support TDVF code
finishing the memory initialization. Because the other unaccepted memory
cannot be accessed until they're accepted.
Since AMD SEV has already defined some SEV specific memory region in
MEMFD. SEV and TDX will not run at the same time. So TDX re-use the
memory region defined by SEV.
- MailBox : PcdOvmfSecGhcbBackupBase|PcdOvmfSecGhcbBackupSize
- TdHob : PcdOvmfSecGhcbBase|PcdOvmfSecGhcbSize
- TdxPageTable : PcdOvmfSecGhcbPageTableBase|PcdOvmfSecGhcbPageTableSize
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm | 17 ++++
OvmfPkg/ResetVector/ResetVector.inf | 11 ++-
OvmfPkg/ResetVector/ResetVector.nasmb | 47 +++++++++-
OvmfPkg/ResetVector/X64/TdxMetadata.asm | 97 ++++++++++++++++++++
4 files changed, 169 insertions(+), 3 deletions(-)
create mode 100644 OvmfPkg/ResetVector/X64/TdxMetadata.asm
diff --git a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
index 9c0b5853a46f..ac86ce69ebe8 100644
--- a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
+++ b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
@@ -47,6 +47,23 @@ TIMES (15 - ((guidedStructureEnd - guidedStructureStart + 15) % 16)) DB 0
;
guidedStructureStart:
+%ifdef ARCH_X64
+;
+; TDX Metadata offset block
+;
+; If TdxMetadata.asm is included then we need below block which describes
+; the offset of TdxMetadata block in Ovmf image
+;
+; GUID : e47a6535-984a-4798-865e-4685a7bf8ec2
+;
+tdxMetadataOffsetStart:
+ DD (OVMF_IMAGE_SIZE_IN_KB * 1024 - (fourGigabytes - TdxMetadataGuid - 16))
+ DD tdxMetadataOffsetEnd - tdxMetadataOffsetStart
+ DB 0x35, 0x65, 0x7a, 0xe4, 0x4a, 0x98, 0x98, 0x47
+ DB 0x86, 0x5e, 0x46, 0x85, 0xa7, 0xbf, 0x8e, 0xc2
+tdxMetadataOffsetEnd:
+
+%endif
;
; SEV Secret block
;
diff --git a/OvmfPkg/ResetVector/ResetVector.inf b/OvmfPkg/ResetVector/ResetVector.inf
index dc38f68919cd..fd65c0c9621d 100644
--- a/OvmfPkg/ResetVector/ResetVector.inf
+++ b/OvmfPkg/ResetVector/ResetVector.inf
@@ -1,7 +1,7 @@
## @file
# Reset Vector
#
-# Copyright (c) 2006 - 2014, Intel Corporation. All rights reserved.<BR>
+# Copyright (c) 2006 - 2021, Intel Corporation. All rights reserved.<BR>
#
# SPDX-License-Identifier: BSD-2-Clause-Patent
#
@@ -43,6 +43,15 @@
gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPageTablesSize
gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPeiTempRamBase
gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPeiTempRamSize
+ gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecGhcbBackupBase
+ gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecGhcbBackupSize
+ gUefiOvmfPkgTokenSpaceGuid.PcdOvmfImageSizeInKb
+ gUefiOvmfPkgTokenSpaceGuid.PcdCfvBase
+ gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataOffset
+ gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataSize
+ gUefiOvmfPkgTokenSpaceGuid.PcdBfvBase
+ gUefiOvmfPkgTokenSpaceGuid.PcdBfvRawDataOffset
+ gUefiOvmfPkgTokenSpaceGuid.PcdBfvRawDataSize
[FixedPcd]
gUefiOvmfPkgTokenSpaceGuid.PcdSevLaunchSecretBase
diff --git a/OvmfPkg/ResetVector/ResetVector.nasmb b/OvmfPkg/ResetVector/ResetVector.nasmb
index 5fbacaed5f9d..b653fe87abd6 100644
--- a/OvmfPkg/ResetVector/ResetVector.nasmb
+++ b/OvmfPkg/ResetVector/ResetVector.nasmb
@@ -4,6 +4,7 @@
;
; Copyright (c) 2008 - 2013, Intel Corporation. All rights reserved.<BR>
; Copyright (c) 2020, Advanced Micro Devices, Inc. All rights reserved.<BR>
+; Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
; SPDX-License-Identifier: BSD-2-Clause-Patent
;
;------------------------------------------------------------------------------
@@ -67,6 +68,44 @@
%error "This implementation inherently depends on PcdOvmfSecGhcbBase not straddling a 2MB boundary"
%endif
+ ;
+ ; TDX meta data
+ ;
+ %define TDX_METADATA_SECTION_TYPE_BFV 0
+ %define TDX_METADATA_SECTION_TYPE_CFV 1
+ %define TDX_METADATA_SECTION_TYPE_TD_HOB 2
+ %define TDX_METADATA_SECTION_TYPE_TEMP_MEM 3
+ %define TDX_METADATA_VERSION 1
+ %define TDX_METADATA_ATTRIBUTES_EXTENDMR 0x00000001
+
+ %define TDX_BFV_RAW_DATA_OFFSET FixedPcdGet32 (PcdBfvRawDataOffset)
+ %define TDX_BFV_RAW_DATA_SIZE FixedPcdGet32 (PcdBfvRawDataSize)
+ %define TDX_BFV_MEMORY_BASE FixedPcdGet32 (PcdBfvBase)
+ %define TDX_BFV_MEMORY_SIZE FixedPcdGet32 (PcdBfvRawDataSize)
+
+ %define TDX_CFV_RAW_DATA_OFFSET FixedPcdGet32 (PcdCfvRawDataOffset)
+ %define TDX_CFV_RAW_DATA_SIZE FixedPcdGet32 (PcdCfvRawDataSize)
+ %define TDX_CFV_MEMORY_BASE FixedPcdGet32 (PcdCfvBase),
+ %define TDX_CFV_MEMORY_SIZE FixedPcdGet32 (PcdCfvRawDataSize),
+
+ %define TDX_HEAP_MEMORY_BASE FixedPcdGet32 (PcdOvmfSecPeiTempRamBase)
+ %define TDX_HEAP_MEMORY_SIZE FixedPcdGet32 (PcdOvmfSecPeiTempRamSize) / 2
+
+ %define TDX_STACK_MEMORY_BASE (TDX_HEAP_MEMORY_BASE + TDX_HEAP_MEMORY_SIZE)
+ %define TDX_STACK_MEMORY_SIZE FixedPcdGet32 (PcdOvmfSecPeiTempRamSize) / 2
+
+ %define TDX_HOB_MEMORY_BASE FixedPcdGet32 (PcdOvmfSecGhcbBase)
+ %define TDX_HOB_MEMORY_SIZE FixedPcdGet32 (PcdOvmfSecGhcbSize)
+
+ %define TDX_MAILBOX_MEMORY_BASE FixedPcdGet32 (PcdOvmfSecGhcbBackupBase)
+ %define TDX_MAILBOX_MEMORY_SIZE FixedPcdGet32 (PcdOvmfSecGhcbBackupSize)
+
+ %define OVMF_PAGE_TABLE_BASE FixedPcdGet32 (PcdOvmfSecPageTablesBase)
+ %define OVMF_PAGE_TABLE_SIZE FixedPcdGet32 (PcdOvmfSecPageTablesSize)
+
+ %define TDX_EXTRA_PAGE_TABLE_BASE FixedPcdGet32 (PcdOvmfSecGhcbPageTableBase)
+ %define TDX_EXTRA_PAGE_TABLE_SIZE FixedPcdGet32 (PcdOvmfSecGhcbPageTableSize)
+
%define PT_ADDR(Offset) (FixedPcdGet32 (PcdOvmfSecPageTablesBase) + (Offset))
%define GHCB_PT_ADDR (FixedPcdGet32 (PcdOvmfSecGhcbPageTableBase))
@@ -76,8 +115,11 @@
%define SEV_ES_WORK_AREA_RDRAND (FixedPcdGet32 (PcdSevEsWorkAreaBase) + 8)
%define SEV_ES_WORK_AREA_ENC_MASK (FixedPcdGet32 (PcdSevEsWorkAreaBase) + 16)
%define SEV_ES_VC_TOP_OF_STACK (FixedPcdGet32 (PcdOvmfSecPeiTempRamBase) + FixedPcdGet32 (PcdOvmfSecPeiTempRamSize))
-%include "Ia32/Flat32ToFlat64.asm"
-%include "Ia32/PageTables64.asm"
+
+ %include "X64/TdxMetadata.asm"
+
+ %include "Ia32/Flat32ToFlat64.asm"
+ %include "Ia32/PageTables64.asm"
%endif
%include "Ia16/Real16ToFlat32.asm"
@@ -88,5 +130,6 @@
%define SEV_ES_AP_RESET_IP FixedPcdGet32 (PcdSevEsWorkAreaBase)
%define SEV_LAUNCH_SECRET_BASE FixedPcdGet32 (PcdSevLaunchSecretBase)
%define SEV_LAUNCH_SECRET_SIZE FixedPcdGet32 (PcdSevLaunchSecretSize)
+ %define OVMF_IMAGE_SIZE_IN_KB FixedPcdGet32 (PcdOvmfImageSizeInKb)
%include "Ia16/ResetVectorVtf0.asm"
diff --git a/OvmfPkg/ResetVector/X64/TdxMetadata.asm b/OvmfPkg/ResetVector/X64/TdxMetadata.asm
new file mode 100644
index 000000000000..8dba8daa0165
--- /dev/null
+++ b/OvmfPkg/ResetVector/X64/TdxMetadata.asm
@@ -0,0 +1,97 @@
+;------------------------------------------------------------------------------
+; @file
+; Tdx Virtual Firmware metadata
+;
+; Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+; SPDX-License-Identifier: BSD-2-Clause-Patent
+;
+;------------------------------------------------------------------------------
+
+BITS 64
+
+%define TDX_VIRTUAL_FIRMWARE
+
+ALIGN 16
+TIMES (15 - ((TdxGuidedStructureEnd - TdxGuidedStructureStart + 15) % 16)) DB 0
+
+TdxGuidedStructureStart:
+
+;
+; TDVF meta data
+;
+TdxMetadataGuid:
+ DB 0xf3, 0xf9, 0xea, 0xe9, 0x8e, 0x16, 0xd5, 0x44
+ DB 0xa8, 0xeb, 0x7f, 0x4d, 0x87, 0x38, 0xf6, 0xae
+
+_Descriptor:
+ DB 'T','D','V','F' ; Signature
+ DD TdxGuidedStructureEnd - _Descriptor ; Length
+ DD TDX_METADATA_VERSION ; Version
+ DD (TdxGuidedStructureEnd - _Descriptor - 16)/32 ; Number of sections
+
+_Bfv:
+ DD TDX_BFV_RAW_DATA_OFFSET
+ DD TDX_BFV_RAW_DATA_SIZE
+ DQ TDX_BFV_MEMORY_BASE
+ DQ TDX_BFV_MEMORY_SIZE
+ DD TDX_METADATA_SECTION_TYPE_BFV
+ DD TDX_METADATA_ATTRIBUTES_EXTENDMR
+
+_Cfv:
+ DD TDX_CFV_RAW_DATA_OFFSET
+ DD TDX_CFV_RAW_DATA_SIZE
+ DQ TDX_CFV_MEMORY_BASE
+ DQ TDX_CFV_MEMORY_SIZE
+ DD TDX_METADATA_SECTION_TYPE_CFV
+ DD 0
+
+_Stack:
+ DD 0
+ DD 0
+ DQ TDX_STACK_MEMORY_BASE
+ DQ TDX_STACK_MEMORY_SIZE
+ DD TDX_METADATA_SECTION_TYPE_TEMP_MEM
+ DD 0
+
+_Heap:
+ DD 0
+ DD 0
+ DQ TDX_HEAP_MEMORY_BASE
+ DQ TDX_HEAP_MEMORY_SIZE
+ DD TDX_METADATA_SECTION_TYPE_TEMP_MEM
+ DD 0
+
+_MailBox:
+ DD 0
+ DD 0
+ DQ TDX_MAILBOX_MEMORY_BASE
+ DQ TDX_MAILBOX_MEMORY_SIZE
+ DD TDX_METADATA_SECTION_TYPE_TEMP_MEM
+ DD 0
+
+_TdHob:
+ DD 0
+ DD 0
+ DQ TDX_HOB_MEMORY_BASE
+ DQ TDX_HOB_MEMORY_SIZE
+ DD TDX_METADATA_SECTION_TYPE_TD_HOB
+ DD 0
+
+_TdxPageTable:
+ DD 0
+ DD 0
+ DQ TDX_EXTRA_PAGE_TABLE_BASE
+ DQ TDX_EXTRA_PAGE_TABLE_SIZE
+ DD TDX_METADATA_SECTION_TYPE_TEMP_MEM
+ DD 0
+
+_OvmfPageTable:
+ DD 0
+ DD 0
+ DQ OVMF_PAGE_TABLE_BASE
+ DQ OVMF_PAGE_TABLE_SIZE
+ DD TDX_METADATA_SECTION_TYPE_TEMP_MEM
+ DD 0
+
+TdxGuidedStructureEnd:
+ALIGN 16
--
2.29.2.windows.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 3/6] UefiCpuPkg/ResetVector: Add InitTdx in UefiCpuPkg
2021-07-12 1:19 [PATCH 0/6] Add Intel TDX support in OvmfPkg/ResetVector Min Xu
2021-07-12 1:19 ` [PATCH 1/6] OvmfPkg: Add Tdx BFV/CFV PCDs and PcdOvmfImageSizeInKb Min Xu
2021-07-12 1:19 ` [PATCH 2/6] OvmfPkg: Add Tdx metadata Min Xu
@ 2021-07-12 1:19 ` Min Xu
2021-07-12 1:19 ` [PATCH 4/6] UefiCpuPkg/ResetVector: Add ReloadFlat32 " Min Xu
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Min Xu @ 2021-07-12 1:19 UTC (permalink / raw)
To: devel; +Cc: Min Xu, Eric Dong, Jiewen Yao, Ray Ni, Laszlo Ersek
RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429
In ResetVector phase some Tdx initialization may be needed, for example,
recording the Tdx flag ('TDXG'), CPU GPA width, etc. It will be called by
the Main32 entry point in Main.asm. InitTdx.asm in UefiCpuPkg/ResetVector
is a placeholder and the actual initialization is done in other Pkg, such
as OvmfPkg.
Cc: Eric Dong <eric.dong@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
UefiCpuPkg/ResetVector/Vtf0/Ia32/InitTdx.asm | 15 +++++++++++++++
1 file changed, 15 insertions(+)
create mode 100644 UefiCpuPkg/ResetVector/Vtf0/Ia32/InitTdx.asm
diff --git a/UefiCpuPkg/ResetVector/Vtf0/Ia32/InitTdx.asm b/UefiCpuPkg/ResetVector/Vtf0/Ia32/InitTdx.asm
new file mode 100644
index 000000000000..feb917779fbd
--- /dev/null
+++ b/UefiCpuPkg/ResetVector/Vtf0/Ia32/InitTdx.asm
@@ -0,0 +1,15 @@
+;------------------------------------------------------------------------------
+; @file
+; Tdx Initialization.
+;
+; Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+; SPDX-License-Identifier: BSD-2-Clause-Patent
+;
+;------------------------------------------------------------------------------
+
+BITS 32
+
+InitTdx:
+ nop
+doneTdxInit:
+ OneTimeCallRet InitTdx
--
2.29.2.windows.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 4/6] UefiCpuPkg/ResetVector: Add ReloadFlat32 in UefiCpuPkg
2021-07-12 1:19 [PATCH 0/6] Add Intel TDX support in OvmfPkg/ResetVector Min Xu
` (2 preceding siblings ...)
2021-07-12 1:19 ` [PATCH 3/6] UefiCpuPkg/ResetVector: Add InitTdx in UefiCpuPkg Min Xu
@ 2021-07-12 1:19 ` Min Xu
2021-07-12 1:19 ` [PATCH 5/6] UefiCpuPkg/ResetVector: Add Main32 entry point in Main.asm Min Xu
2021-07-12 1:19 ` [PATCH 6/6] OvmfPkg/ResetVector: Update ResetVector to support Tdx Min Xu
5 siblings, 0 replies; 7+ messages in thread
From: Min Xu @ 2021-07-12 1:19 UTC (permalink / raw)
To: devel; +Cc: Min Xu, Eric Dong, Jiewen Yao, Ray Ni, Laszlo Ersek
RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429
Load the GDT and set the default CR0, then jump to Flat 32 protected mode.
Cc: Eric Dong <eric.dong@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
.../ResetVector/Vtf0/Ia32/ReloadFlat32.asm | 43 +++++++++++++++++++
1 file changed, 43 insertions(+)
create mode 100644 UefiCpuPkg/ResetVector/Vtf0/Ia32/ReloadFlat32.asm
diff --git a/UefiCpuPkg/ResetVector/Vtf0/Ia32/ReloadFlat32.asm b/UefiCpuPkg/ResetVector/Vtf0/Ia32/ReloadFlat32.asm
new file mode 100644
index 000000000000..e8a4c47762df
--- /dev/null
+++ b/UefiCpuPkg/ResetVector/Vtf0/Ia32/ReloadFlat32.asm
@@ -0,0 +1,43 @@
+;------------------------------------------------------------------------------
+; @file
+; Load the GDT and set the CR0, then jump to Flat 32 protected mode.
+;
+; Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+; SPDX-License-Identifier: BSD-2-Clause-Patent
+;
+;------------------------------------------------------------------------------
+
+%define SEC_DEFAULT_CR0 0x00000023
+%define SEC_DEFAULT_CR4 0x640
+
+BITS 32
+
+;
+; Modified: EAX, EBX, CR0, CR4, DS, ES, FS, GS, SS
+;
+ReloadFlat32:
+
+ cli
+ mov ebx, ADDR_OF(gdtr)
+ lgdt [ebx]
+
+ mov eax, SEC_DEFAULT_CR0
+ mov cr0, eax
+
+ jmp LINEAR_CODE_SEL:dword ADDR_OF(jumpToFlat32BitAndLandHere)
+BITS 32
+jumpToFlat32BitAndLandHere:
+
+ mov eax, SEC_DEFAULT_CR4
+ mov cr4, eax
+
+ debugShowPostCode POSTCODE_32BIT_MODE
+
+ mov ax, LINEAR_SEL
+ mov ds, ax
+ mov es, ax
+ mov fs, ax
+ mov gs, ax
+ mov ss, ax
+
+ OneTimeCallRet ReloadFlat32
--
2.29.2.windows.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 5/6] UefiCpuPkg/ResetVector: Add Main32 entry point in Main.asm
2021-07-12 1:19 [PATCH 0/6] Add Intel TDX support in OvmfPkg/ResetVector Min Xu
` (3 preceding siblings ...)
2021-07-12 1:19 ` [PATCH 4/6] UefiCpuPkg/ResetVector: Add ReloadFlat32 " Min Xu
@ 2021-07-12 1:19 ` Min Xu
2021-07-12 1:19 ` [PATCH 6/6] OvmfPkg/ResetVector: Update ResetVector to support Tdx Min Xu
5 siblings, 0 replies; 7+ messages in thread
From: Min Xu @ 2021-07-12 1:19 UTC (permalink / raw)
To: devel; +Cc: Min Xu, Eric Dong, Ray Ni, Jiewen Yao, Laszlo Ersek
RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429
In Tdx all CPUs "reset" to run on 32-bit protected mode with flat
descriptor (paging disabled). Main32 entry point is added in
UefiCpuPkg/ResetVector/Vtf0/Main.asm so that Main.asm can support
the 32-bit protected mode.
InitTdx.asm and ReloadFlat32.asm are added in Vtf0/Vtf0.nasmb as well.
Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
UefiCpuPkg/ResetVector/Vtf0/Main.asm | 45 ++++++++++++++++++++++++++
UefiCpuPkg/ResetVector/Vtf0/Vtf0.nasmb | 2 ++
2 files changed, 47 insertions(+)
diff --git a/UefiCpuPkg/ResetVector/Vtf0/Main.asm b/UefiCpuPkg/ResetVector/Vtf0/Main.asm
index 19d08482f831..90e1dc678233 100644
--- a/UefiCpuPkg/ResetVector/Vtf0/Main.asm
+++ b/UefiCpuPkg/ResetVector/Vtf0/Main.asm
@@ -36,6 +36,51 @@ Main16:
BITS 32
+%ifdef ARCH_X64
+
+ jmp SearchBfv
+
+;
+; Modified: EBX, ECX, EDX, EBP, EDI, ESP
+;
+; @param[in,out] RAX/EAX 0
+; @param[in] RFLAGS 2
+; @param[in] RCX [31:0] TDINITVP - Untrusted Configuration
+; [63:32] 0
+; @param[in] RDX [31:0] VCPUID
+; [63:32] 0
+; @param[in] RBX [6:0] CPU supported GPA width
+; [7:7] 5 level page table support
+; [63:8] 0
+; @param[in] RSI [31:0] VCPU_Index
+; [63:32] 0
+; @param[in] RDI/EDI 0
+; @param[in] RBP/EBP 0
+; @param[in] R8 Same as RCX
+; @param[out] RBP/EBP Address of Boot Firmware Volume (BFV)
+; @param[out] DS Selector allowing flat access to all addresses
+; @param[out] ES Selector allowing flat access to all addresses
+; @param[out] FS Selector allowing flat access to all addresses
+; @param[out] GS Selector allowing flat access to all addresses
+; @param[out] SS Selector allowing flat access to all addresses
+;
+; @return None This routine jumps to SEC and does not return
+Main32:
+ ;
+ ; Save EBX in EBP because EBX will be changed in ReloadFlat32
+ ;
+ mov ebp, ebx
+
+ OneTimeCall ReloadFlat32
+
+ ;
+ ; Init Tdx
+ ;
+ OneTimeCall InitTdx
+
+%endif
+
+SearchBfv:
;
; Search for the Boot Firmware Volume (BFV)
;
diff --git a/UefiCpuPkg/ResetVector/Vtf0/Vtf0.nasmb b/UefiCpuPkg/ResetVector/Vtf0/Vtf0.nasmb
index 493738c79c1c..663d6ddc4d24 100644
--- a/UefiCpuPkg/ResetVector/Vtf0/Vtf0.nasmb
+++ b/UefiCpuPkg/ResetVector/Vtf0/Vtf0.nasmb
@@ -51,6 +51,8 @@
%include "Ia32/SearchForSecEntry.asm"
%ifdef ARCH_X64
+%include "Ia32/InitTdx.asm"
+%include "Ia32/ReloadFlat32.asm"
%include "Ia32/Flat32ToFlat64.asm"
%include "Ia32/PageTables64.asm"
%endif
--
2.29.2.windows.2
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6/6] OvmfPkg/ResetVector: Update ResetVector to support Tdx
2021-07-12 1:19 [PATCH 0/6] Add Intel TDX support in OvmfPkg/ResetVector Min Xu
` (4 preceding siblings ...)
2021-07-12 1:19 ` [PATCH 5/6] UefiCpuPkg/ResetVector: Add Main32 entry point in Main.asm Min Xu
@ 2021-07-12 1:19 ` Min Xu
5 siblings, 0 replies; 7+ messages in thread
From: Min Xu @ 2021-07-12 1:19 UTC (permalink / raw)
To: devel
Cc: Min Xu, Brijesh Singh, Erdem Aktas, James Bottomley, Jiewen Yao,
Laszlo Ersek, Tom Lendacky
RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429
In Tdx all CPUs "reset" to run on 32-bit protected mode with flat
descriptor (paging disabled). But in Non-Td guest the initial state of
CPUs is 16-bit real mode. To resolve this conflict, BITS 16/32 is used
in the very beginning of ResetVector. It will check the 32-bit protected
mode or 16-bit real mode, then jump to the corresponding entry point.
This is done in OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm.
InitTdx.asm is called to record the Tdx signature ('TDXG') and other tdx
information in a TDX_WORK_AREA which can be used by the other routines in
ResetVector.
Main32 in UefiCpuPkg/ResetVector/Vtf0/Main.asm is the entry point for
Tdx guest. It call ReloadFlat32 to load the GDT and set the CR0, then
jump to flat32. After that InitTdx is called to do the above Tdx initialization.
Then Tdx jumps to 64-bit long mode by doing following tasks:
1. SetCr3ForPageTables64
For OVMF, some initial page tables is built at:
PcdOvmfSecPageTablesBase - (PcdOvmfSecPageTablesBase + 0x6000)
This page table supports the 4-level page table.
But Tdx support 4-level and 5-level page table based on the CPU GPA width.
48bit is 4-level paging, 52-bit is 5-level paging.
If 5-level page table is supported (GPAW is 52), then a top level
page directory pointers (1 * 256TB entry) is generated in the
TdxPageTable.
2. Set Cr4
Enable PAE.
3. Adjust Cr3
If GPAW is 48, then Cr3 is PT_ADDR (0). If GPAW is 52, then Cr3 is
TDX_PT_ADDR (0).
Tdx MailBox [0x10, 0x800] is reserved for OS. So we initialize piece of this
area ([0x10, 0x20]) to record the Tdx flag ('TDXG') and other Tdx info so that
they can be used in the following flow.
After all above is successfully done, Tdx jump to SecEntry.
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm | 21 ++++++++
OvmfPkg/ResetVector/Ia32/Flat32ToFlat64.asm | 47 ++++++++++++++++
OvmfPkg/ResetVector/Ia32/InitTdx.asm | 57 ++++++++++++++++++++
OvmfPkg/ResetVector/Ia32/PageTables64.asm | 41 ++++++++++++++
OvmfPkg/ResetVector/ResetVector.nasmb | 17 ++++++
5 files changed, 183 insertions(+)
create mode 100644 OvmfPkg/ResetVector/Ia32/InitTdx.asm
diff --git a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
index ac86ce69ebe8..a390ed81d021 100644
--- a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
+++ b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
@@ -155,10 +155,31 @@ resetVector:
;
; This is where the processor will begin execution
;
+; In IA32 we follow the standard reset vector flow. While in X64, Td guest
+; may be supported. Td guest requires the startup mode to be 32-bit
+; protected mode but the legacy VM startup mode is 16-bit real mode.
+; To make NASM generate such shared entry code that behaves correctly in
+; both 16-bit and 32-bit mode, more BITS directives are added.
+;
+%ifdef ARCH_IA32
+
nop
nop
jmp EarlyBspInitReal16
+%else
+
+ smsw ax
+ test al, 1
+ jz .Real
+BITS 32
+ jmp Main32
+BITS 16
+.Real:
+ jmp EarlyBspInitReal16
+
+%endif
+
ALIGN 16
fourGigabytes:
diff --git a/OvmfPkg/ResetVector/Ia32/Flat32ToFlat64.asm b/OvmfPkg/ResetVector/Ia32/Flat32ToFlat64.asm
index c6d0d898bcd1..2206ca719593 100644
--- a/OvmfPkg/ResetVector/Ia32/Flat32ToFlat64.asm
+++ b/OvmfPkg/ResetVector/Ia32/Flat32ToFlat64.asm
@@ -17,6 +17,9 @@ Transition32FlatTo64Flat:
OneTimeCall SetCr3ForPageTables64
+ cmp dword[TDX_WORK_AREA], 0x47584454 ; 'TDXG'
+ jz TdxTransition32FlatTo64Flat
+
mov eax, cr4
bts eax, 5 ; enable PAE
mov cr4, eax
@@ -65,10 +68,54 @@ EnablePaging:
bts eax, 31 ; set PG
mov cr0, eax ; enable paging
+ jmp _jumpTo64Bit
+
+;
+; Tdx Transition from 32Flat to 64Flat
+;
+TdxTransition32FlatTo64Flat:
+
+ mov eax, cr4
+ bts eax, 5 ; enable PAE
+
+ ;
+ ; byte[TDX_WORK_AREA_PAGELEVEL5] holds the indicator whether 52bit is supported.
+ ; if it is the case, need to set LA57 and use 5-level paging
+ ;
+ cmp byte[TDX_WORK_AREA_PAGELEVEL5], 0
+ jz .set_cr4
+ bts eax, 12
+.set_cr4:
+ mov cr4, eax
+ mov ebx, cr3
+
+ ;
+ ; if la57 is not set, we are ok
+ ; if using 5-level paging, adjust top-level page directory
+ ;
+ bt eax, 12
+ jnc .set_cr3
+ mov ebx, TDX_PT_ADDR (0)
+.set_cr3:
+ mov cr3, ebx
+
+ mov eax, cr0
+ bts eax, 31 ; set PG
+ mov cr0, eax ; enable paging
+
+_jumpTo64Bit:
jmp LINEAR_CODE64_SEL:ADDR_OF(jumpTo64BitAndLandHere)
+
BITS 64
jumpTo64BitAndLandHere:
+ ;
+ ; For Td guest we are done and jump to the end
+ ;
+ mov eax, TDX_WORK_AREA
+ cmp dword [eax], 0x47584454 ; 'TDXG'
+ jz GoodCompare
+
;
; Check if the second step of the SEV-ES mitigation is to be performed.
;
diff --git a/OvmfPkg/ResetVector/Ia32/InitTdx.asm b/OvmfPkg/ResetVector/Ia32/InitTdx.asm
new file mode 100644
index 000000000000..de8273da6a0c
--- /dev/null
+++ b/OvmfPkg/ResetVector/Ia32/InitTdx.asm
@@ -0,0 +1,57 @@
+;------------------------------------------------------------------------------
+; @file
+; Initialize TDX_WORK_AREA to record the Tdx flag ('TDXG') and other Tdx info
+; so that the following codes can use these information.
+;
+; Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+; SPDX-License-Identifier: BSD-2-Clause-Patent
+;
+;------------------------------------------------------------------------------
+
+BITS 32
+
+;
+; Modified: EBP
+;
+InitTdx:
+ ;
+ ; In Td guest, BSP/AP shares the same entry point
+ ; BSP builds up the page table, while APs shouldn't do the same task.
+ ; Instead, APs just leverage the page table which is built by BSP.
+ ; APs will wait until the page table is ready.
+ ; In Td guest, vCPU 0 is treated as the BSP, the others are APs.
+ ; ESI indicates the vCPU ID.
+ ;
+ cmp esi, 0
+ je tdBspEntry
+
+apWait:
+ cmp byte[TDX_WORK_AREA_PGTBL_READY], 0
+ je apWait
+ jmp doneTdxInit
+
+tdBspEntry:
+ ;
+ ; It is of Tdx Guest
+ ; Save the Tdx info in TDX_WORK_AREA so that the following code can use
+ ; these information.
+ ;
+ mov dword [TDX_WORK_AREA], 0x47584454 ; 'TDXG'
+
+ ;
+ ; EBP[6:0] CPU supported GPA width
+ ;
+ and ebp, 0x3f
+ cmp ebp, 52
+ jl NotPageLevel5
+ mov byte[TDX_WORK_AREA_PAGELEVEL5], 1
+
+NotPageLevel5:
+ ;
+ ; ECX[31:0] TDINITVP - Untrusted Configuration
+ ;
+ mov DWORD[TDX_WORK_AREA_INITVP], ecx
+ mov DWORD[TDX_WORK_AREA_INFO], ebp
+
+doneTdxInit:
+ OneTimeCallRet InitTdx
diff --git a/OvmfPkg/ResetVector/Ia32/PageTables64.asm b/OvmfPkg/ResetVector/Ia32/PageTables64.asm
index 5fae8986d9da..508df6cf5967 100644
--- a/OvmfPkg/ResetVector/Ia32/PageTables64.asm
+++ b/OvmfPkg/ResetVector/Ia32/PageTables64.asm
@@ -218,6 +218,24 @@ SevEsDisabled:
;
SetCr3ForPageTables64:
+ ;
+ ; Check Td guest
+ ;
+ cmp dword[TDX_WORK_AREA], 0x47584454 ; 'TDXG'
+ jnz CheckSev
+
+ xor edx, edx
+
+ ;
+ ; In Td guest, BSP builds the page table and set the flag of
+ ; TDX_WORK_AREA_PGTBL_READY. APs check this flag and then set
+ ; cr3 directly.
+ ;
+ cmp byte[TDX_WORK_AREA_PGTBL_READY], 1
+ jz SetCr3
+ jmp SevNotActive
+
+CheckSev:
OneTimeCall CheckSevFeatures
xor edx, edx
test eax, eax
@@ -277,6 +295,29 @@ pageTableEntriesLoop:
mov [(ecx * 8 + PT_ADDR (0x2000 - 8)) + 4], edx
loop pageTableEntriesLoop
+ ;
+ ; If it is Td guest, TdxExtraPageTable should be initialized as well
+ ;
+ cmp dword[TDX_WORK_AREA], 0x47584454 ; 'TDXG'
+ jnz IsSevEs
+
+ xor eax, eax
+ mov ecx, 0x400
+tdClearTdxPageTablesMemoryLoop:
+ mov dword [ecx * 4 + TDX_PT_ADDR (0) - 4], eax
+ loop tdClearTdxPageTablesMemoryLoop
+
+ xor edx, edx
+ ;
+ ; Top level Page Directory Pointers (1 * 256TB entry)
+ ;
+ mov dword[TDX_PT_ADDR (0)], PT_ADDR (0) + PAGE_PDP_ATTR
+ mov dword[TDX_PT_ADDR (4)], edx
+
+ mov byte[TDX_WORK_AREA_PGTBL_READY], 1
+ jmp SetCr3
+
+IsSevEs:
OneTimeCall IsSevEsEnabled
test eax, eax
jz SetCr3
diff --git a/OvmfPkg/ResetVector/ResetVector.nasmb b/OvmfPkg/ResetVector/ResetVector.nasmb
index b653fe87abd6..47ea23095c0a 100644
--- a/OvmfPkg/ResetVector/ResetVector.nasmb
+++ b/OvmfPkg/ResetVector/ResetVector.nasmb
@@ -106,6 +106,21 @@
%define TDX_EXTRA_PAGE_TABLE_BASE FixedPcdGet32 (PcdOvmfSecGhcbPageTableBase)
%define TDX_EXTRA_PAGE_TABLE_SIZE FixedPcdGet32 (PcdOvmfSecGhcbPageTableSize)
+ ;
+ ; TdMailboxBase [0x10, 0x800] is reserved for OS.
+ ; Td guest initialize piece of this area (TdMailboxBase [0x10,0x20]) to
+ ; record the Td guest info so that this information can be used in the
+ ; following ResetVector flow.
+ ;
+ %define TD_MAILBOX_WORKAREA_OFFSET 0x10
+ %define TDX_WORK_AREA (TDX_MAILBOX_MEMORY_BASE + TD_MAILBOX_WORKAREA_OFFSET)
+ %define TDX_WORK_AREA_PAGELEVEL5 (TDX_WORK_AREA + 4)
+ %define TDX_WORK_AREA_PGTBL_READY (TDX_WORK_AREA + 5)
+ %define TDX_WORK_AREA_INITVP (TDX_WORK_AREA + 8)
+ %define TDX_WORK_AREA_INFO (TDX_WORK_AREA + 8 + 4)
+
+ %define TDX_PT_ADDR(Offset) (TDX_EXTRA_PAGE_TABLE_BASE + (Offset))
+
%define PT_ADDR(Offset) (FixedPcdGet32 (PcdOvmfSecPageTablesBase) + (Offset))
%define GHCB_PT_ADDR (FixedPcdGet32 (PcdOvmfSecGhcbPageTableBase))
@@ -117,6 +132,8 @@
%define SEV_ES_VC_TOP_OF_STACK (FixedPcdGet32 (PcdOvmfSecPeiTempRamBase) + FixedPcdGet32 (PcdOvmfSecPeiTempRamSize))
%include "X64/TdxMetadata.asm"
+ %include "Ia32/InitTdx.asm"
+ %include "Ia32/ReloadFlat32.asm"
%include "Ia32/Flat32ToFlat64.asm"
%include "Ia32/PageTables64.asm"
--
2.29.2.windows.2
^ permalink raw reply related [flat|nested] 7+ messages in thread