From: "Ard Biesheuvel via groups.io" <ardb=kernel.org@groups.io>
To: devel@edk2.groups.io, f.weber@proxmox.com
Subject: Re: [edk2-devel] Avoid split lock detection warnings triggered by OVMF?
Date: Thu, 5 Jun 2025 09:04:36 +0200 [thread overview]
Message-ID: <CAMj1kXEjXe7be5MZiRyos8TCOttOwLzVYemyuFtdfedYDTNSkg@mail.gmail.com> (raw)
In-Reply-To: <5d302e42-970f-4468-8d0c-fb564250dea9@proxmox.com>
Thanks for the elaborate report. Does the following help?
diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.h
b/UefiCpuPkg/Library/MpInitLib/MpLib.h
index a63bb81bef88..a9b3fa77e14d 100644
--- a/UefiCpuPkg/Library/MpInitLib/MpLib.h
+++ b/UefiCpuPkg/Library/MpInitLib/MpLib.h
@@ -213,8 +213,6 @@ typedef struct {
UINTN StackStart;
UINTN StackSize;
UINTN CFunction;
- IA32_DESCRIPTOR GdtrProfile;
- IA32_DESCRIPTOR IdtrProfile;
UINTN BufferStart;
UINTN ModeOffset;
UINTN ApIndex;
@@ -227,6 +225,8 @@ typedef struct {
UINTN NumApsExecuting;
CPU_MP_DATA *CpuMpData;
UINTN InitializeFloatingPointUnitsAddress;
+ IA32_DESCRIPTOR GdtrProfile;
+ IA32_DESCRIPTOR IdtrProfile;
UINT32 ModeTransitionMemory;
UINT16 ModeTransitionSegment;
UINT32 ModeHighMemory;
diff --git a/UefiCpuPkg/Library/MpInitLib/MpEqu.inc
b/UefiCpuPkg/Library/MpInitLib/MpEqu.inc
index d8ba9ea1246c..d67c4c923c82 100644
--- a/UefiCpuPkg/Library/MpInitLib/MpEqu.inc
+++ b/UefiCpuPkg/Library/MpInitLib/MpEqu.inc
@@ -74,8 +74,6 @@ struc MP_CPU_EXCHANGE_INFO
.StackStart: CTYPE_UINTN 1
.StackSize: CTYPE_UINTN 1
.CFunction: CTYPE_UINTN 1
- .GdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size
- .IdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size
.BufferStart: CTYPE_UINTN 1
.ModeOffset: CTYPE_UINTN 1
.ApIndex: CTYPE_UINTN 1
@@ -88,6 +86,8 @@ struc MP_CPU_EXCHANGE_INFO
.NumApsExecuting: CTYPE_UINTN 1
.CpuMpData: CTYPE_UINTN 1
.InitializeFloatingPointUnits: CTYPE_UINTN 1
+ .GdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size
+ .IdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size
.ModeTransitionMemory: CTYPE_UINT32 1
.ModeTransitionSegment: CTYPE_UINT16 1
.ModeHighMemory: CTYPE_UINT32 1
On Thu, 5 Jun 2025 at 06:18, Friedrich Weber <f.weber@proxmox.com> wrote:
>
> Hi,
>
> Some of our Proxmox VE users report that on CPUs with the split_lock_detect
> flag (such as recent Intel CPUs), QEMU+KVM VMs using our build of OVMF trigger
> split lock detection warnings on the host on VM start (only for VMs with more
> than one core, though), e.g.:
>
> Jun 03 10:16:25 host kernel: x86/split lock detection: #AC: CPU 1/KVM/26750 took a split_lock trap at address: 0x7ff4b050
>
> But I can also reproduce this on upstream QEMU and current OVMF master, see
> below.
>
> In my understanding, split locks can be caused by misaligned atomic memory
> accesses, and by default kernels >= 5.19 artifically slow down threads that
> cause a split-lock operation for ~10ms [1]. One complication is that the
> warning above is only logged once per thread, so if a thread causes more such
> operations and is again slowed down, this is not logged again. We have put
> together a bpftrace command that logs subsequent slowdowns, see [2].
>
> Using bpftrace, we can see that each thread can cause 0-2 such split-lock
> operation on boot, so each thread is potentially slowed down for 10-20ms in
> total. This slowdown is negligible, but the warnings have other downsides:
> First, they look a bit scary, and second, in case the actual VM workload causes
> split-lock operations later on, these early and somewhat spurious warnings
> "mask" later actual relevant warnings. So I was wondering -- do you think OVMF
> could be patched to avoid the split-lock operations and thus avoid the warning?
>
> Some more details -- I can reproduce the issue on current EDK II master
> (8c04bcc7ed0efbb2e3fde23f787ff0249e62f874) with the following steps:
>
> - have a CPU with split_lock_detect flag, in my case Intel i7-13700K,
> split_lock_mitigate sysctl at its default 1
>
> - build OVMF (the PcdFirmwareVendor is not necessary, just to avoid confusing
> myself)
>
> build -a IA32 -a X64 -t GCC -p OvmfPkg/OvmfPkgIa32X64.dsc --pcd PcdFirmwareVendor=L"built at $(date)\\0"
>
> - copy over the resulting Build/Ovmf3264/DEBUG_GCC/FV/OVMF_CODE.fd
> - run bpftrace in parallel:
>
> bpftrace -e 'kfunc:vmlinux:split_lock_warn /0<*(uint32*)kaddr("sysctl_sld_mitigate")/{time("%H:%M:%S: "); printf("comm=%s tid=%d ip=0x%x\n", comm, tid, args->ip);}'
>
> - run upstream QEMU 9.2.4 with this firmware:
>
> /path/to/qemu-9.2.4/qemu-system-x86_64 -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd -smp 4,sockets=1,cores=4 -accel kvm
>
> - find in the journal:
>
> Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62190 took a split_lock trap at address: 0x7e8a050
> Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62188 took a split_lock trap at address: 0x7e8a050
> Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62189 took a split_lock trap at address: 0x7e8a050
>
> - find in the bpftrace:
>
> 10:37:34: comm=qemu-system-x86 tid=62190 ip=0x7e8a050
> 10:37:34: comm=qemu-system-x86 tid=62188 ip=0x7e8a050
> 10:37:34: comm=qemu-system-x86 tid=62189 ip=0x7e8a050
>
> The ip seems to be the same on each QEMU run. So I dumped the surrounding
> instructions using -monitor stdio and:
>
> memsave 0x7e8a000 0x1000 memdump.raw
>
> and
>
> ndisasm -b64 memsave.raw
>
> and 0x50 with some context is (full output in [3]):
>
> 0000004B 89F7 mov edi,esi
> 0000004D 83C77D add edi,byte +0x7d
> 00000050 F0FF07 lock inc dword [rdi]
>
> where the last one may indeed be a misaligned atomic memory access?
>
> The disassembly looks like it could match this portion of LongModeStart in
> UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm:
>
> LongModeStart:
> [...]
>
> ; Increment the number of APs executing here as early as possible
> ; This is decremented in C code when AP is finished executing
> mov edi, esi
> add edi, MP_CPU_EXCHANGE_INFO_FIELD (NumApsExecuting)
> lock inc dword [edi]
>
> ... so NumApsExecuting in MP_CPU_EXCHANGE_INFO *may* be the culprit, but this
> is really speculation on my part :)
>
> I know nothing about EDK II/OVMF internals, so even when assuming the above
> speculation makes sense, I can't judge whether the above MP_CPU_EXCHANGE_INFO
> could perhaps be rearranged/padded such that it doesn't trigger the split lock
> detection. What do you think?
>
> Happy to provide more information if needed, just let me know.
>
> Thanks and best wishes,
>
> Friedrich
>
> [1] https://lwn.net/Articles/911219/
> [2] https://pve.proxmox.com/wiki/Split_lock_detection
> [3]
>
> 00000000 8EDA mov ds,edx
> 00000002 8EC2 mov es,edx
> 00000004 8EE2 mov fs,edx
> 00000006 8EEA mov gs,edx
> 00000008 8ED2 mov ss,edx
> 0000000A 89DE mov esi,ebx
> 0000000C 89F7 mov edi,esi
> 0000000E 83C76D add edi,byte +0x6d
> 00000011 803F00 cmp byte [rdi],0x0
> 00000014 742B jz 0x41
> 00000016 B9800000C0 mov ecx,0xc0000080
> 0000001B 0F32 rdmsr
> 0000001D 0FBAE80B bts eax,byte 0xb
> 00000021 0F30 wrmsr
> 00000023 89F7 mov edi,esi
> 00000025 83C771 add edi,byte +0x71
> 00000028 8B07 mov eax,[rdi]
> 0000002A 0F22D8 mov cr3,rax
> 0000002D 0F20E0 mov rax,cr4
> 00000030 0FBAE805 bts eax,byte 0x5
> 00000034 0F22E0 mov cr4,rax
> 00000037 0F20C0 mov rax,cr0
> 0000003A 0FBAE81F bts eax,byte 0x1f
> 0000003E 0F22C0 mov cr0,rax
> 00000041 89F7 mov edi,esi
> 00000043 83C775 add edi,byte +0x75
> 00000046 833F01 cmp dword [rdi],byte +0x1
> 00000049 752E jnz 0x79
> 0000004B 89F7 mov edi,esi
> 0000004D 83C77D add edi,byte +0x7d
> 00000050 F0FF07 lock inc dword [rdi]
> 00000053 89F7 mov edi,esi
> 00000055 83C761 add edi,byte +0x61
> 00000058 BB01000000 mov ebx,0x1
> 0000005D F00FC11F lock xadd [rdi],ebx
> 00000061 4389F7 mov r15d,esi
> 00000064 83C745 add edi,byte +0x45
> 00000067 8B07 mov eax,[rdi]
> 00000069 89D9 mov ecx,ebx
> 0000006B 41F7E1 mul r9d
> 0000006E 89F7 mov edi,esi
> 00000070 83C741 add edi,byte +0x41
> 00000073 0307 add eax,[rdi]
> 00000075 89C4 mov esp,eax
> 00000077 EB3F jmp short 0xb8
> 00000079 B800000000 mov eax,0x0
> 0000007E 0FA2 cpuid
> 00000080 83F80B cmp eax,byte +0xb
> 00000083 7213 jc 0x98
> 00000085 B80B000000 mov eax,0xb
> 0000008A 31C9 xor ecx,ecx
> 0000008C 0FA2 cpuid
> 0000008E F7C3FFFF0000 test ebx,0xffff
> 00000094 7402 jz 0x98
> 00000096 EB0C jmp short 0xa4
> 00000098 B801000000 mov eax,0x1
> 0000009D 0FA2 cpuid
> 0000009F C1EB18 shr ebx,byte 0x18
> 000000A2 89DA mov edx,ebx
> 000000A4 31DB xor ebx,ebx
> 000000A6 8D4679 lea eax,[rsi+0x79]
> 000000A9 8B38 mov edi,[rax]
> 000000AB 3917 cmp [rdi],edx
> 000000AD 7406 jz 0xb5
> 000000AF 83C714 add edi,byte +0x14
> 000000B2 43EBF6 jmp short 0xab
> 000000B5 8B670C mov esp,[rdi+0xc]
> 000000B8 83EC04 sub esp,byte +0x4
> 000000BB 55 push rbp
> 000000BC 31ED xor ebp,ebp
> 000000BE 55 push rbp
> 000000BF 89E5 mov ebp,esp
> 000000C1 B8F0CAEB07 mov eax,0x7ebcaf0
> 000000C6 FFD0 call rax
> 000000C8 53 push rbx
> 000000C9 89F0 mov eax,esi
> 000000CB 0581000000 add eax,0x81
> 000000D0 FF30 push qword [rax]
> 000000D2 89F7 mov edi,esi
> 000000D4 83C749 add edi,byte +0x49
> 000000D7 8B07 mov eax,[rdi]
> 000000D9 FFD0 call rax
> 000000DB EBFE jmp short 0xdb
> 000000DD EBFE jmp short 0xdd
> ... only zeroes follow
>
>
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#121394): https://edk2.groups.io/g/devel/message/121394
Mute This Topic: https://groups.io/mt/113480235/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-
next prev parent reply other threads:[~2025-06-05 7:04 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-06-04 9:56 [edk2-devel] Avoid split lock detection warnings triggered by OVMF? Friedrich Weber
2025-06-05 7:04 ` Ard Biesheuvel via groups.io [this message]
2025-06-05 9:00 ` Friedrich Weber
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAMj1kXEjXe7be5MZiRyos8TCOttOwLzVYemyuFtdfedYDTNSkg@mail.gmail.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox