From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 64F679417D8 for ; Thu, 5 Jun 2025 07:04:53 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=RDI2Hqi1w9r73766aldJ625TWkKMUl5Lu1BUBOWfNuM=; c=relaxed/simple; d=groups.io; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Resent-Date:Resent-From:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type; s=20240830; t=1749107093; v=1; x=1749366291; b=lIV3zVlGPFgzkA/mn5eHOouGVWMOwIIOzn8u9ea2rCsm9Z3iN4w2MyzJifbQXEKSP1pY+fUk nvC03ESnD6v12qOHQQeRRf3ZE2N97JeNb6otTYqf2dQfq51y4MpFrZ4SD1u2/1XD0GjuEs5inCx e3bmYdkaNUSziWP9CZdwO0DYIxnpx83JoWmwAHSE+PruuGT92YEAN+vk62wfVE0nxXBjEQAX24e bpZbjqFQM1NjG0baKWxIUvjh9YSgHnGQl1uUGfgT93ppoy43sZzC25ZKEujNSaVkC0XSuklJ1c1 xQy+15HWkIpBpD5aagp/9ioNsb+x5Umj6NqEiziwoCYXw== X-Received: by 127.0.0.2 with SMTP id 4S25YY7687511xm2spAtke5R; Thu, 05 Jun 2025 00:04:51 -0700 X-Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by mx.groups.io with SMTP id smtpd.web11.1477.1749107090663050222 for ; Thu, 05 Jun 2025 00:04:50 -0700 X-Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 0216B440B5 for ; Thu, 5 Jun 2025 07:04:50 +0000 (UTC) X-Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6B29C4CEEB for ; Thu, 5 Jun 2025 07:04:49 +0000 (UTC) X-Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-54afb5fcebaso710015e87.3 for ; Thu, 05 Jun 2025 00:04:49 -0700 (PDT) X-Gm-Message-State: p3GBHI8sKdi5eBxFcRa1XpKrx7686176AA= X-Google-Smtp-Source: AGHT+IG0w3byuKc5jO7Y0L4wS9DSSMox/ebgzr6QnFithsu34LIcgFeWlKuMjzCmgyNRHyM14Lu0LbTBDW4x+FWuN8Q= X-Received: by 2002:a05:6512:b9a:b0:553:2fb1:cfe9 with SMTP id 2adb3069b0e04-55356ae9f4fmr1704665e87.21.1749107088072; Thu, 05 Jun 2025 00:04:48 -0700 (PDT) MIME-Version: 1.0 References: <5d302e42-970f-4468-8d0c-fb564250dea9@proxmox.com> In-Reply-To: <5d302e42-970f-4468-8d0c-fb564250dea9@proxmox.com> From: "Ard Biesheuvel via groups.io" Date: Thu, 5 Jun 2025 09:04:36 +0200 X-Gmail-Original-Message-ID: X-Gm-Features: AX0GCFsNBx8Zu0x9IlpNUehnbl3YilkW0WRj6ZwKN5iNXwZVbxEe_fgkHH2Nm7Y Message-ID: Subject: Re: [edk2-devel] Avoid split lock detection warnings triggered by OVMF? To: devel@edk2.groups.io, f.weber@proxmox.com Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Resent-Date: Thu, 05 Jun 2025 00:04:50 -0700 Resent-From: ardb@kernel.org Reply-To: devel@edk2.groups.io,ardb@kernel.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: Content-Type: text/plain; charset="UTF-8" X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b=lIV3zVlG; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io Thanks for the elaborate report. Does the following help? diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.h b/UefiCpuPkg/Library/MpInitLib/MpLib.h index a63bb81bef88..a9b3fa77e14d 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpLib.h +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.h @@ -213,8 +213,6 @@ typedef struct { UINTN StackStart; UINTN StackSize; UINTN CFunction; - IA32_DESCRIPTOR GdtrProfile; - IA32_DESCRIPTOR IdtrProfile; UINTN BufferStart; UINTN ModeOffset; UINTN ApIndex; @@ -227,6 +225,8 @@ typedef struct { UINTN NumApsExecuting; CPU_MP_DATA *CpuMpData; UINTN InitializeFloatingPointUnitsAddress; + IA32_DESCRIPTOR GdtrProfile; + IA32_DESCRIPTOR IdtrProfile; UINT32 ModeTransitionMemory; UINT16 ModeTransitionSegment; UINT32 ModeHighMemory; diff --git a/UefiCpuPkg/Library/MpInitLib/MpEqu.inc b/UefiCpuPkg/Library/MpInitLib/MpEqu.inc index d8ba9ea1246c..d67c4c923c82 100644 --- a/UefiCpuPkg/Library/MpInitLib/MpEqu.inc +++ b/UefiCpuPkg/Library/MpInitLib/MpEqu.inc @@ -74,8 +74,6 @@ struc MP_CPU_EXCHANGE_INFO .StackStart: CTYPE_UINTN 1 .StackSize: CTYPE_UINTN 1 .CFunction: CTYPE_UINTN 1 - .GdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size - .IdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size .BufferStart: CTYPE_UINTN 1 .ModeOffset: CTYPE_UINTN 1 .ApIndex: CTYPE_UINTN 1 @@ -88,6 +86,8 @@ struc MP_CPU_EXCHANGE_INFO .NumApsExecuting: CTYPE_UINTN 1 .CpuMpData: CTYPE_UINTN 1 .InitializeFloatingPointUnits: CTYPE_UINTN 1 + .GdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size + .IdtrProfile: CTYPE_UINT8 IA32_DESCRIPTOR_size .ModeTransitionMemory: CTYPE_UINT32 1 .ModeTransitionSegment: CTYPE_UINT16 1 .ModeHighMemory: CTYPE_UINT32 1 On Thu, 5 Jun 2025 at 06:18, Friedrich Weber wrote: > > Hi, > > Some of our Proxmox VE users report that on CPUs with the split_lock_detect > flag (such as recent Intel CPUs), QEMU+KVM VMs using our build of OVMF trigger > split lock detection warnings on the host on VM start (only for VMs with more > than one core, though), e.g.: > > Jun 03 10:16:25 host kernel: x86/split lock detection: #AC: CPU 1/KVM/26750 took a split_lock trap at address: 0x7ff4b050 > > But I can also reproduce this on upstream QEMU and current OVMF master, see > below. > > In my understanding, split locks can be caused by misaligned atomic memory > accesses, and by default kernels >= 5.19 artifically slow down threads that > cause a split-lock operation for ~10ms [1]. One complication is that the > warning above is only logged once per thread, so if a thread causes more such > operations and is again slowed down, this is not logged again. We have put > together a bpftrace command that logs subsequent slowdowns, see [2]. > > Using bpftrace, we can see that each thread can cause 0-2 such split-lock > operation on boot, so each thread is potentially slowed down for 10-20ms in > total. This slowdown is negligible, but the warnings have other downsides: > First, they look a bit scary, and second, in case the actual VM workload causes > split-lock operations later on, these early and somewhat spurious warnings > "mask" later actual relevant warnings. So I was wondering -- do you think OVMF > could be patched to avoid the split-lock operations and thus avoid the warning? > > Some more details -- I can reproduce the issue on current EDK II master > (8c04bcc7ed0efbb2e3fde23f787ff0249e62f874) with the following steps: > > - have a CPU with split_lock_detect flag, in my case Intel i7-13700K, > split_lock_mitigate sysctl at its default 1 > > - build OVMF (the PcdFirmwareVendor is not necessary, just to avoid confusing > myself) > > build -a IA32 -a X64 -t GCC -p OvmfPkg/OvmfPkgIa32X64.dsc --pcd PcdFirmwareVendor=L"built at $(date)\\0" > > - copy over the resulting Build/Ovmf3264/DEBUG_GCC/FV/OVMF_CODE.fd > - run bpftrace in parallel: > > bpftrace -e 'kfunc:vmlinux:split_lock_warn /0<*(uint32*)kaddr("sysctl_sld_mitigate")/{time("%H:%M:%S: "); printf("comm=%s tid=%d ip=0x%x\n", comm, tid, args->ip);}' > > - run upstream QEMU 9.2.4 with this firmware: > > /path/to/qemu-9.2.4/qemu-system-x86_64 -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd -smp 4,sockets=1,cores=4 -accel kvm > > - find in the journal: > > Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62190 took a split_lock trap at address: 0x7e8a050 > Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62188 took a split_lock trap at address: 0x7e8a050 > Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62189 took a split_lock trap at address: 0x7e8a050 > > - find in the bpftrace: > > 10:37:34: comm=qemu-system-x86 tid=62190 ip=0x7e8a050 > 10:37:34: comm=qemu-system-x86 tid=62188 ip=0x7e8a050 > 10:37:34: comm=qemu-system-x86 tid=62189 ip=0x7e8a050 > > The ip seems to be the same on each QEMU run. So I dumped the surrounding > instructions using -monitor stdio and: > > memsave 0x7e8a000 0x1000 memdump.raw > > and > > ndisasm -b64 memsave.raw > > and 0x50 with some context is (full output in [3]): > > 0000004B 89F7 mov edi,esi > 0000004D 83C77D add edi,byte +0x7d > 00000050 F0FF07 lock inc dword [rdi] > > where the last one may indeed be a misaligned atomic memory access? > > The disassembly looks like it could match this portion of LongModeStart in > UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm: > > LongModeStart: > [...] > > ; Increment the number of APs executing here as early as possible > ; This is decremented in C code when AP is finished executing > mov edi, esi > add edi, MP_CPU_EXCHANGE_INFO_FIELD (NumApsExecuting) > lock inc dword [edi] > > ... so NumApsExecuting in MP_CPU_EXCHANGE_INFO *may* be the culprit, but this > is really speculation on my part :) > > I know nothing about EDK II/OVMF internals, so even when assuming the above > speculation makes sense, I can't judge whether the above MP_CPU_EXCHANGE_INFO > could perhaps be rearranged/padded such that it doesn't trigger the split lock > detection. What do you think? > > Happy to provide more information if needed, just let me know. > > Thanks and best wishes, > > Friedrich > > [1] https://lwn.net/Articles/911219/ > [2] https://pve.proxmox.com/wiki/Split_lock_detection > [3] > > 00000000 8EDA mov ds,edx > 00000002 8EC2 mov es,edx > 00000004 8EE2 mov fs,edx > 00000006 8EEA mov gs,edx > 00000008 8ED2 mov ss,edx > 0000000A 89DE mov esi,ebx > 0000000C 89F7 mov edi,esi > 0000000E 83C76D add edi,byte +0x6d > 00000011 803F00 cmp byte [rdi],0x0 > 00000014 742B jz 0x41 > 00000016 B9800000C0 mov ecx,0xc0000080 > 0000001B 0F32 rdmsr > 0000001D 0FBAE80B bts eax,byte 0xb > 00000021 0F30 wrmsr > 00000023 89F7 mov edi,esi > 00000025 83C771 add edi,byte +0x71 > 00000028 8B07 mov eax,[rdi] > 0000002A 0F22D8 mov cr3,rax > 0000002D 0F20E0 mov rax,cr4 > 00000030 0FBAE805 bts eax,byte 0x5 > 00000034 0F22E0 mov cr4,rax > 00000037 0F20C0 mov rax,cr0 > 0000003A 0FBAE81F bts eax,byte 0x1f > 0000003E 0F22C0 mov cr0,rax > 00000041 89F7 mov edi,esi > 00000043 83C775 add edi,byte +0x75 > 00000046 833F01 cmp dword [rdi],byte +0x1 > 00000049 752E jnz 0x79 > 0000004B 89F7 mov edi,esi > 0000004D 83C77D add edi,byte +0x7d > 00000050 F0FF07 lock inc dword [rdi] > 00000053 89F7 mov edi,esi > 00000055 83C761 add edi,byte +0x61 > 00000058 BB01000000 mov ebx,0x1 > 0000005D F00FC11F lock xadd [rdi],ebx > 00000061 4389F7 mov r15d,esi > 00000064 83C745 add edi,byte +0x45 > 00000067 8B07 mov eax,[rdi] > 00000069 89D9 mov ecx,ebx > 0000006B 41F7E1 mul r9d > 0000006E 89F7 mov edi,esi > 00000070 83C741 add edi,byte +0x41 > 00000073 0307 add eax,[rdi] > 00000075 89C4 mov esp,eax > 00000077 EB3F jmp short 0xb8 > 00000079 B800000000 mov eax,0x0 > 0000007E 0FA2 cpuid > 00000080 83F80B cmp eax,byte +0xb > 00000083 7213 jc 0x98 > 00000085 B80B000000 mov eax,0xb > 0000008A 31C9 xor ecx,ecx > 0000008C 0FA2 cpuid > 0000008E F7C3FFFF0000 test ebx,0xffff > 00000094 7402 jz 0x98 > 00000096 EB0C jmp short 0xa4 > 00000098 B801000000 mov eax,0x1 > 0000009D 0FA2 cpuid > 0000009F C1EB18 shr ebx,byte 0x18 > 000000A2 89DA mov edx,ebx > 000000A4 31DB xor ebx,ebx > 000000A6 8D4679 lea eax,[rsi+0x79] > 000000A9 8B38 mov edi,[rax] > 000000AB 3917 cmp [rdi],edx > 000000AD 7406 jz 0xb5 > 000000AF 83C714 add edi,byte +0x14 > 000000B2 43EBF6 jmp short 0xab > 000000B5 8B670C mov esp,[rdi+0xc] > 000000B8 83EC04 sub esp,byte +0x4 > 000000BB 55 push rbp > 000000BC 31ED xor ebp,ebp > 000000BE 55 push rbp > 000000BF 89E5 mov ebp,esp > 000000C1 B8F0CAEB07 mov eax,0x7ebcaf0 > 000000C6 FFD0 call rax > 000000C8 53 push rbx > 000000C9 89F0 mov eax,esi > 000000CB 0581000000 add eax,0x81 > 000000D0 FF30 push qword [rax] > 000000D2 89F7 mov edi,esi > 000000D4 83C749 add edi,byte +0x49 > 000000D7 8B07 mov eax,[rdi] > 000000D9 FFD0 call rax > 000000DB EBFE jmp short 0xdb > 000000DD EBFE jmp short 0xdd > ... only zeroes follow > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#121394): https://edk2.groups.io/g/devel/message/121394 Mute This Topic: https://groups.io/mt/113480235/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-