From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 3B846740032 for ; Thu, 5 Jun 2025 04:17:57 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=4sfl36N+KHnf33m6VNqsGaPgUI/mtYY77i8QjmLoCyE=; c=relaxed/simple; d=groups.io; h=Message-ID:Date:MIME-Version:User-Agent:From:Subject:To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Resent-Date:Resent-From:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20240830; t=1749097077; v=1; x=1749356275; b=NWb2DEXYzokVPwDXidZjhn01S7XtloUeH5nkuLeRKRrs4g8CwoErCkTjO33iYiYYpM/9vaUb qDzeSk/n6FmMF6K/g3jGyILZTxdD6ssh2BXzcc/cpk6+Whop1eU/u9pIyFiBQyALLQsv0xyglTa T9ngtrU5jvMoDnYkhuJ4bv9WMatA+oivvMjgKkJVQECG4A0tIusYHNoCQHHzxSvHqYStIakbI3z cQbF0ve/r+SwK3QfPKyqf6fwT8MGeIAzWqrSHm0RKMhXSPrcwpxQw9cF/jesJbl1+HB0fRe1paI 6QrHCBBMXzvH8q5DcyIb7seBmcwpFFFpELG0/03uEDZOQ== X-Received: by 127.0.0.2 with SMTP id 2KCIYY7687511xGCnZ5hdqTD; Wed, 04 Jun 2025 21:17:55 -0700 X-Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) by mx.groups.io with SMTP id smtpd.web11.12928.1749030981184867051 for ; Wed, 04 Jun 2025 02:56:22 -0700 X-Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C96C6427EF for ; Wed, 4 Jun 2025 11:56:18 +0200 (CEST) Message-ID: <5d302e42-970f-4468-8d0c-fb564250dea9@proxmox.com> Date: Wed, 4 Jun 2025 11:56:17 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: "Friedrich Weber" Subject: [edk2-devel] Avoid split lock detection warnings triggered by OVMF? To: devel@edk2.groups.io Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Resent-Date: Wed, 04 Jun 2025 21:17:54 -0700 Resent-From: f.weber@proxmox.com Reply-To: devel@edk2.groups.io,f.weber@proxmox.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: 14W1BL0fAT7pb4A3LOMwuNqvx7686176AA= Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b=NWb2DEXY; dmarc=none; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io Hi, Some of our Proxmox VE users report that on CPUs with the split_lock_detect flag (such as recent Intel CPUs), QEMU+KVM VMs using our build of OVMF trig= ger split lock detection warnings on the host on VM start (only for VMs with mo= re than one core, though), e.g.: Jun 03 10:16:25 host kernel: x86/split lock detection: #AC: CPU 1/KVM/26750= took a split_lock trap at address: 0x7ff4b050 But I can also reproduce this on upstream QEMU and current OVMF master, see below. In my understanding, split locks can be caused by misaligned atomic memory accesses, and by default kernels >=3D 5.19 artifically slow down threads th= at cause a split-lock operation for ~10ms [1]. One complication is that the warning above is only logged once per thread, so if a thread causes more su= ch operations and is again slowed down, this is not logged again. We have put together a bpftrace command that logs subsequent slowdowns, see [2]. Using bpftrace, we can see that each thread can cause 0-2 such split-lock operation on boot, so each thread is potentially slowed down for 10-20ms in total. This slowdown is negligible, but the warnings have other downsides: First, they look a bit scary, and second, in case the actual VM workload ca= uses split-lock operations later on, these early and somewhat spurious warnings "mask" later actual relevant warnings. So I was wondering -- do you think O= VMF could be patched to avoid the split-lock operations and thus avoid the warn= ing? Some more details -- I can reproduce the issue on current EDK II master (8c04bcc7ed0efbb2e3fde23f787ff0249e62f874) with the following steps: - have a CPU with split_lock_detect flag, in my case Intel i7-13700K, split_lock_mitigate sysctl at its default 1 - build OVMF (the PcdFirmwareVendor is not necessary, just to avoid confusi= ng myself) build -a IA32 -a X64 -t GCC -p OvmfPkg/OvmfPkgIa32X64.dsc --pcd PcdFirmware= Vendor=3DL"built at $(date)\\0" - copy over the resulting Build/Ovmf3264/DEBUG_GCC/FV/OVMF_CODE.fd - run bpftrace in parallel: bpftrace -e 'kfunc:vmlinux:split_lock_warn /0<*(uint32*)kaddr("sysctl_sld_m= itigate")/{time("%H:%M:%S: "); printf("comm=3D%s tid=3D%d ip=3D0x%x\n", com= m, tid, args->ip);}' - run upstream QEMU 9.2.4 with this firmware: /path/to/qemu-9.2.4/qemu-system-x86_64 -drive if=3Dpflash,format=3Draw,read= only=3Don,file=3DOVMF_CODE.fd -smp 4,sockets=3D1,cores=3D4 -accel kvm - find in the journal: Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86= /62190 took a split_lock trap at address: 0x7e8a050 Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86= /62188 took a split_lock trap at address: 0x7e8a050 Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86= /62189 took a split_lock trap at address: 0x7e8a050 - find in the bpftrace: 10:37:34: comm=3Dqemu-system-x86 tid=3D62190 ip=3D0x7e8a050 10:37:34: comm=3Dqemu-system-x86 tid=3D62188 ip=3D0x7e8a050 10:37:34: comm=3Dqemu-system-x86 tid=3D62189 ip=3D0x7e8a050 The ip seems to be the same on each QEMU run. So I dumped the surrounding instructions using -monitor stdio and: memsave 0x7e8a000 0x1000 memdump.raw and ndisasm -b64 memsave.raw and 0x50 with some context is (full output in [3]): 0000004B 89F7 mov edi,esi 0000004D 83C77D add edi,byte +0x7d 00000050 F0FF07 lock inc dword [rdi] where the last one may indeed be a misaligned atomic memory access? The disassembly looks like it could match this portion of LongModeStart in UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm: LongModeStart: [...] ; Increment the number of APs executing here as early as possible ; This is decremented in C code when AP is finished executing mov edi, esi add edi, MP_CPU_EXCHANGE_INFO_FIELD (NumApsExecuting) lock inc dword [edi] ... so NumApsExecuting in MP_CPU_EXCHANGE_INFO *may* be the culprit, but th= is is really speculation on my part :) I know nothing about EDK II/OVMF internals, so even when assuming the above speculation makes sense, I can't judge whether the above MP_CPU_EXCHANGE_IN= FO could perhaps be rearranged/padded such that it doesn't trigger the split l= ock detection. What do you think? Happy to provide more information if needed, just let me know. Thanks and best wishes, Friedrich [1] https://lwn.net/Articles/911219/ [2] https://pve.proxmox.com/wiki/Split_lock_detection [3] 00000000 8EDA mov ds,edx 00000002 8EC2 mov es,edx 00000004 8EE2 mov fs,edx 00000006 8EEA mov gs,edx 00000008 8ED2 mov ss,edx 0000000A 89DE mov esi,ebx 0000000C 89F7 mov edi,esi 0000000E 83C76D add edi,byte +0x6d 00000011 803F00 cmp byte [rdi],0x0 00000014 742B jz 0x41 00000016 B9800000C0 mov ecx,0xc0000080 0000001B 0F32 rdmsr 0000001D 0FBAE80B bts eax,byte 0xb 00000021 0F30 wrmsr 00000023 89F7 mov edi,esi 00000025 83C771 add edi,byte +0x71 00000028 8B07 mov eax,[rdi] 0000002A 0F22D8 mov cr3,rax 0000002D 0F20E0 mov rax,cr4 00000030 0FBAE805 bts eax,byte 0x5 00000034 0F22E0 mov cr4,rax 00000037 0F20C0 mov rax,cr0 0000003A 0FBAE81F bts eax,byte 0x1f 0000003E 0F22C0 mov cr0,rax 00000041 89F7 mov edi,esi 00000043 83C775 add edi,byte +0x75 00000046 833F01 cmp dword [rdi],byte +0x1 00000049 752E jnz 0x79 0000004B 89F7 mov edi,esi 0000004D 83C77D add edi,byte +0x7d 00000050 F0FF07 lock inc dword [rdi] 00000053 89F7 mov edi,esi 00000055 83C761 add edi,byte +0x61 00000058 BB01000000 mov ebx,0x1 0000005D F00FC11F lock xadd [rdi],ebx 00000061 4389F7 mov r15d,esi 00000064 83C745 add edi,byte +0x45 00000067 8B07 mov eax,[rdi] 00000069 89D9 mov ecx,ebx 0000006B 41F7E1 mul r9d 0000006E 89F7 mov edi,esi 00000070 83C741 add edi,byte +0x41 00000073 0307 add eax,[rdi] 00000075 89C4 mov esp,eax 00000077 EB3F jmp short 0xb8 00000079 B800000000 mov eax,0x0 0000007E 0FA2 cpuid 00000080 83F80B cmp eax,byte +0xb 00000083 7213 jc 0x98 00000085 B80B000000 mov eax,0xb 0000008A 31C9 xor ecx,ecx 0000008C 0FA2 cpuid 0000008E F7C3FFFF0000 test ebx,0xffff 00000094 7402 jz 0x98 00000096 EB0C jmp short 0xa4 00000098 B801000000 mov eax,0x1 0000009D 0FA2 cpuid 0000009F C1EB18 shr ebx,byte 0x18 000000A2 89DA mov edx,ebx 000000A4 31DB xor ebx,ebx 000000A6 8D4679 lea eax,[rsi+0x79] 000000A9 8B38 mov edi,[rax] 000000AB 3917 cmp [rdi],edx 000000AD 7406 jz 0xb5 000000AF 83C714 add edi,byte +0x14 000000B2 43EBF6 jmp short 0xab 000000B5 8B670C mov esp,[rdi+0xc] 000000B8 83EC04 sub esp,byte +0x4 000000BB 55 push rbp 000000BC 31ED xor ebp,ebp 000000BE 55 push rbp 000000BF 89E5 mov ebp,esp 000000C1 B8F0CAEB07 mov eax,0x7ebcaf0 000000C6 FFD0 call rax 000000C8 53 push rbx 000000C9 89F0 mov eax,esi 000000CB 0581000000 add eax,0x81 000000D0 FF30 push qword [rax] 000000D2 89F7 mov edi,esi 000000D4 83C749 add edi,byte +0x49 000000D7 8B07 mov eax,[rdi] 000000D9 FFD0 call rax 000000DB EBFE jmp short 0xdb 000000DD EBFE jmp short 0xdd ... only zeroes follow -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#121393): https://edk2.groups.io/g/devel/message/121393 Mute This Topic: https://groups.io/mt/113480235/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-