[edk2-devel] Avoid split lock detection warnings triggered by OVMF?

["Friedrich Weber" <f.weber@proxmox.com>]

Hi,

Some of our Proxmox VE users report that on CPUs with the split_lock_detect
flag (such as recent Intel CPUs), QEMU+KVM VMs using our build of OVMF trigger
split lock detection warnings on the host on VM start (only for VMs with more
than one core, though), e.g.:

Jun 03 10:16:25 host kernel: x86/split lock detection: #AC: CPU 1/KVM/26750 took a split_lock trap at address: 0x7ff4b050

But I can also reproduce this on upstream QEMU and current OVMF master, see
below.

In my understanding, split locks can be caused by misaligned atomic memory
accesses, and by default kernels >= 5.19 artifically slow down threads that
cause a split-lock operation for ~10ms [1]. One complication is that the
warning above is only logged once per thread, so if a thread causes more such
operations and is again slowed down, this is not logged again. We have put
together a bpftrace command that logs subsequent slowdowns, see [2].

Using bpftrace, we can see that each thread can cause 0-2 such split-lock
operation on boot, so each thread is potentially slowed down for 10-20ms in
total. This slowdown is negligible, but the warnings have other downsides:
First, they look a bit scary, and second, in case the actual VM workload causes
split-lock operations later on, these early and somewhat spurious warnings
"mask" later actual relevant warnings. So I was wondering -- do you think OVMF
could be patched to avoid the split-lock operations and thus avoid the warning?

Some more details -- I can reproduce the issue on current EDK II master
(8c04bcc7ed0efbb2e3fde23f787ff0249e62f874) with the following steps:

- have a CPU with split_lock_detect flag, in my case Intel i7-13700K,
  split_lock_mitigate sysctl at its default 1

- build OVMF (the PcdFirmwareVendor is not necessary, just to avoid confusing
  myself)

build -a IA32 -a X64 -t GCC -p OvmfPkg/OvmfPkgIa32X64.dsc --pcd PcdFirmwareVendor=L"built at $(date)\\0"

- copy over the resulting Build/Ovmf3264/DEBUG_GCC/FV/OVMF_CODE.fd
- run bpftrace in parallel:

bpftrace -e 'kfunc:vmlinux:split_lock_warn /0<*(uint32*)kaddr("sysctl_sld_mitigate")/{time("%H:%M:%S: "); printf("comm=%s tid=%d ip=0x%x\n", comm, tid, args->ip);}'

- run upstream QEMU 9.2.4 with this firmware:

/path/to/qemu-9.2.4/qemu-system-x86_64 -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd -smp 4,sockets=1,cores=4 -accel kvm

- find in the journal:

Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62190 took a split_lock trap at address: 0x7e8a050
Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62188 took a split_lock trap at address: 0x7e8a050
Jun 04 10:37:34 host kernel: x86/split lock detection: #AC: qemu-system-x86/62189 took a split_lock trap at address: 0x7e8a050

- find in the bpftrace:

10:37:34: comm=qemu-system-x86 tid=62190 ip=0x7e8a050
10:37:34: comm=qemu-system-x86 tid=62188 ip=0x7e8a050
10:37:34: comm=qemu-system-x86 tid=62189 ip=0x7e8a050

The ip seems to be the same on each QEMU run. So I dumped the surrounding
instructions using -monitor stdio and:

memsave 0x7e8a000 0x1000 memdump.raw

and

ndisasm -b64 memsave.raw

and 0x50 with some context is (full output in [3]):

0000004B  89F7              mov edi,esi
0000004D  83C77D            add edi,byte +0x7d
00000050  F0FF07            lock inc dword [rdi]

where the last one may indeed be a misaligned atomic memory access?

The disassembly looks like it could match this portion of LongModeStart in
UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm:

LongModeStart:
    [...]

    ; Increment the number of APs executing here as early as possible
    ; This is decremented in C code when AP is finished executing
    mov        edi, esi
    add        edi, MP_CPU_EXCHANGE_INFO_FIELD (NumApsExecuting)
    lock inc   dword [edi]

... so NumApsExecuting in MP_CPU_EXCHANGE_INFO *may* be the culprit, but this
is really speculation on my part :)

I know nothing about EDK II/OVMF internals, so even when assuming the above
speculation makes sense, I can't judge whether the above MP_CPU_EXCHANGE_INFO
could perhaps be rearranged/padded such that it doesn't trigger the split lock
detection. What do you think?

Happy to provide more information if needed, just let me know.

Thanks and best wishes,

Friedrich

[1] https://lwn.net/Articles/911219/
[2] https://pve.proxmox.com/wiki/Split_lock_detection
[3]

00000000  8EDA              mov ds,edx
00000002  8EC2              mov es,edx
00000004  8EE2              mov fs,edx
00000006  8EEA              mov gs,edx
00000008  8ED2              mov ss,edx
0000000A  89DE              mov esi,ebx
0000000C  89F7              mov edi,esi
0000000E  83C76D            add edi,byte +0x6d
00000011  803F00            cmp byte [rdi],0x0
00000014  742B              jz 0x41
00000016  B9800000C0        mov ecx,0xc0000080
0000001B  0F32              rdmsr
0000001D  0FBAE80B          bts eax,byte 0xb
00000021  0F30              wrmsr
00000023  89F7              mov edi,esi
00000025  83C771            add edi,byte +0x71
00000028  8B07              mov eax,[rdi]
0000002A  0F22D8            mov cr3,rax
0000002D  0F20E0            mov rax,cr4
00000030  0FBAE805          bts eax,byte 0x5
00000034  0F22E0            mov cr4,rax
00000037  0F20C0            mov rax,cr0
0000003A  0FBAE81F          bts eax,byte 0x1f
0000003E  0F22C0            mov cr0,rax
00000041  89F7              mov edi,esi
00000043  83C775            add edi,byte +0x75
00000046  833F01            cmp dword [rdi],byte +0x1
00000049  752E              jnz 0x79
0000004B  89F7              mov edi,esi
0000004D  83C77D            add edi,byte +0x7d
00000050  F0FF07            lock inc dword [rdi]
00000053  89F7              mov edi,esi
00000055  83C761            add edi,byte +0x61
00000058  BB01000000        mov ebx,0x1
0000005D  F00FC11F          lock xadd [rdi],ebx
00000061  4389F7            mov r15d,esi
00000064  83C745            add edi,byte +0x45
00000067  8B07              mov eax,[rdi]
00000069  89D9              mov ecx,ebx
0000006B  41F7E1            mul r9d
0000006E  89F7              mov edi,esi
00000070  83C741            add edi,byte +0x41
00000073  0307              add eax,[rdi]
00000075  89C4              mov esp,eax
00000077  EB3F              jmp short 0xb8
00000079  B800000000        mov eax,0x0
0000007E  0FA2              cpuid
00000080  83F80B            cmp eax,byte +0xb
00000083  7213              jc 0x98
00000085  B80B000000        mov eax,0xb
0000008A  31C9              xor ecx,ecx
0000008C  0FA2              cpuid
0000008E  F7C3FFFF0000      test ebx,0xffff
00000094  7402              jz 0x98
00000096  EB0C              jmp short 0xa4
00000098  B801000000        mov eax,0x1
0000009D  0FA2              cpuid
0000009F  C1EB18            shr ebx,byte 0x18
000000A2  89DA              mov edx,ebx
000000A4  31DB              xor ebx,ebx
000000A6  8D4679            lea eax,[rsi+0x79]
000000A9  8B38              mov edi,[rax]
000000AB  3917              cmp [rdi],edx
000000AD  7406              jz 0xb5
000000AF  83C714            add edi,byte +0x14
000000B2  43EBF6            jmp short 0xab
000000B5  8B670C            mov esp,[rdi+0xc]
000000B8  83EC04            sub esp,byte +0x4
000000BB  55                push rbp
000000BC  31ED              xor ebp,ebp
000000BE  55                push rbp
000000BF  89E5              mov ebp,esp
000000C1  B8F0CAEB07        mov eax,0x7ebcaf0
000000C6  FFD0              call rax
000000C8  53                push rbx
000000C9  89F0              mov eax,esi
000000CB  0581000000        add eax,0x81
000000D0  FF30              push qword [rax]
000000D2  89F7              mov edi,esi
000000D4  83C749            add edi,byte +0x49
000000D7  8B07              mov eax,[rdi]
000000D9  FFD0              call rax
000000DB  EBFE              jmp short 0xdb
000000DD  EBFE              jmp short 0xdd
... only zeroes follow



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#121393): https://edk2.groups.io/g/devel/message/121393
Mute This Topic: https://groups.io/mt/113480235/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-