Re: OvmfPkg: VM crashed trying to write to RO memory from CommonInterruptEntry

[Evgeny Yakovlev <insoreiges@gmail.com>]

Looks like we're actually based on OVMF tree for RHEL7.2:
ovmf/rhel-20150414-2.gitc9e5618.el7, ovmf/rhel-srpm-7.2

So maybe this affects those deployments as well

2016-11-22 19:58 GMT+03:00 Laszlo Ersek <lersek@redhat.com>:

> On 11/22/16 14:58, Evgeny Yakovlev wrote:
> > Wow, that is more than i expected :)
> >
> >> I wonder if you started to see this issue very recently.
> > Very recently, however we use a pretty old OVMF build, circa 2015
>
> Ugh. Please update OVMF first... A whole lot of things has changed in
> edk2 in this year.
>
> >
> >>  OVMF debug log
> > Sorry, we hadn't had it enabled when VM crashed and these crashes are
> very
> > rare. We will try to capture it when it happens again
> >
> >> - your host CPU model,
> > cpu family      : 6
> > model           : 42
> > model name      : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
> > stepping        : 7
> >
> >> - the host kernel (KVM) version,
> > Our kernel is roughly based on RHEL7.2 (kernel version 3.10.0-327.36.1).
> We
> > also have some upstream KVM patches backported.
> >
> >> - the guest CPU model,
> > -cpu
> > SandyBridge,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+
> monitor,+ds_cpl,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+
> osxsave,-arat,-xsaveopt,-xgetbv1,-vmx,-xsavec,hv_time,
> hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vpindex,
> hv_runtime,hv_synic,hv_stimer,hv_reset,hv_crash
> >
> >> - the guest CPU topology.
> > 8 sockets, 1 core per socket, 1 thread per core
> >
> > Hope that helps!
>
> The fact that you are using 8 VCPUs is definitely relevant. However, I
> don't think it would make sense to try to analyze any errors with an
> OVMF / edk2 tree this old. Please try to reproduce the issue with a
> fresh build from master.
>
> Thanks!
> Laszlo
>
> > 2016-11-22 16:41 GMT+03:00 Laszlo Ersek <lersek@redhat.com>:
> >
> >> Hello Evgeny,
> >>
> >> On 11/22/16 13:57, Evgeny Yakovlev wrote:
> >>> We are running windows UEFI-based VMs on QEMU/KVM with OvmfPkg.
> >>>
> >>> Very rarely we are experiencing a crash when VM tries to write to RO
> >> memory
> >>> very early during UEFI boot process.
> >>>
> >>> Crash happens when VM tries to execute this code in interrupt handler:
> >>> https://github.com/tianocore/edk2/blob/master/UefiCpuPkg/Library/
> >> CpuExceptionHandlerLib/X64/ExceptionHandlerAsm.asm#L244-L246
> >>>
> >>>
> >>> fxsave [rdi], where RDI = 0xffe60
> >>>
> >>> Which is bad - it points to ISA BIOS F-segment area.
> >>>
> >>> This memory was mapped by qemu for read only access, which is reflected
> >> in
> >>> KVM EPT:
> >>> 00000000000e0000-00000000000fffff (prio 1, R-): isa-bios
> >>>
> >>> This is a very early IRQ0 interrupt, presumably during early
> >> initialization
> >>> phase (Sec or Pei).
> >>>
> >>> Looks like CommonInterruptHandler does not switch to a separate stack
> and
> >>> works on interrupted context's stack, which was fairly close to 1MB
> >>> boundary when IRQ0 fired (RSP around 1002c0). When CommonInterruptEntry
> >>> reached highlighted code it subtracted 512 bytes from current RSP which
> >>> dropped to 0xffe60, below 1MB and into QEMU RO region.
> >>>
> >>> We were figuring out how to best fix this. Possible solutions are to
> >> switch
> >>> to a separate stack in CommonInterruptEntry, relocate early OvmfPkg
> stack
> >>> to somewhere farther away from 1MB, to run with interrupts disabled
> until
> >>> we reach a later phase or maybe something else.
> >>>
> >>> Any comments would be very appreciated!
> >>
> >> I wonder if you started to see this issue very recently.
> >>
> >> I suspect (hope!) that the symptoms you are experiencing are a
> >> consequence of a bug in UefiCpuPkg that I've debugged and fixed just
> >> today. (I hope to post the patches today.)
> >>
> >> While testing those patches on your end will of course tell us if your
> >> issue has the same root cause, you could gather a few more symptoms even
> >> before I get around posting the patches. The bug that I'm working on has
> >> extremely varied crash symptoms (basically the APs wander off into the
> >> weeds), and some of those symptoms have involved CpuExceptionHandlerLib.
> >> The point is, by the time we get into CpuExceptionHandlerLib, all is
> >> lost -- it is executing on an AP whose state is corrupt anyway. The
> >> fxsave symptom is a red herring, most likely.
> >>
> >> CpuExceptionHandlerLib works fine otherwise, especially when invoked
> >> from the BSP -- we've used the output dumped by CpuExceptionHandlerLib
> >> to the serial port several times to track down issues.
> >>
> >> So, my request is that you please capture the OVMF debug log (please see
> >> the "OvmfPkg/README" file for how). I'm curious if it crashes where and
> >> how I suspect it crashes.
> >>
> >> Also, it would help if you provided
> >> - your host CPU model,
> >> - the host kernel (KVM) version,
> >> - the guest CPU model,
> >> - the guest CPU topology.
> >>
> >> Thanks!
> >> Laszlo
> >>
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> >
>
>