Re: [PATCH V2 0/6] Enable SMM page level protection.

[Paolo Bonzini <pbonzini@redhat.com>]

> Another question I have -- and I feel I should really know it, but I
> don't... -- is *why* the APs are executing code from the page at
> 0x9f000.

This I can answer. :)

The APs have done their INIT-SIPI-SIPI, and then went into the CLI;HLT;JMP
loop.  When the AP exits SMM, it is in the JMP instruction.

As suggested by Jiewen, edk2 could jump to a 32-bit loop that is _not_
in the 0-640K area (perhaps it could be in what your doc calls the
"permanent PEI memory for the S3 resume path"?).  After thinking a
bit more about it, it seems simplest to me if CpuS3.c just uses
SwitchStack or AsmDisablePaging64 at the end of MPRendezvousProcedure,
to jump to a small stub like

    POP EAX   ; pop return address
    POP EAX   ; pop Context1 which is &mNumberToFinish
    DEC [EAX]
 1: CLI
    HLT
    JMP 1

> I could be utterly and inexcusably wrong, but I think that the
> RIP=0x9f0fd symptom is a red herring.

I wouldn't call it a red herring.  After all, CR3 points to SMM
exactly because the CR3 that was set up for the 0x9f000 stub is
CpuS3.c's SMRAM page table root.

What _is_ a red herring is KVM's trace showing a RSM instruction
at RIP=0x9f0fd.  That is clearly bogus, RSM was rather the last
instruction executed _before_ getting to that RIP.

> >       vcpu#0  vcpu#1  vcpu#2  vcpu#3
> >       ------  ------  ------  ------
> >               enter           enter
> >        enter    |     enter     |
> >          |      |       |       |
> >        leave    |       |       |
> >             <--------------------------- BAD
> >        enter    |       |       |
> >          |      |       |       |
> >        leave  leave   leave   leave
> 
> I don't understand why we don't get horrible faults on the APs
> *immediately* when the BSP closes/locks down SMRAM. Everything in SMRAM,
> page tables, executable code, everything, will read as 0xff on QEMU. How
> can the APs continue in SMM long enough to
> 
> (a) time out and pull the BSP back into SMM,
> (b) complete the rendezvous and exit SMM?

Because the "0xff" only applies when you're out of SMM.  The three
states (open, closed, closed/locked) only apply when you're not in SMM.
While the AP is in SMM they are executing in a separate address space
where SMRAM is "not closed".  (In QEMU that's a separate AddressSpace
struct, smram_address_space in target-i386/kvm.c).

> I think I sort of answered question (2). (Apologies if Paolo and Jiewen
> explained the exact same thing before; I had to spell it out for
> myself.) That leaves question (1) open. Why enter SMM in
> S3ResumeExecuteBootScript() at all?
> 
> Anyway, I think if the BSP and the APs are properly synchronized around
> the SMI injections in S3ResumeExecuteBootScript(), then this bug is
> fixed. In that case, the APs' RSMs will restore the full context for the
> APs, including their sleep in the HLT instruction, in CpuMpPei's wakeup
> buffer. The BSP will proceed, exit PEI (restoring the CpuMpPei wakeup
> buffer -- but the APs will sleep on), and then Linux will bring up the
> APs, after taking control.

Agreed.

Paolo