Re: Stack issue after warm UEFI reset and MMU enabling on an Armv8 platform

[Bill Paul <wpaul@windriver.com>]

Of all the gin joints in all the towns in all the world, Vladimir Olovyannikov 
had to walk into mine at 16:58 on Wednesday 19 September 2018 and say:

> >From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> >Sent: Wednesday, September 19, 2018 4:38 PM
> >To: Vladimir Olovyannikov
> >Cc: edk2-devel@lists.01.org
> >Subject: Re: Stack issue after warm UEFI reset and MMU enabling on an
> >Armv8 platform
> >
> >
> >On 19 September 2018 at 15:55, Vladimir Olovyannikov
> >
> ><vladimir.olovyannikov@broadcom.com> wrote:
> >>Hi All,
> >>
> >>I need UEFI experts help on the problem with Armv8 board on warm UEFI
> >>reset.
> >>Cold reset works fine.
> >>
> >>Here is how I set up a warm reset:
> >>
> >>STATIC
> >>EFI_STATUS
> >>ShutdownUefiBootServices (
> >>
> >>  VOID
> >>  )
> >>
> >>{
> >>
> >>  EFI_STATUS              Status;
> >>  UINTN                   MemoryMapSize;
> >>  EFI_MEMORY_DESCRIPTOR   *MemoryMap;
> >>  UINTN                   MapKey;
> >>  UINTN                   DescriptorSize;
> >>  UINT32                  DescriptorVersion;
> >>  UINTN                   Pages;
> >>  
> >>  MemoryMap = NULL;
> >>  MemoryMapSize = 0;
> >>  Pages = 0;
> >>  
> >>  do {
> >>  
> >>    Status = gBS->GetMemoryMap (
> >>    
> >>                    &MemoryMapSize,
> >>                    MemoryMap,
> >>                    &MapKey,
> >>                    &DescriptorSize,
> >>                    &DescriptorVersion
> >>                    );
> >>    
> >>    if (Status == EFI_BUFFER_TOO_SMALL) {
> >>    
> >>      Pages = EFI_SIZE_TO_PAGES (MemoryMapSize) + 1;
> >>      MemoryMap = AllocatePages (Pages);
> >>      
> >>      //
> >>      // Get System MemoryMap
> >>      //
> >>      Status = gBS->GetMemoryMap (
> >>      
> >>                      &MemoryMapSize,
> >>                      MemoryMap,
> >>                      &MapKey,
> >>                      &DescriptorSize,
> >>                      &DescriptorVersion
> >>                      );
> >>    
> >>    }
> >>    
> >>    // Don't do anything between the GetMemoryMap() and
> >>    ExitBootServices() if (!EFI_ERROR(Status)) {
> >>    
> >>      Status = gBS->ExitBootServices (gImageHandle, MapKey);
> >>      if (EFI_ERROR(Status)) {
> >>      
> >>        FreePages (MemoryMap, Pages);
> >>        MemoryMap = NULL;
> >>        MemoryMapSize = 0;
> >>      
> >>      }
> >>    
> >>    }
> >>  
> >>  } while (EFI_ERROR(Status));
> >>  
> >>  return Status;
> >>
> >>}
> >>
> >>Then perform
> >>ArmCleanDataCache ();
> >>ArmInvalidateDataCache ();
> >>ArmDisableInstructionCache ();
> >>ArmInvalidateInstructionCache ();
> >
> >These don't do anything useful on ARM. You can only reliably perform cache
> >maintenance by virtual address.
> 
> So, should I just remove them altogether?
> 
> >>ArmDisableMmu ();
> >
> >... so after this call returns, all bets are off with regards to whether
> >what is popped from the stack is actually what we pushed when we entered
> >the function.
> 
> OK, thank you for explanation.
> But this call returns back into ResetLib implementation as it should, and
> then there is a direct jump to the start of FV.
> Am I doing anything wrong here?
> Then, up to the point of enabling of MMU the stack is OK. But right after
> enabling MMU it points at _ModuleEntryPoint end of function in
> DxeCoreEntryPoint.c
> Am I missing anything? Maybe some stack cleanup before jumping to the start
> of FV?

When the MMU is enabled, does the mapping for the stack pages change? That is, 
could the stack now be mapped to different physical page now?

Instead of showing a stack trace, can you dump the stack pages and compare the 
before and after contents?

Assuming the same physical memory pages are still being used, then there could 
be a cache flushing problem. What could happen is:

- some stack memory has been touched recently and is now in the data cache
- changes are made, which are written to the cache, but not yet flushed out to 
RAM
- enabling the MMU causes a full invalidate of the cache

Now when you look at the stack, you see the earlier contents that were in RAM 
-- the changes previously only written to the cache have been lost.

Enabling/disabling caches and MMU is always tricky. I can't say for sure, but 
I wouldn't be surprised if there's some subtle bug that causes a flush 
operation to be missed and things may just work by coincidence in the cold 
start case.

-Bill
 
> >>Then jump to start of FV:
> >>
> >>typedef
> >>VOID
> >>
> >> (EFIAPI *START_FV)(
> >> 
> >>  VOID
> >>
> >>);
> >>StartOfFv = (START_FV)(UINTN)PcdGet64(PcdFvBaseAddress);
> >>StartOfFv ();
> >>
> >>Now this is what happens on warm reset:
> >>reset -c warm
> >>1. Until ArmEnableMmu() gets called, everything works as expected.
> >>
> >>    Here is the stack right before ArmEnableMmu() is called:
> >>     ArmConfigureMmu+0x4f8
> >>     InitMmu+0x24
> >>     MemoryPeim+0x440
> >>     PrePiMain+0x114
> >>     PrimaryMain+0x68
> >>     CEntryPoint+0xC4
> >>     EL2:0x00000000800008BC
> >>     -----  End of stack info -----
> >>
> >>2. Here is the stack as soon as Mmu is enabled with ArmEnableMmu() :
> >>    ArmConfigureMmu+0x4fc <-- This one is correct, at line 745 in
> >> 
> >> ArmConfigureMmu() in ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c
> >> (return EFI_SUCCESS)
> >> 
> >>   _ModuleEntryPoint+0x24 <-- Wrong. This points directly to
> >> 
> >> ASSERT(FALSE); and to CpuDeadLoop() in DxeCoreEntryPoint.c, lines 59-60.
> >> 
> >>   El2:0x000000008E5E8300 <-- Absolutely bogus
> >>   
> >>    --- End of stack info ---
> >>
> >>So, as soon as ArmEnableMmu() exits, execution jumps directly to
> >>CpuDeadLoop() in DxeCoreEntryPoint of _ModuleEntryPoint().
> >>
> >>Would be grateful for any advice.
> >>
> >>Thank you,
> >>Vladimir
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
-- 
=============================================================================
-Bill Paul            (510) 749-2329 | Senior Member of Technical Staff,
                 wpaul@windriver.com | Master of Unix-Fu - Wind River Systems
=============================================================================
   "I put a dollar in a change machine. Nothing changed." - George Carlin
=============================================================================