From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=2607:f8b0:4864:20::341; helo=mail-ot1-x341.google.com; envelope-from=vladimir.olovyannikov@broadcom.com; receiver=edk2-devel@lists.01.org Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 1076F2115471B for ; Fri, 21 Sep 2018 16:24:38 -0700 (PDT) Received: by mail-ot1-x341.google.com with SMTP id w17-v6so14623435otk.3 for ; Fri, 21 Sep 2018 16:24:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Mz3sgDDKf2Poz5mZEQRH1dZ7qUvnujqJq8Tc4eLHxoE=; b=cey9QiFbl2o5vNrpkAYdcGh0b9GmOwZbgCeMnFLj5lce3iTM5ZvILYa/7mddNSDT9v gcvmKq4AAP3jl7uRPLeGLiEef6XOqCDJEGRhE/Ws2Qs0th3h7BEDmeF05oz8HQ5AnlFD 6g1aLyJ26KITHtuAFK6bfMeaOr3/R1wsnwlUo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Mz3sgDDKf2Poz5mZEQRH1dZ7qUvnujqJq8Tc4eLHxoE=; b=mnhc9Kb85ja7Ebl1NA0KNBKGm93leFlodvv7wAN0R81qNT0UZnu5VHHfNxwrUDKG28 1PDbOu8T17JrkZxWeToJ7sI3UY2aO3AxQV7d2SdEnjFBypGbOmd+YURPPhIfOIKcC79x yRr4Ph7wpU2qCE5M8rY93YQwNsJdUs4EyWa2w7k/SrCoAFHop4zl16XOb5MAdAbvnW2J +UagV/8PT0LGqD3b+jRDSiqs1aCJ7xgEe/wjUOePEkZp2lqB1/uXR1FWpgzACAlNAqTA tUR8hoZmMUojI39OiexzZj08YkqLQZvw3hos+OLGKfEHxULONz51lMMZshg6go6ySF7x QgNg== X-Gm-Message-State: APzg51CrmAPo6L0s4/hjr1B+w3aOjsv4cuiwBMTaWpjyNBKXjY8Fk2en E9aZwG1M0bM2xZpHx8qGmgiRxQvZK8ZXdqrQycy8TQ== X-Google-Smtp-Source: ANB0VdYuaFC4tH7rJ1k8282vcL4Wo0E+tVBnFQE2FO8ghm5bM48w5tTiFntF9A14VKewyXG2otGu1YENNa2fPPKtfW8= X-Received: by 2002:a9d:689:: with SMTP id 9-v6mr25682318otx.149.1537572276487; Fri, 21 Sep 2018 16:24:36 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vladimir Olovyannikov Date: Fri, 21 Sep 2018 16:24:24 -0700 Message-ID: To: Bill Paul Cc: "edk2-devel@lists.01.org" , Ard Biesheuvel Subject: Re: Stack issue after warm UEFI reset and MMU enabling on an Armv8 platform X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Sep 2018 23:24:39 -0000 Content-Type: text/plain; charset="UTF-8" On Thu, Sep 20, 2018 at 2:52 PM Vladimir Olovyannikov wrote: > > On Wed, Sep 19, 2018 at 5:21 PM Bill Paul wrote: > > > > Of all the gin joints in all the towns in all the world, Vladimir > > Olovyannikov > > had to walk into mine at 16:58 on Wednesday 19 September 2018 and say: > > > > > >From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org] > > > >Sent: Wednesday, September 19, 2018 4:38 PM > > > >To: Vladimir Olovyannikov > > > >Cc: edk2-devel@lists.01.org > > > >Subject: Re: Stack issue after warm UEFI reset and MMU enabling on an > > > >Armv8 platform > > > > > > > > > > > >On 19 September 2018 at 15:55, Vladimir Olovyannikov > > > > > > > > wrote: > > > >>Hi All, > > > >> > > > >>I need UEFI experts help on the problem with Armv8 board on warm UEFI > > > >>reset. > > > >>Cold reset works fine. > > > >> > > > >>Here is how I set up a warm reset: > > > >> > > > >>STATIC > > > >>EFI_STATUS > > > >>ShutdownUefiBootServices ( > > > >> > > > >> VOID > > > >> ) > > > >> > > > >>{ > > > >> > > > >> EFI_STATUS Status; > > > >> UINTN MemoryMapSize; > > > >> EFI_MEMORY_DESCRIPTOR *MemoryMap; > > > >> UINTN MapKey; > > > >> UINTN DescriptorSize; > > > >> UINT32 DescriptorVersion; > > > >> UINTN Pages; > > > >> > > > >> MemoryMap = NULL; > > > >> MemoryMapSize = 0; > > > >> Pages = 0; > > > >> > > > >> do { > > > >> > > > >> Status = gBS->GetMemoryMap ( > > > >> > > > >> &MemoryMapSize, > > > >> MemoryMap, > > > >> &MapKey, > > > >> &DescriptorSize, > > > >> &DescriptorVersion > > > >> ); > > > >> > > > >> if (Status == EFI_BUFFER_TOO_SMALL) { > > > >> > > > >> Pages = EFI_SIZE_TO_PAGES (MemoryMapSize) + 1; > > > >> MemoryMap = AllocatePages (Pages); > > > >> > > > >> // > > > >> // Get System MemoryMap > > > >> // > > > >> Status = gBS->GetMemoryMap ( > > > >> > > > >> &MemoryMapSize, > > > >> MemoryMap, > > > >> &MapKey, > > > >> &DescriptorSize, > > > >> &DescriptorVersion > > > >> ); > > > >> > > > >> } > > > >> > > > >> // Don't do anything between the GetMemoryMap() and > > > >> ExitBootServices() if (!EFI_ERROR(Status)) { > > > >> > > > >> Status = gBS->ExitBootServices (gImageHandle, MapKey); > > > >> if (EFI_ERROR(Status)) { > > > >> > > > >> FreePages (MemoryMap, Pages); > > > >> MemoryMap = NULL; > > > >> MemoryMapSize = 0; > > > >> > > > >> } > > > >> > > > >> } > > > >> > > > >> } while (EFI_ERROR(Status)); > > > >> > > > >> return Status; > > > >> > > > >>} > > > >> > > > >>Then perform > > > >>ArmCleanDataCache (); > > > >>ArmInvalidateDataCache (); > > > >>ArmDisableInstructionCache (); > > > >>ArmInvalidateInstructionCache (); > > > > > > > >These don't do anything useful on ARM. You can only reliably perform > > > >cache > > > >maintenance by virtual address. > > > > > > So, should I just remove them altogether? > > > > > > >>ArmDisableMmu (); > > > > > > > >... so after this call returns, all bets are off with regards to > > > >whether > > > >what is popped from the stack is actually what we pushed when we > > > >entered > > > >the function. > > > > > > OK, thank you for explanation. > > > But this call returns back into ResetLib implementation as it should, > > > and > > > then there is a direct jump to the start of FV. > > > Am I doing anything wrong here? > > > Then, up to the point of enabling of MMU the stack is OK. But right > > > after > > > enabling MMU it points at _ModuleEntryPoint end of function in > > > DxeCoreEntryPoint.c > > > Am I missing anything? Maybe some stack cleanup before jumping to the > > > start > > > of FV? > > > > When the MMU is enabled, does the mapping for the stack pages change? That > > is, > > could the stack now be mapped to different physical page now? > Thanks for ideas Bill, > No, the mapping stays the same. > The issue is only with warm reset, and only on an A72 board. > There is another platform on A53 sharing the same code, which has no issues > with warm reset. > I cannot explain why. > > > > Instead of showing a stack trace, can you dump the stack pages and compare > > the > > before and after contents? > I can clearly see that before and after contents are different. > > > > Assuming the same physical memory pages are still being used, then there > > could > > be a cache flushing problem. What could happen is: > > > > - some stack memory has been touched recently and is now in the data cache > > - changes are made, which are written to the cache, but not yet flushed > > out to > > RAM > > - enabling the MMU causes a full invalidate of the cache > > > > Now when you look at the stack, you see the earlier contents that were in > > RAM > > -- the changes previously only written to the cache have been lost. > > > > Enabling/disabling caches and MMU is always tricky. I can't say for sure, > > but > > I wouldn't be surprised if there's some subtle bug that causes a flush > > operation to be missed and things may just work by coincidence in the cold > > start case. > I might be missing something preparing for warm reset. > Disabling interrupts does not help though. > Ard, I switched off all DMA-capable devices, so am just booting into UEFI > with no disks or network, > disabled interrupts. The issue is here. Any ideas on how to debug it and > fix? Update: when I add this as an experiment: UINTN StackBottom; __asm__ volatile ("mov %0,SP" : "=r"(StackBottom)); WriteBackInvalidateDataCacheRange((VOID *)StackBottom, PcdGet64(PcdSystemMemorySize) + PcdGet64(PcdSystemMemoryBase) - StackBottom); then the stack data were not corrupted anymore on the next ArmEnableMmu() call, and warm reset worked (though unreliably, can throw exception on memcpy down the road on UEFI boot; probably because I invalidated only the stack area in the experiment). Considering that A53 board does not have issues, does this means that ArmInvalidateDataCache() implementation is useless for A72? Based on this, should the approach be "find all data regions and invalidate them using InvalidateDataCacheRange()"? > > > > -Bill > > > > > >>Then jump to start of FV: > > > >> > > > >>typedef > > > >>VOID > > > >> > > > >> (EFIAPI *START_FV)( > > > >> > > > >> VOID > > > >> > > > >>); > > > >>StartOfFv = (START_FV)(UINTN)PcdGet64(PcdFvBaseAddress); > > > >>StartOfFv (); > > > >> > > > >>Now this is what happens on warm reset: > > > >>reset -c warm > > > >>1. Until ArmEnableMmu() gets called, everything works as expected. > > > >> > > > >> Here is the stack right before ArmEnableMmu() is called: > > > >> ArmConfigureMmu+0x4f8 > > > >> InitMmu+0x24 > > > >> MemoryPeim+0x440 > > > >> PrePiMain+0x114 > > > >> PrimaryMain+0x68 > > > >> CEntryPoint+0xC4 > > > >> EL2:0x00000000800008BC > > > >> ----- End of stack info ----- > > > >> > > > >>2. Here is the stack as soon as Mmu is enabled with ArmEnableMmu() : > > > >> ArmConfigureMmu+0x4fc <-- This one is correct, at line 745 in > > > >> > > > >> ArmConfigureMmu() in ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c > > > >> (return EFI_SUCCESS) > > > >> > > > >> _ModuleEntryPoint+0x24 <-- Wrong. This points directly to > > > >> > > > >> ASSERT(FALSE); and to CpuDeadLoop() in DxeCoreEntryPoint.c, lines > > > >> 59-60. > > > >> > > > >> El2:0x000000008E5E8300 <-- Absolutely bogus > > > >> > > > >> --- End of stack info --- > > > >> > > > >>So, as soon as ArmEnableMmu() exits, execution jumps directly to > > > >>CpuDeadLoop() in DxeCoreEntryPoint of _ModuleEntryPoint(). > > > >> > > > >>Would be grateful for any advice. > > > >> > > > >>Thank you, > > > >>Vladimir > > > > > > _______________________________________________ > > > edk2-devel mailing list > > > edk2-devel@lists.01.org > > > https://lists.01.org/mailman/listinfo/edk2-devel > > -- > > ============================================================================= > > -Bill Paul (510) 749-2329 | Senior Member of Technical Staff, > > wpaul@windriver.com | Master of Unix-Fu - Wind River > > Systems > > ============================================================================= > > "I put a dollar in a change machine. Nothing changed." - George Carlin > > =============================================================================