Re: [PATCH] ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe: Fixed crash on Juno R0

[Daniil Egranov <daniil.egranov@arm.com>]

Hi Ryan,


On 01/20/2017 04:30 AM, Ryan Harkin wrote:
> On 20 January 2017 at 01:34, Daniil Egranov <daniil.egranov@arm.com> wrote:
>> Hi Leif, Ryan
>>
>>
>> On 01/19/2017 09:13 AM, Leif Lindholm wrote:
>>
>> On Thu, Jan 19, 2017 at 01:49:04PM +0000, Ryan Harkin wrote:
>>
>> On 18 January 2017 at 23:27, Daniil Egranov <daniil.egranov@arm.com> wrote:
>>
>> The Marvell Yukon MAC address load supported only on Juno R1 and R2.
>> It disabled for Juno R0 due to PCI issues on this board.
>>
>> Contributed-under: TianoCore Contribution Agreement 1.0
>> Signed-off-by: Daniil Egranov <daniil.egranov@arm.com>
>>
>> Tested-by: Ryan Harkin <ryan.harkin@linaro.org>
>>
>> ---
>>   ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c | 9 +++++++--
>>   1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c
>> b/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c
>> index 47ff587..e9e6990 100644
>> --- a/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c
>> +++ b/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c
>> @@ -378,6 +378,7 @@ OnEndOfDxe (
>>     EFI_DEVICE_PATH_PROTOCOL* PciRootComplexDevicePath;
>>     EFI_HANDLE                Handle;
>>     EFI_STATUS                Status;
>> +  UINT32                    JunoRevision;
>>
>>     //
>>     // PCI Root Complex initialization
>> @@ -393,8 +394,12 @@ OnEndOfDxe (
>>     Status = gBS->ConnectController (Handle, NULL, PciRootComplexDevicePath,
>> FALSE);
>>     ASSERT_EFI_ERROR (Status);
>>
>> -  Status = ArmJunoSetNicMacAddress ();
>> -  ASSERT_EFI_ERROR (Status);
>> +  GetJunoRevision (JunoRevision);
>> +
>> +  if (JunoRevision != JUNO_REVISION_R0) {
>> +    Status = ArmJunoSetNicMacAddress ();
>> +    ASSERT_EFI_ERROR (Status);
>>
>> This is just an FYI, but I stacked your patch on top of mainline, like this:
>>
>> 5f81f61  2017-01-18  ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe:
>> Fixed crash on Juno R0       [Daniil Egranov]
>> 19ca06b  2017-01-19  OvmfPkg: Remove superfluous return statements.
>>   [Thomas Huth]
>>
>> The first time I ran this, Juno R0 worked fine, but on R1 and R2, the
>> assert triggered:
>>
>> UEFI firmware (version 5f81f61 built at 11:56:52 on Jan 19 2017)
>> [snip]
>> ASSERT_EFI_ERROR (Status = Not Found)
>> ASSERT [ArmJunoDxe]
>> /linaro/platforms/uefi/edk2/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c(401):
>> !EFI_ERROR (Status)
>>
>> I worked out what is happening. And it's not to do with this patch.
>> It's another fall-out from the re-work you did to the previous patch.
>> It's also ultimately due to a bug the firmware.
>>
>> With the initial version of your "Set Marvell Yukon MAC address"
>> patch, this hang didn't happen. I suspect that was because your error
>> checking was weaker and certain PCIe failures didn't trigger the
>> assert.
>>
>> To reproduce the error with this commit:
>> 1) power on and boot R1 or R2 into Shell
>>    I do this by interrupting the boot by pressing ESCAPE and using the boot
>> menu
>> 2) At the Shell prompt, run "reset -s" to shutdown
>> 3) At the ARM Boot Loader "Cmd>" prompt, run "reboot"
>> 4) the board will hang while booting UEFI, assuming the board firmware
>> doesn't die with constant messages like this:
>>
>>      ERROR: PCIe CSR read failed to respond
>>      ERROR: SMBus transaction not claimed
>>
>> Assuming the problem is firmware, not EDK2, what should we do about it?
>>
>> OK, so instinctively, my reaction was that "the reset -s bug is a
>> system controller firmware bug and we shouldn't work around
>> it". However, since it is actually disrupting Ryan's workflow, which
>> frequently doesn't touch PCI at all, I think downgrading the ASSERT to
>> an error message is a good idea short-term.
>>
>> Daniil - could you make that change please?
>>
>> /
>>      Leif
>>
>>
>> I've been able to reproduce "PCIe CSR read failed to respond" and "SMBus
>> transaction not claimed" errors on my Juno R2. I disabled Marvell Yukon
>> driver (.dsc/.fdf) and removed ArmJunoDxe patch but still see the same error
>> messages during the initial boot.
>>
>> Testing motherboard interfaces (FPGA build 118)...
>> SRAM 32MB test: PASSED
>> LAN9118   test: PASSED
>> KMI1/2    test: PASSED
>> MMC       test: PASSED
>> PB/LEDs   test: PASSED
>> FPGA UART test: PASSED
>> ERROR: PCIe CSR read failed to respond
>> ERROR: SMBus transaction not claimed
>> ERROR: PCIe CSR read failed to respond
>> ...
>>
>> Once it went through reporting these errors, the UEFI starts loading but
>> still fails in OnEndOfDxe():
>> ASSERT_EFI_ERROR (Status = Not Found)
>> ASSERT [ArmJunoDxe]
>> /home/user/workspace/juno/uefi/edk2/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c(110):
>> !EFI_ERROR (Status)
>>
>> This is the original ArmJunoDxe code:
>>    Status = gBS->ConnectController (Handle, NULL, PciRootComplexDevicePath,
>> FALSE);
>>    ASSERT_EFI_ERROR (Status); <---- line 110
> I'm confused at which point in the commit history this is line 110.
>
> Just before your MAC commit, it's line 108. After your commit, and in
> the current mainline, it's line 394.
>
> Then, I have to go back to Jeremy's commit
> 58a4bff071832e587d18f97597b5d571dcebc9d7 on 27 July 2016 before I see
> another change to ArmJunoDxe.c. Before that commit, the assert appears
> on line 116.
>
> On my tree, 17.01, 16.12 and 16.11 has the assert on line 246, 16.10
> on line 125.
>
> So I'm unsure what code you are running.
>

This shows the code where it fails for me after MAC address code was 
removed from ArmJunoDxe.c. It's part of my development branch and lines 
may not matching any public commits ... but i pulled 16.06 release for a 
test and see the ASSERT on the same code in case of  "PCIe .." and 
"SMBus .." errors. The line number should be corresponding to 
032082bf9906f82b77c161c1c2efbddfd4bb3751 commit:

ASSERT_EFI_ERROR (Status = Not Found)
ASSERT [ArmJunoDxe] 
/home/user/workspace/juno_16.06_orig/uefi/edk2/ArmPlatformPkg/ArmJunoPkg/Drivers/ArmJunoDxe/ArmJunoDxe.c(124): 
!EFI_ERROR (Status)

>
>> but it actually fails here first:
>>    Status = gBS->LocateDevicePath (&gEfiPciRootBridgeIoProtocolGuid,
>>                                    &PciRootComplexDevicePath,
>>                                    &Handle);
>>
>> Ryan - could you try to remove Marvell patches and check if you also
>> catching "PCIe .." and "SMBus .." errors without them and your build still
>> fails with other ASSERTs related to PCI.
>>
>> Leif - in my tests, if it fails with "PCIe .." and "SMBus ..", the UEFI PCI
>> enumeration is completely corrupted.
>>
>> These errors do not appear if board was reset with the nPBRESET button. It
>> never fails with "reset -w" and "reset -c". I also loaded Debian and used
>> the shutdown command from there and got the same "PCIe .." and "SMBus .."
>> errors after "reboot" command from the "Cmd>" prompt.  Possibly, the
>> "reboot" command from the board shell prompt does not reset the board
>> correctly so it looks like a firmware issue.
>>
> Yes, that's pretty much what I've been saying. There appears to be a
> firmware bug that causes PCIe to be in an unstable state after
> software shutdown of the AP by the motherboard, followed by reset.
>
> The "PCIe CSR read failed to respond" error is nothing to do with EDK2
> and happens before EDK2 runs.
>
> However, unlike you, if I don't get the "PCIe CSR read failed to
> respond" error, upstream EDK2 before your MAC address commit will boot
> without ASSERTing on R0/1/2.

Do you mean, when you do have the "PCIe .." error ? As i understand, if 
you power-cycle the board (no "PCIe .." errors) you do not see ASSERT at 
the MAC address code during the first boot and can start UEFI Shell?

> So I've never seen the assert at line 110 as you pointed out above.
> That's curious. I can see why that would/should assert, but it doesn't
> for me. Perhaps it's something to do with the tree you are building
> from? Or the devices you have connected? I have no additional SATA or
> PCIe devices connected to my board.
>
> Cheers,
> Ryan.

I am using R2 for the tests. The tests were done without SATA or PCI 
devices connected to the board. Unfortunately, I do not have R1 board at 
this moment to check it.

This is my R2 firmware versions:

ARM V2M-Juno Boot loader v1.0.0
HBI0262 build 1844

ARM V2M_Juno r2 Firmware v1.4.3
Build Date: Apr 12 2016

MIC RAM configuration (pms_v105.bin)...

Configuring motherboard (rev D, var A)...
IOFPGA image \MB\HBI0262D\io_b118.bit

As Leif suggested, i will replace the ASSERT with an error message and 
send another patch.

Could you please check one thing for me on your boards? Use a build 
which worked for you (no ASSERT) and your steps: UEFI shell -> reset -s 
-> reboot -> see "PCIe .." and "SMBus .." errors -> boot UEFI. As i 
understand, you do not have the ASSERT with your old builds so you can 
start the UEFI shell. Use the "pci" shell command to list all PCI 
devices. Do you see them correctly enumerated? I'd like to make sure 
that by ignoring the ASSERT errors we will not face any further problems 
with PCI.

Thanks,
Daniil

>> Thanks,
>> Daniil
>>
>>
>> Prior to your "Set Marvell Yukon MAC address" patch, or with the
>> earlier version, the board would boot anyway, but the Yukon device
>> would be missing.
>>
>> Now it dies.
>>
>> I don't know which is worse, but I think hanging is worse than an
>> ethernet port dropping out. Although hanging is a bit more obvious
>> that there's a problem...
>>
>>
>> +  }
>>   }
>>
>>   STATIC
>> --
>> 2.7.4
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>>
>>
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel