Re: [platforms: PATCH 4/8] Marvell/Armada: Add support from DRAM remapping

[Marcin Wojtas <mw@semihalf.com>]

Hi Leif,

2017-10-11 20:18 GMT+02:00 Ard Biesheuvel <ard.biesheuvel@linaro.org>:
> On 11 October 2017 at 18:08, Leif Lindholm <leif.lindholm@linaro.org> wrote:
>> Subject: from->for ?
>>
>> On Wed, Oct 11, 2017 at 05:40:45PM +0200, Marcin Wojtas wrote:
>>> From: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>>
>>> The Armada 70x0/80x0 DRAM controller allows a single window of DRAM
>>> to be remapped to another location in the physical address space. This
>>> allows us to free up some memory in the 32-bit addressable region for
>>> peripheral MMIO and PCI MMIO32 and CONFIG spaces. This patch adjusts
>>> memory blocks to the configuration done in ATF.
>>
>> The ARM Trusted Firmware developers tend to frown at that
>> abbreviation. Please use ARM-TF in official context.

Ok.

>>
>> Which configuration is this? Presumably, if this was changed in
>> ARM-TF, we would need to change it this end too, so does not hurt to
>> be explicit. (It uses a FixedPcd, so clearly it is not dynamically
>> detected, which the commit message can be read as.)

I checked and I can modify the code, to obtain from registers an
information about source/target and whether Dram remap is enabled at
all. This way we would be immune to any further changes in ARM-TF.
However I have a question to that setting, please see below.

>>
>>> Contributed-under: TianoCore Contribution Agreement 1.1
>>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> Signed-off-by: Marcin Wojtas <mw@semihalf.com>
>>> ---
>>>  Platform/Marvell/Armada/Library/Armada70x0Lib/AArch64/ArmPlatformHelper.S | 15 +++++
>>>  Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0Lib.inf           |  3 +
>>>  Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0LibMem.c          | 60 ++++++++++++++++----
>>>  Platform/Marvell/Marvell.dec                                              | 13 +++++
>>>  4 files changed, 81 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/Platform/Marvell/Armada/Library/Armada70x0Lib/AArch64/ArmPlatformHelper.S b/Platform/Marvell/Armada/Library/Armada70x0Lib/AArch64/ArmPlatformHelper.S
>>> index 72f8cfc..c5be1a9 100644
>>> --- a/Platform/Marvell/Armada/Library/Armada70x0Lib/AArch64/ArmPlatformHelper.S
>>> +++ b/Platform/Marvell/Armada/Library/Armada70x0Lib/AArch64/ArmPlatformHelper.S
>>> @@ -17,6 +17,21 @@
>>>
>>>  ASM_FUNC(ArmPlatformPeiBootAction)
>>>    mov   x29, xzr
>>> +
>>> +  .if   FixedPcdGet64 (PcdSystemMemoryBase) != 0
>>> +  .err  PcdSystemMemoryBase should be 0x0 on this platform!
>>> +  .endif
>>> +
>>> +  .if   FixedPcdGet64 (PcdSystemMemorySize) > FixedPcdGet32 (PcdDramRemapTarget)
>>> +    //
>>> +    // Use the low range for UEFI itself. The remaining memory will be mapped
>>> +    // and added to the GCD map later.
>>> +    //
>>> +    adr   x0, mSystemMemoryEnd
>>> +    MOV64 (x1, FixedPcdGet32 (PcdDramRemapTarget) - 1)
>>> +    str   x1, [x0]
>>> +  .endif
>>> +

The original commit, due lack of dram size detection, increased total
memory to 4GB, which required above modification with limiting
mSystemMemoryEnd to dram target (0xc0000000). Now PcdSystemMemorySize
is left at somewhat lowest reasonable value (1GB), so above code is in
fact never executed. Are you ok that
I remove it and leave mSystemMemoryEnd to be set as-is in PrePi.c:

UINT64 mSystemMemoryEnd = FixedPcdGet64(PcdSystemMemoryBase) +
                          FixedPcdGet64(PcdSystemMemorySize) - 1;

?

>>>    ret
>>>
>>>  //UINTN
>>> diff --git a/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0Lib.inf b/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0Lib.inf
>>> index 2e198c3..838a670 100644
>>> --- a/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0Lib.inf
>>> +++ b/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0Lib.inf
>>> @@ -67,5 +67,8 @@
>>>    gArmTokenSpaceGuid.PcdArmPrimaryCoreMask
>>>    gArmTokenSpaceGuid.PcdArmPrimaryCore
>>>
>>> +  gMarvellTokenSpaceGuid.PcdDramRemapSize
>>> +  gMarvellTokenSpaceGuid.PcdDramRemapTarget
>>> +
>>>  [Ppis]
>>>    gArmMpCoreInfoPpiGuid
>>> diff --git a/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0LibMem.c b/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0LibMem.c
>>> index 74c9956..2cb2e15 100644
>>> --- a/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0LibMem.c
>>> +++ b/Platform/Marvell/Armada/Library/Armada70x0Lib/Armada70x0LibMem.c
>>> @@ -35,6 +35,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>  #include <Base.h>
>>>  #include <Library/ArmPlatformLib.h>
>>>  #include <Library/DebugLib.h>
>>> +#include <Library/HobLib.h>
>>>  #include <Library/MemoryAllocationLib.h>
>>>
>>>  // The total number of descriptors, including the final "end-of-table" descriptor.
>>> @@ -44,6 +45,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>  #define DDR_ATTRIBUTES_CACHED           ARM_MEMORY_REGION_ATTRIBUTE_NONSECURE_WRITE_BACK
>>>  #define DDR_ATTRIBUTES_UNCACHED         ARM_MEMORY_REGION_ATTRIBUTE_UNCACHED_UNBUFFERED
>>>
>>> +STATIC ARM_MEMORY_REGION_DESCRIPTOR VirtualMemoryTable[MAX_VIRTUAL_MEMORY_MAP_DESCRIPTORS];
>>
>> mVirtualMemoryTable?

Ok.

>>
>>> +
>>>  /**
>>>    Return the Virtual Memory Map of your platform
>>>
>>> @@ -59,20 +62,41 @@ ArmPlatformGetVirtualMemoryMap (
>>>    IN ARM_MEMORY_REGION_DESCRIPTOR** VirtualMemoryMap
>>>    )
>>>  {
>>> -  ARM_MEMORY_REGION_DESCRIPTOR  *VirtualMemoryTable;
>>>    UINTN                         Index = 0;
>>> +  UINT64                        MemSize;
>>> +  UINT64                        MemLowSize;
>>> +  UINT64                        MemHighStart;
>>> +  UINT64                        MemHighSize;
>>> +  EFI_RESOURCE_ATTRIBUTE_TYPE   ResourceAttributes;
>>>
>>>    ASSERT (VirtualMemoryMap != NULL);
>>>
>>> -  VirtualMemoryTable = (ARM_MEMORY_REGION_DESCRIPTOR*)AllocatePages(EFI_SIZE_TO_PAGES (sizeof(ARM_MEMORY_REGION_DESCRIPTOR) * MAX_VIRTUAL_MEMORY_MAP_DESCRIPTORS));
>>> -  if (VirtualMemoryTable == NULL) {
>>> -      return;
>>> -  }
>>> +  MemSize = FixedPcdGet64 (PcdSystemMemorySize);
>>> +  MemLowSize = MIN (FixedPcdGet64 (PcdDramRemapTarget), MemSize);
>>> +  MemHighStart = (UINT64)FixedPcdGet64 (PcdDramRemapTarget) +
>>> +                 FixedPcdGet32 (PcdDramRemapSize);
>>> +  MemHighSize = MemSize - MemLowSize;
>>> +
>>> +  ResourceAttributes = (
>>> +      EFI_RESOURCE_ATTRIBUTE_PRESENT |
>>> +      EFI_RESOURCE_ATTRIBUTE_INITIALIZED |
>>> +      EFI_RESOURCE_ATTRIBUTE_WRITE_COMBINEABLE |
>>> +      EFI_RESOURCE_ATTRIBUTE_WRITE_THROUGH_CACHEABLE |
>>> +      EFI_RESOURCE_ATTRIBUTE_WRITE_BACK_CACHEABLE |
>>> +      EFI_RESOURCE_ATTRIBUTE_TESTED
>>> +  );
>>> +
>>> +  BuildResourceDescriptorHob (
>>> +    EFI_RESOURCE_SYSTEM_MEMORY,
>>> +    ResourceAttributes,
>>> +    FixedPcdGet64 (PcdSystemMemoryBase),
>>> +    MemLowSize
>>> +    );
>>>
>>>    // DDR
>>> -  VirtualMemoryTable[Index].PhysicalBase    = PcdGet64 (PcdSystemMemoryBase);
>>> -  VirtualMemoryTable[Index].VirtualBase     = PcdGet64 (PcdSystemMemoryBase);
>>> -  VirtualMemoryTable[Index].Length          = PcdGet64 (PcdSystemMemorySize);
>>> +  VirtualMemoryTable[Index].PhysicalBase    = FixedPcdGet64 (PcdSystemMemoryBase);
>>> +  VirtualMemoryTable[Index].VirtualBase     = FixedPcdGet64 (PcdSystemMemoryBase);
>>> +  VirtualMemoryTable[Index].Length          = MemLowSize;
>>>    VirtualMemoryTable[Index].Attributes      = DDR_ATTRIBUTES_CACHED;
>>>
>>>    // Configuration space 0xF000_0000 - 0xFFFF_FFFF
>>> @@ -81,13 +105,29 @@ ArmPlatformGetVirtualMemoryMap (
>>>    VirtualMemoryTable[Index].Length          = 0x10000000;
>>>    VirtualMemoryTable[Index].Attributes      = ARM_MEMORY_REGION_ATTRIBUTE_DEVICE;
>>>
>>> +  if (MemSize > MemLowSize) {
>>> +    //
>>> +    // If we have more than MemLowSize worth of DRAM, the remainder will be
>>> +    // mapped at the top of the remapped window.
>>> +    //
>>> +    VirtualMemoryTable[++Index].PhysicalBase  = MemHighStart;
>>> +    VirtualMemoryTable[Index].VirtualBase     = MemHighStart;
>>> +    VirtualMemoryTable[Index].Length          = MemHighSize;
>>> +    VirtualMemoryTable[Index].Attributes      = DDR_ATTRIBUTES_CACHED;
>>> +
>>> +    BuildResourceDescriptorHob (
>>> +      EFI_RESOURCE_SYSTEM_MEMORY,
>>> +      ResourceAttributes,
>>> +      MemHighStart,
>>> +      MemHighSize
>>> +      );
>>> +  }
>>> +
>>>    // End of Table
>>>    VirtualMemoryTable[++Index].PhysicalBase  = 0;
>>>    VirtualMemoryTable[Index].VirtualBase     = 0;
>>>    VirtualMemoryTable[Index].Length          = 0;
>>>    VirtualMemoryTable[Index].Attributes      = 0;
>>>
>>> -  ASSERT((Index + 1) <= MAX_VIRTUAL_MEMORY_MAP_DESCRIPTORS);
>>> -
>>
>> Why delete this assert?

Indeed, it looks like unnecessarily removed. I will restore it,

>>
>>>    *VirtualMemoryMap = VirtualMemoryTable;
>>>  }
>>> diff --git a/Platform/Marvell/Marvell.dec b/Platform/Marvell/Marvell.dec
>>> index 434d6cb..db1c7fa 100644
>>> --- a/Platform/Marvell/Marvell.dec
>>> +++ b/Platform/Marvell/Marvell.dec
>>> @@ -194,6 +194,19 @@
>>>  #TRNG
>>>    gMarvellTokenSpaceGuid.PcdEip76TrngBaseAddress|0x0|UINT64|0x50000053
>>>
>>> +  #
>>> +  # DRAM remapping controls.
>>> +  # On the 70x0/80x0 SOCs, the DRAM is mapped at 0x0, and could be up to
>>> +  # 16 GB in size. To allow for 32-bit addressable MMIO peripherals or PCI
>>> +  # windows, a single window of up to 4 GB in size can be remapped elsewhere.
>>> +  # So let's define a 1 GB window at 0xC000000 by default: this is the minimum
>>> +  # alignment that Linux can map optimally (i.e., it's section shift is 30 bits)
>>> +  # and gives us an additional 768 MB (on top of the 256 MB platform MMIO window
>>> +  # at 0xF0000000) for the PCI MMIO32 and CONFIG spaces.
>>
>> If DRAM is mapped at 0, and can be up to 16GB in size, why do we need
>> to remap some of it below 4GB?
>> I guess what you're actually doing is mapping it _away_. Where to?
>>
>> (And a short-short version of this in the commit message should take
>> care of that item.)
>>
>
> Indeed. Physical RAM starts at 0x0, and to avoid wasting all 32-bit
> addressable memory space on RAM, the hardware allows some of it to be
> remapped elsewhere. So by remapping 1 GB, we make 1 GB of address
> space available for MMIO, PCI config space and PCI MMIO32 space.

If we agree to detect remap size and target from registers, those PCDs
(and comment) will be removed. I will modify the commit log only.

Best regards,
Marcin