From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id C89949414A1 for ; Wed, 1 Nov 2023 08:03:43 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=fQzec9kyaSUl6ZjtZ+rOosPJ41E1XM9k8lwPHh+Bu2o=; c=relaxed/simple; d=groups.io; h=Subject:To:From:User-Agent:MIME-Version:Date:References:In-Reply-To:Message-ID:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type; s=20140610; t=1698825822; v=1; b=KQaTapBRgYEW2+G7k37jDlUQeFh3WFQh34HI5YT+UVe+nvaT8gl2G570MfYpqGOmd1cSjbmh XRZQMITG76mNE41fEDFnhOb1dEzaXcxE+FnYCsXQtFAblybXx7ntcm0lhJ7mD1wcfO6oN1anfYf OmC4ue1ExNLHQulubQz/n/a0= X-Received: by 127.0.0.2 with SMTP id ccXDYY7687511xucE4sAkgHy; Wed, 01 Nov 2023 01:03:42 -0700 Subject: Re: [edk2-devel] [PATCH v7 3/5] MdePkg: Implement RISC-V Cache Management Operations To: Dhaval Sharma ,devel@edk2.groups.io From: "Jingyu Li via groups.io" X-Originating-Location: Central, Central and Western District, HK (103.68.183.115) X-Originating-Platform: Windows Chrome 118 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Wed, 01 Nov 2023 01:03:41 -0700 References: In-Reply-To: Message-ID: <28122.1698825821629324796@groups.io> Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,jingyu.li01@sophgo.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: KlJcBlYBrqbxNUuquYHHyJDyx7686176AA= Content-Type: multipart/alternative; boundary="HkMb6UoOVRqc6oRKres2" X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=KQaTapBR; dmarc=none; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io --HkMb6UoOVRqc6oRKres2 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 31, 2023 at 05:55 PM, Dhaval Sharma wrote: >=20 > I am posting an update on behalf of Jingyu as he had trouble with posting= . > CC'ing him here: > In summary what we have verified so far: > * I have verified that instructions/op codes are okay. I have also > verified on Qemu that functionally it seems to be calling correct > instructions. Ensured with negative test cases that any other op codes=C2= =A0do > cause exceptions as expected. > * Jingyu was able to verify the CpuFlushCpuDataCache function with this > framework (he had to use custom op code based on his soc implementation) > on SG2042. There is one issue that he=C2=A0is debugging now which is rela= ted to > other cache instructions and he will get back with more data. P.S. SG2042 > does not implement the exact same CMO opcodes but equivalent ones. So thi= s > experiment is just an additional data point that helps verify the > framework and not CMO itself. > * In general it sounds like framework flows are alright and as long as > instructions do their job as claimed in the spec, it is lower risk. > Guess this is what we have so far. If it makes sense to everyone, we coul= d > go ahead with merging with this *feature disabled by default* after Jingy= u > provides clarity reg failures on SG2042 platform. Otherwise we can wait > until newer Si is available where these exact instructions can be tested > and then upstreamed. >=20 > [From Jingyu] > I verified this CMO framework on an actual HW platform. >=20 > SW: > edk2: https://github.com/rivosinc/edk2/tree/dev-rv-cmo-v7 branch: > dev-rv-cmo-v7 > edk2-platforms: https://github.com/sophgo/edk2-platforms branch: sg2042-d= ev >=20 >=20 > HW: > Milk-V Pioneer Box, a developer motherboard based on SG2042 with 64-Core > T-HEAD C920. >=20 > Attention: > The T-HEAD C920 implemented its own CMO Extension and is different from > the standard CMO Extension. >=20 > Test steps: > 1. Modified the opcodes in RiscVasm.inc to accommodate the C920 CMO > feature. Update the test status. The InvalidateInstructionCacheRange and the InvalidateDataCacheRange execut= e the same instruction "cbo.inval", but the T-HEAD C920 executes different = instructions for the two functions. Please see the form below for details. I replaced ".long 0x02a5000b" with "fence.i", which solved unexpected excep= tions that occurred yesterday. RISC-V standard CMO Extension C920 CMO Extension WriteBackInvalidateDataCache WriteBackInvalidateDataCacheRange cbo. flush ".long 0x02b5000b" WriteBackDataCache WriteBackDataCacheRange cbo. clean ".long 0x02b5000b" InvalidateDataCache fence fence InvalidateDataCacheRange cbo. inval ".long 0x02a5000b" InvalidateInstructionCache fence. i fence. i InvalidateInstructionCacheRange cbo. inval fence. i Now, I enable CMO in the entire edk2 phase, not just during PCIe inbound. N= o exceptions found so far.=C2=A0Hope to give you some reference. Update the patches: diff --git a/MdePkg/Include/RiscV64/RiscVasm.inc b/MdePkg/Include/RiscV64/R= iscVasm.inc index 29de735885..c2e573eb3d 100644 --- a/MdePkg/Include/RiscV64/RiscVasm.inc +++ b/MdePkg/Include/RiscV64/RiscVasm.inc @@ -7,13 +7,13 @@ */ .macro RISCVCMOFLUSH -=C2=A0 =C2=A0 .word 0x25200f +=C2=A0 =C2=A0 .long 0x02b5000b^M .endm .macro RISCVCMOINVALIDATE -=C2=A0 =C2=A0 .word 0x05200f +=C2=A0 =C2=A0 .long 0x02a5000b^M .endm .macro RISCVCMOCLEAN -=C2=A0 =C2=A0 .word 0x15200f +=C2=A0 =C2=A0 .long 0x02b5000b^M .endm diff --git a/MdePkg/Library/BaseCacheMaintenanceLib/RiscVCache.c b/MdePkg/L= ibrary/BaseCacheMaintenanceLib/RiscVCache.c index 5b3104afb6..ee85d0548c 100644 --- a/MdePkg/Library/BaseCacheMaintenanceLib/RiscVCache.c +++ b/MdePkg/Library/BaseCacheMaintenanceLib/RiscVCache.c @@ -89,7 +89,7 @@ CacheOpCacheRange ( Start &=3D ~((UINTN)CacheLineSize - 1); DEBUG ( -=C2=A0 =C2=A0 (DEBUG_INFO, +=C2=A0 =C2=A0 (DEBUG_VERBOSE,^M "CacheOpCacheRange:\ Performing Cache Management Operation %d \n", Op) ); @@ -163,7 +163,8 @@ InvalidateInstructionCacheRange ( ) { if (RiscVIsCMOEnabled ()) { -=C2=A0 =C2=A0 CacheOpCacheRange (Address, Length, Invld); +=C2=A0 =C2=A0 // CacheOpCacheRange (Address, Length, Invld);^M +=C2=A0 =C2=A0 InvalidateInstructionCache ();^M } else { DEBUG ( (DEBUG_VERBOSE, diff --git a/UefiCpuPkg/CpuDxeRiscV64/CpuDxe.c b/UefiCpuPkg/CpuDxeRiscV64/C= puDxe.c index 2af3b62234..824667bc87 100644 --- a/UefiCpuPkg/CpuDxeRiscV64/CpuDxe.c +++ b/UefiCpuPkg/CpuDxeRiscV64/CpuDxe.c @@ -9,6 +9,7 @@ **/ #include "CpuDxe.h" +#include ^M // // Global Variables @@ -59,7 +60,7 @@ EFI_CPU_ARCH_PROTOCOL=C2=A0 gCpu =3D { CpuGetTimerValue, CpuSetMemoryAttributes, 1,=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 // NumberOfTimers -=C2=A0 4=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0// DmaBufferAlignment +=C2=A0 64=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 // DmaBufferAlignment^M }; // @@ -90,6 +91,20 @@ CpuFlushCpuDataCache ( IN EFI_CPU_FLUSH_TYPE=C2=A0 =C2=A0 =C2=A0FlushType ) { +=C2=A0 switch (FlushType) {^M +=C2=A0 =C2=A0 case EfiCpuFlushTypeWriteBack:^M +=C2=A0 =C2=A0 =C2=A0 WriteBackDataCacheRange ((VOID *)(UINTN)Start, (UINTN= )Length);^M +=C2=A0 =C2=A0 =C2=A0 break;^M +=C2=A0 =C2=A0 case EfiCpuFlushTypeInvalidate:^M +=C2=A0 =C2=A0 =C2=A0 InvalidateDataCacheRange ((VOID *)(UINTN)Start, (UINT= N)Length);^M +=C2=A0 =C2=A0 =C2=A0 break;^M +=C2=A0 =C2=A0 case EfiCpuFlushTypeWriteBackInvalidate:^M +=C2=A0 =C2=A0 =C2=A0 WriteBackInvalidateDataCacheRange ((VOID *)(UINTN)Sta= rt, (UINTN)Length);^M +=C2=A0 =C2=A0 =C2=A0 break;^M +=C2=A0 =C2=A0 default:^M +=C2=A0 =C2=A0 =C2=A0 return EFI_INVALID_PARAMETER;^M +=C2=A0 }^M +^M return EFI_SUCCESS; } diff --git a/Platform/Sophgo/SG2042_EVB_Board/SG2042.fdf b/Platform/Sophgo/= SG2042_EVB_Board/SG2042.fdf index 844fc3eac0..9cbb1d3f65 100644 --- a/Platform/Sophgo/SG2042_EVB_Board/SG2042.fdf +++ b/Platform/Sophgo/SG2042_EVB_Board/SG2042.fdf @@ -77,7 +77,7 @@ INF=C2=A0 Silicon/Sophgo/SG2042Pkg/Drivers/SdHostDxe/SdHo= stDxe.inf # RISC-V Core Drivers INF=C2=A0 UefiCpuPkg/CpuTimerDxeRiscV64/CpuTimerDxeRiscV64.inf -INF=C2=A0 Silicon/Sophgo/SG2042Pkg/Override/UefiCpuPkg/CpuDxeRiscV64/CpuDx= eRiscV64.inf +INF=C2=A0 UefiCpuPkg/CpuDxeRiscV64/CpuDxeRiscV64.inf INF=C2=A0 MdeModulePkg/Universal/FaultTolerantWriteDxe/FaultTolerantWriteDx= e.inf INF=C2=A0 MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe.inf diff --git a/Platform/Sophgo/SG2042_EVB_Board/SG2042.dsc b/Platform/Sophgo/= SG2042_EVB_Board/SG2042.dsc index 51ff89678c..bc3f96e21b 100644 --- a/Platform/Sophgo/SG2042_EVB_Board/SG2042.dsc +++ b/Platform/Sophgo/SG2042_EVB_Board/SG2042.dsc @@ -389,6 +389,7 @@ [PcdsPatchableInModule] gSophgoSG2042PlatformPkgTokenSpaceGuid.PcdSG2042PhyAddrToVirAddr|0 +=C2=A0 gEfiMdePkgTokenSpaceGuid.PcdRiscVFeatureOverride|0x1 ###########################################################################= ##### # @@ -500,7 +501,7 @@ # RISC-V Core module # UefiCpuPkg/CpuTimerDxeRiscV64/CpuTimerDxeRiscV64.inf -=C2=A0 Silicon/Sophgo/SG2042Pkg/Override/UefiCpuPkg/CpuDxeRiscV64/CpuDxeRi= scV64.inf +=C2=A0 UefiCpuPkg/CpuDxeRiscV64/CpuDxeRiscV64.inf MdeModulePkg/Universal/ResetSystemRuntimeDxe/ResetSystemRuntimeDxe.inf MdeModulePkg/Universal/FaultTolerantWriteDxe/FaultTolerantWriteDxe.inf Thanks, Jingyu -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#110470): https://edk2.groups.io/g/devel/message/110470 Mute This Topic: https://groups.io/mt/102256466/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- --HkMb6UoOVRqc6oRKres2 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 31, 2023 at 05:55 PM, Dhaval Sharma wrote:
I am posting an update on behalf of Jingyu as he had trouble wi= th posting. CC'ing him here:
In summary what we have verified so far:
  1. I have verified that instructions/op codes are okay. I have also verifi= ed on Qemu that functionally it seems to be calling correct instructions. E= nsured with negative test cases that any other op codes do cause excep= tions as expected.
  2. Jingyu was able to verify the CpuFlushCpuDataCache function with this f= ramework (he had to use custom op code based on his soc implementation) on = SG2042. There is one issue that he is debugging now which is related t= o other cache instructions and he will get back with more data. P.S. SG2042= does not implement the exact same CMO opcodes but equivalent ones. So this= experiment is just an additional data point that helps verify the framewor= k and not CMO itself.
  3. In general it sounds like framework flows are alright and as long as in= structions do their job as claimed in the spec, it is lower risk.
Guess this is what we have so far. If it makes sense to everyone, we c= ould go ahead with merging with this *feature disabled by default* after Ji= ngyu provides clarity reg failures on SG2042 platform. Otherwise we can wai= t until newer Si is available where these exact instructions can be tested = and then upstreamed.
 
[From Jingyu]
I verified this CMO framework on an actual HW platform.

SW:=
edk2: https://github.com/rivosi= nc/edk2/tree/dev-rv-cmo-v7 branch: dev-rv-cmo-v7
edk2-platfor= ms: https://github.com/sophgo/edk2-platforms=   branch: sg2042-dev

HW:
Milk-V Pioneer Box, a develop= er motherboard based on SG2042 with 64-Core T-HEAD C920.

Attention:
The T-HEAD C920 implemented its own CMO Extension and = is different from the standard CMO Extension.

Test steps:
1. Modified the opcodes in RiscVasm.inc to accommodate the C920 CMO feature= .
Update the test status.
The InvalidateInstructionCacheRange and the In= validateDataCacheRange execute the same instruction "cbo.inval", but the T-= HEAD C920 executes different instructions for the two functions. Please see= the form below for details.
I replaced ".long 0x02a5000b" with "fence= .i", which solved unexpected exceptions that occurred yesterday.

 

RISC-V standard CMO Extension

C920 CMO Extension

WriteBackInvalidateDataCache

 

 

WriteBackInvalidateDataCacheRange=

cbo.flush

".long 0x02b5000b"

WriteBackDataCache

 

 

WriteBackDataCacheRange

cbo.clean

".long 0x02b5000b"

InvalidateDataCache

fence

fence

InvalidateDataCacheRange

cbo.inval

".long 0x02a5000b"

InvalidateInstructionCache=

fence.i

fence.i

InvalidateInstructionCacheRange

cbo.inval

fence.i

Now, I enable CMO in the entire edk2 phase, not just during PCIe inbo= und. No exceptions found so far. Hope to give you some reference.

Update the patches:
diff --git a/MdePkg/Include/RiscV64/RiscVasm.inc b/MdePkg/Include/Risc= V64/RiscVasm.inc
index 29de735885..c2e573eb3d 100644
--- a/MdePkg/Include/RiscV64/RiscVasm.inc
+++ b/MdePkg/Include/RiscV64/RiscVasm.inc
@@ -7,13 +7,13 @@
  */
 
 .macro RISCVCMOFLUSH
-    .word 0x25200f
+    .long 0x02b5000b^M
 .endm
 
 .macro RISCVCMOINVALIDATE
-    .word 0x05200f
+    .long 0x02a5000b^M
 .endm
 
 .macro RISCVCMOCLEAN
-    .word 0x15200f
+    .long 0x02b5000b^M
 .endm

diff --git a/MdePkg/Library/BaseCacheMaintenanceLib/RiscVCache.c b/Mde= Pkg/Library/BaseCacheMaintenanceLib/RiscVCache.c
index 5b3104afb6..ee85d0548c 100644
--- a/MdePkg/Library/BaseCacheMaintenanceLib/RiscVCache.c
+++ b/MdePkg/Library/BaseCacheMaintenanceLib/RiscVCache.c
@@ -89,7 +89,7 @@ CacheOpCacheRange (
   Start &=3D ~((UINTN)CacheLineSize - 1);
 
   DEBUG (
-    (DEBUG_INFO,
+    (DEBUG_VERBOSE,^M
      "CacheOpCacheRange:\
      Performing Cache Management Operation %d \n", Op)=
     );
@@ -163,7 +163,8 @@ InvalidateInstructionCacheRange (
   )
 {
   if (RiscVIsCMOEnabled ()) {
-    CacheOpCacheRange (Address, Length, Invld);
+    // CacheOpCacheRange (Address, Length, Invld);^M
+    InvalidateInstructionCache ();^M
   } else {
     DEBUG (
       (DEBUG_VERBOSE,

diff --git a/UefiCpuPkg/CpuDxeRiscV64/CpuDxe.c b/UefiCpuPkg/CpuDxeRisc= V64/CpuDxe.c
index 2af3b62234..824667bc87 100644
--- a/UefiCpuPkg/CpuDxeRiscV64/CpuDxe.c
+++ b/UefiCpuPkg/CpuDxeRiscV64/CpuDxe.c
@@ -9,6 +9,7 @@
 **/
 
 #include "CpuDxe.h"
+#include <Library/CacheMaintenanceLib.h>^M
 
 //
 // Global Variables
@@ -59,7 +60,7 @@ EFI_CPU_ARCH_PROTOCOL  gCpu =3D {
   CpuGetTimerValue,
   CpuSetMemoryAttributes,
   1,               =           // NumberOfTimers
-  4                 = ;          // DmaBufferAlignment
+  64                &nbs= p;         // DmaBufferAlignment^M
 };
 
 //
@@ -90,6 +91,20 @@ CpuFlushCpuDataCache (
   IN EFI_CPU_FLUSH_TYPE     FlushType
   )
 {
+  switch (FlushType) {^M
+    case EfiCpuFlushTypeWriteBack:^M
+      WriteBackDataCacheRange ((VOID *)(UINTN)Start, (= UINTN)Length);^M
+      break;^M
+    case EfiCpuFlushTypeInvalidate:^M
+      InvalidateDataCacheRange ((VOID *)(UINTN)Start, = (UINTN)Length);^M
+      break;^M
+    case EfiCpuFlushTypeWriteBackInvalidate:^M
+      WriteBackInvalidateDataCacheRange ((VOID *)(UINT= N)Start, (UINTN)Length);^M
+      break;^M
+    default:^M
+      return EFI_INVALID_PARAMETER;^M
+  }^M
+^M
   return EFI_SUCCESS;
 }

diff --git a/Platform/Sophgo/SG2042_EVB_Board/SG2042.fdf b/Platform/So= phgo/SG2042_EVB_Board/SG2042.fdf
index 844fc3eac0..9cbb1d3f65 100644
--- a/Platform/Sophgo/SG2042_EVB_Board/SG2042.fdf
+++ b/Platform/Sophgo/SG2042_EVB_Board/SG2042.fdf
@@ -77,7 +77,7 @@ INF  Silicon/Sophgo/SG2042Pkg/Drivers/SdHostDxe= /SdHostDxe.inf
 
 # RISC-V Core Drivers
 INF  UefiCpuPkg/CpuTimerDxeRiscV64/CpuTimerDxeRiscV64.inf
-INF  Silicon/Sophgo/SG2042Pkg/Override/UefiCpuPkg/CpuDxeRiscV64/= CpuDxeRiscV64.inf
+INF  UefiCpuPkg/CpuDxeRiscV64/CpuDxeRiscV64.inf
 
 INF  MdeModulePkg/Universal/FaultTolerantWriteDxe/FaultTole= rantWriteDxe.inf
 INF  MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRun= timeDxe.inf

diff --git a/Platform/Sophgo/SG2042_EVB_Board/SG2042.dsc b/Platform/So= phgo/SG2042_EVB_Board/SG2042.dsc
index 51ff89678c..bc3f96e21b 100644
--- a/Platform/Sophgo/SG2042_EVB_Board/SG2042.dsc
+++ b/Platform/Sophgo/SG2042_EVB_Board/SG2042.dsc
@@ -389,6 +389,7 @@
 
 [PcdsPatchableInModule]
   gSophgoSG2042PlatformPkgTokenSpaceGuid.PcdSG2042PhyAddrTo= VirAddr|0
+  gEfiMdePkgTokenSpaceGuid.PcdRiscVFeatureOverride|0x1
 
 ################################################################= ################
 #
@@ -500,7 +501,7 @@
   # RISC-V Core module
   #
   UefiCpuPkg/CpuTimerDxeRiscV64/CpuTimerDxeRiscV64.inf
-  Silicon/Sophgo/SG2042Pkg/Override/UefiCpuPkg/CpuDxeRiscV64/Cpu= DxeRiscV64.inf
+  UefiCpuPkg/CpuDxeRiscV64/CpuDxeRiscV64.inf
   MdeModulePkg/Universal/ResetSystemRuntimeDxe/ResetSystemR= untimeDxe.inf
 
   MdeModulePkg/Universal/FaultTolerantWriteDxe/FaultToleran= tWriteDxe.inf

Thanks,
Jingyu
_._,_._,_
Groups.io Links:

=20 You receive all messages sent to this group. =20 =20

View/Reply Online (#110470) | =20 | Mute= This Topic | New Topic
Your Subscriptio= n | Contact Group Owner | Unsubscribe [rebecca@openfw.io]

_._,_._,_
--HkMb6UoOVRqc6oRKres2--