[Patch 0/2] Convert Split tool to python

[Patch 0/2] Convert Split tool to python

[Bob Feng]

There are 2 reasons to convert Split tool from C to Python.
1. We are in the process of moving the Basetools Python code
to a separate repository. But there still are many C tools under
edk2/BaseTools. To make all Basetools be in the separate repo,
we can convert the C tools to Python tools.
2. The original Split tool is very slow. This python tool can reduce
90% time

Bob Feng (2):
  BaseTools: Convert Split tool to python
  BaseTools: Add unittest for Split tool

 BaseTools/Source/C/Split/Split.c            | 466 --------------------
 BaseTools/BinWrappers/PosixLike/Split       |  31 +-
 BaseTools/BinWrappers/WindowsLike/Split.bat |   3 +
 BaseTools/Source/C/GNUmakefile              |   1 -
 BaseTools/Source/C/Makefile                 |   1 -
 BaseTools/Source/C/Split/GNUmakefile        |  17 -
 BaseTools/Source/C/Split/Makefile           |  16 -
 BaseTools/Source/Python/Split/Split.py      | 203 +++++++++
 BaseTools/Source/Python/Split/__init__.py   |  10 +
 BaseTools/Source/Python/tests/test_split.py | 101 +++++
 10 files changed, 325 insertions(+), 524 deletions(-)
 delete mode 100644 BaseTools/Source/C/Split/Split.c
 create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
 delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
 delete mode 100644 BaseTools/Source/C/Split/Makefile
 create mode 100644 BaseTools/Source/Python/Split/Split.py
 create mode 100644 BaseTools/Source/Python/Split/__init__.py
 create mode 100644 BaseTools/Source/Python/tests/test_split.py

-- 
2.29.1.windows.1

[Patch 1/2] BaseTools: Convert Split tool to python

[Bob Feng]

There are 2 reasons to convert Split tool from C to Python.
1. We are in the process of moving the Basetools Python code
to a separate repository. But there still are many C tools under
edk2/BaseTools. To make all Basetools be in the separate repo,
we can convert the C tools to Python tools.
2. The original Split tool is very slow. This python tool can reduce
90% time.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>

---
 BaseTools/Source/C/Split/Split.c            | 466 --------------------
 BaseTools/BinWrappers/PosixLike/Split       |  31 +-
 BaseTools/BinWrappers/WindowsLike/Split.bat |   3 +
 BaseTools/Source/C/GNUmakefile              |   1 -
 BaseTools/Source/C/Makefile                 |   1 -
 BaseTools/Source/C/Split/GNUmakefile        |  17 -
 BaseTools/Source/C/Split/Makefile           |  16 -
 BaseTools/Source/Python/Split/Split.py      | 203 +++++++++
 BaseTools/Source/Python/Split/__init__.py   |  10 +
 9 files changed, 224 insertions(+), 524 deletions(-)
 delete mode 100644 BaseTools/Source/C/Split/Split.c
 create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
 delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
 delete mode 100644 BaseTools/Source/C/Split/Makefile
 create mode 100644 BaseTools/Source/Python/Split/Split.py
 create mode 100644 BaseTools/Source/Python/Split/__init__.py

diff --git a/BaseTools/Source/C/Split/Split.c b/BaseTools/Source/C/Split/Split.c
deleted file mode 100644
index be0ee124bfe0..000000000000
--- a/BaseTools/Source/C/Split/Split.c
+++ /dev/null
@@ -1,466 +0,0 @@
-/** @file
-
-  Split a file into two pieces at the request offset.
-
-Copyright (c) 1999 - 2017, Intel Corporation. All rights reserved.<BR>
-SPDX-License-Identifier: BSD-2-Clause-Patent
-
-**/
-
-// GC_TODO: fix comment to start with /*++
-#include <stdio.h>
-#include <string.h>
-#include <stdlib.h>
-#ifdef __GNUC__
-#include <unistd.h>
-#else
-#include <direct.h>
-#endif
-#include <ctype.h>
-#include "ParseInf.h"
-#include "CommonLib.h"
-#include "EfiUtilityMsgs.h"
-//
-// Utility Name
-//
-#define UTILITY_NAME  "Split"
-
-//
-// Utility version information
-//
-#define UTILITY_MAJOR_VERSION 1
-#define UTILITY_MINOR_VERSION 0
-
-void
-Version (
-  void
-  )
-/*++
-
-Routine Description:
-
-  Displays the standard utility information to SDTOUT
-
-Arguments:
-
-  None
-
-Returns:
-
-  None
-
---*/
-{
-  printf ("%s Version %d.%d Build %s\n", UTILITY_NAME, UTILITY_MAJOR_VERSION, UTILITY_MINOR_VERSION, __BUILD_VERSION);
-}
-
-void
-Usage (
-  void
-  )
-/*++
-
-Routine Description:
-
-  GC_TODO: Add function description
-
-Arguments:
-
-
-Returns:
-
-  GC_TODO: add return values
-
---*/
-{
-  Version();
-  printf ("Copyright (c) 1999-2017 Intel Corporation. All rights reserved.\n");
-  printf ("\n  SplitFile creates two Binary files either in the same directory as the current working\n");
-  printf ("  directory or in the specified directory.\n");
-  printf ("\nUsage: \n\
-   Split\n\
-     -f, --filename inputFile to split\n\
-     -s, --split VALUE the number of bytes in the first file\n\
-     [-p, --prefix OutputDir]\n\
-     [-o, --firstfile Filename1]\n\
-     [-t, --secondfile Filename2]\n\
-     [-v, --verbose]\n\
-     [--version]\n\
-     [-q, --quiet disable all messages except fatal errors]\n\
-     [-d, --debug[#]\n\
-     [-h, --help]\n");
-}
-
-EFI_STATUS
-GetSplitValue (
-  IN CONST CHAR8* SplitValueString,
-  OUT UINT64 *ReturnValue
-)
-{
-  UINT64 len = 0;
-  UINT64 base = 1;
-  UINT64 index = 0;
-  UINT64 number = 0;
-  CHAR8 lastCHAR = 0;
-  EFI_STATUS Status = EFI_SUCCESS;
-
-  if (SplitValueString != NULL){
-    len = strlen(SplitValueString);
-  }
-
-  if (len == 0) {
-    return EFI_ABORTED;
-  }
-
-  Status = AsciiStringToUint64 (SplitValueString, FALSE, ReturnValue);
-  if (!EFI_ERROR (Status)) {
-    return Status;
-  }
-
-  if (SplitValueString[0] == '0' && (SplitValueString[1] == 'x' || SplitValueString[1] == 'X')) {
-    Status = AsciiStringToUint64 (SplitValueString, TRUE, ReturnValue);
-    if (!EFI_ERROR (Status)) {
-      return Status;
-    }
-  }
-
-  lastCHAR = (CHAR8)toupper((int)SplitValueString[len - 1]);
-
-  if (lastCHAR != 'K' && lastCHAR != 'M' && lastCHAR != 'G') {
-    return STATUS_ERROR;
-  }
-
-  for (;index < len - 1; ++index) {
-    if (!isdigit((int)SplitValueString[index])) {
-      return EFI_ABORTED;
-    }
-  }
-
-  number = atol (SplitValueString);
-  if (lastCHAR == 'K')
-    base = 1024;
-  else if (lastCHAR == 'M')
-    base = 1024*1024;
-  else
-    base = 1024*1024*1024;
-
-  *ReturnValue = number*base;
-
-  return EFI_SUCCESS;
-}
-
-EFI_STATUS
-CountVerboseLevel (
-  IN CONST CHAR8* VerboseLevelString,
-  IN CONST UINT64 Length,
-  OUT UINT64 *ReturnValue
-)
-{
-  UINT64 i = 0;
-  for (;i < Length; ++i) {
-    if (VerboseLevelString[i] != 'v' && VerboseLevelString[i] != 'V') {
-      return EFI_ABORTED;
-    }
-    ++(*ReturnValue);
-  }
-
-  return EFI_SUCCESS;
-}
-
-EFI_STATUS
-CreateDir (
-  IN OUT CHAR8** FullFileName
-)
-{
-  CHAR8* temp = *FullFileName;
-  CHAR8* start = temp;
-  CHAR8  tempchar;
-  UINT64 index = 0;
-
-  for (;index < strlen(temp); ++index) {
-    if (temp[index] == '\\' || temp[index] == '/') {
-      if (temp[index + 1] != '\0') {
-        tempchar = temp[index + 1];
-        temp[index + 1] = 0;
-        if (chdir(start)) {
-          if (mkdir(start, S_IRWXU | S_IRWXG | S_IRWXO) != 0) {
-            return EFI_ABORTED;
-          }
-          chdir(start);
-        }
-        start = temp + index + 1;
-        temp[index] = '/';
-        temp[index + 1] = tempchar;
-      }
-    }
-  }
-
-  return EFI_SUCCESS;
-}
-
-int
-main (
-  int argc,
-  char*argv[]
-  )
-/*++
-
-Routine Description:
-
-  GC_TODO: Add function description
-
-Arguments:
-
-  argc  - GC_TODO: add argument description
-  ]     - GC_TODO: add argument description
-
-Returns:
-
-  GC_TODO: add return values
-
---*/
-{
-  EFI_STATUS    Status = EFI_SUCCESS;
-  INTN          ReturnStatus = STATUS_SUCCESS;
-  FILE          *In;
-  CHAR8         *InputFileName = NULL;
-  CHAR8         *OutputDir = NULL;
-  CHAR8         *OutFileName1 = NULL;
-  CHAR8         *OutFileName2 = NULL;
-  UINT64        SplitValue = (UINT64) -1;
-  FILE          *Out1 = NULL;
-  FILE          *Out2 = NULL;
-  CHAR8         *OutName1 = NULL;
-  CHAR8         *OutName2 = NULL;
-  CHAR8         *CurrentDir = NULL;
-  UINT64        Index;
-  CHAR8         CharC;
-  UINT64        DebugLevel = 0;
-  UINT64        VerboseLevel = 0;
-
-  SetUtilityName(UTILITY_NAME);
-  if (argc == 1) {
-    Usage();
-    return STATUS_ERROR;
-  }
-
-  argc --;
-  argv ++;
-
-  if ((stricmp (argv[0], "-h") == 0) || (stricmp (argv[0], "--help") == 0)) {
-    Usage();
-    return STATUS_SUCCESS;
-  }
-
-  if (stricmp (argv[0], "--version") == 0) {
-    Version();
-    return STATUS_SUCCESS;
-  }
-
-  while (argc > 0) {
-    if ((stricmp (argv[0], "-p") == 0) || (stricmp (argv[0], "--prefix") == 0)) {
-      OutputDir = argv[1];
-      if (OutputDir == NULL) {
-        Warning (NULL, 0, 0, "NO output directory specified.", NULL);
-        return STATUS_ERROR;
-      }
-      argc -= 2;
-      argv += 2;
-      continue;
-    }
-
-    if ((stricmp (argv[0], "-f") == 0) || (stricmp (argv[0], "--filename") == 0)) {
-      InputFileName = argv[1];
-      if (InputFileName == NULL) {
-        Error (NULL, 0, 0x1001, "NO Input file specified.", NULL);
-        return STATUS_ERROR;
-      }
-      argc -= 2;
-      argv += 2;
-      continue;
-    }
-
-    if ((stricmp (argv[0], "-s") == 0) || (stricmp (argv[0], "--split") == 0)) {
-      Status = GetSplitValue(argv[1], &SplitValue);
-      if (EFI_ERROR (Status)) {
-        Error (NULL, 0, 0x1003, "Input split value is not one valid integer.", NULL);
-        return STATUS_ERROR;
-      }
-      argc -= 2;
-      argv += 2;
-      continue;
-    }
-
-    if ((stricmp (argv[0], "-o") == 0) || (stricmp (argv[0], "--firstfile") == 0)) {
-      OutFileName1 = argv[1];
-      if (OutFileName1 == NULL) {
-        Warning (NULL, 0, 0, NULL, "No output file1 specified.");
-      }
-      argc -= 2;
-      argv += 2;
-      continue;
-    }
-
-    if ((stricmp (argv[0], "-t") == 0) || (stricmp (argv[0], "--secondfile") == 0)) {
-      OutFileName2 = argv[1];
-      if (OutFileName2 == NULL) {
-        Warning (NULL, 0, 0, NULL, "No output file2 specified.");
-      }
-      argc -= 2;
-      argv += 2;
-      continue;
-    }
-
-    if ((stricmp (argv[0], "-q") == 0) || (stricmp (argv[0], "--quiet") == 0)) {
-      argc --;
-      argv ++;
-      continue;
-    }
-
-    if ((strlen(argv[0]) >= 2 && argv[0][0] == '-' && (argv[0][1] == 'v' || argv[0][1] == 'V')) || (stricmp (argv[0], "--verbose") == 0)) {
-      VerboseLevel = 1;
-      if (strlen(argv[0]) > 2) {
-        Status = CountVerboseLevel (&argv[0][2], strlen(argv[0]) - 2, &VerboseLevel);
-        if (EFI_ERROR (Status)) {
-          Error (NULL, 0, 0x1003, NULL, "%s is invalid parameter!", argv[0]);
-          return STATUS_ERROR;
-        }
-      }
-
-      argc --;
-      argv ++;
-      continue;
-    }
-
-    if ((stricmp (argv[0], "-d") == 0) || (stricmp (argv[0], "--debug") == 0)) {
-      Status = AsciiStringToUint64 (argv[1], FALSE, &DebugLevel);
-      if (EFI_ERROR (Status)) {
-        Error (NULL, 0, 0x1003, "Input debug level is not one valid integrator.", NULL);
-        return STATUS_ERROR;
-      }
-      argc -= 2;
-      argv += 2;
-      continue;
-    }
-    //
-    // Don't recognize the parameter.
-    //
-    Error (NULL, 0, 0x1003, NULL, "%s is invalid parameter!", argv[0]);
-    return STATUS_ERROR;
-  }
-
-  if (InputFileName == NULL) {
-    Error (NULL, 0, 0x1001, "NO Input file specified.", NULL);
-    return STATUS_ERROR;
-  }
-
-  In = fopen (LongFilePath (InputFileName), "rb");
-  if (In == NULL) {
-    // ("Unable to open file \"%s\"\n", InputFileName);
-    Error (InputFileName, 0, 1, "File open failure", NULL);
-    return STATUS_ERROR;
-  }
-
-  if (OutFileName1 == NULL) {
-    OutName1 = (CHAR8*)malloc(strlen(InputFileName) + 16);
-    if (OutName1 == NULL) {
-      Warning (NULL, 0, 0, NULL, "Memory Allocation Fail.");
-      ReturnStatus = STATUS_ERROR;
-      goto Finish;
-    }
-    strcpy (OutName1, InputFileName);
-    strcat (OutName1, "1");
-    OutFileName1 = OutName1;
-
-  }
-  if (OutFileName2 == NULL) {
-    OutName2 = (CHAR8*)malloc(strlen(InputFileName) + 16);
-    if (OutName2 == NULL) {
-      Warning (NULL, 0, 0, NULL, "Memory Allocation Fail.");
-      ReturnStatus = STATUS_ERROR;
-      goto Finish;
-    }
-    strcpy (OutName2, InputFileName);
-    strcat (OutName2, "2");
-    OutFileName2 = OutName2;
-
-  }
-
-  if (OutputDir != NULL) {
-    //OutputDirSpecified = TRUE;
-    if (chdir(OutputDir) != 0) {
-      Warning (NULL, 0, 0, NULL, "Change dir to OutputDir Fail.");
-      ReturnStatus = STATUS_ERROR;
-      goto Finish;
-    }
-  }
-
-  CurrentDir = (CHAR8*)getcwd((CHAR8*)0, 0);
-  if (EFI_ERROR(CreateDir(&OutFileName1))) {
-      Error (OutFileName1, 0, 5, "Create Dir for File1 Fail.", NULL);
-      ReturnStatus = STATUS_ERROR;
-      goto Finish;
-  }
-  chdir(CurrentDir);
-
-  if (EFI_ERROR(CreateDir(&OutFileName2))) {
-      Error (OutFileName2, 0, 5, "Create Dir for File2 Fail.", NULL);
-      ReturnStatus = STATUS_ERROR;
-      goto Finish;
-  }
-  chdir(CurrentDir);
-  free(CurrentDir);
-
-  Out1 = fopen (LongFilePath (OutFileName1), "wb");
-  if (Out1 == NULL) {
-    // ("Unable to open file \"%s\"\n", OutFileName1);
-    Error (OutFileName1, 0, 1, "File open failure", NULL);
-    ReturnStatus = STATUS_ERROR;
-    goto Finish;
-  }
-
-  Out2 = fopen (LongFilePath (OutFileName2), "wb");
-  if (Out2 == NULL) {
-    // ("Unable to open file \"%s\"\n", OutFileName2);
-    Error (OutFileName2, 0, 1, "File open failure", NULL);
-    ReturnStatus = STATUS_ERROR;
-    goto Finish;
-  }
-
-  for (Index = 0; Index < SplitValue; Index++) {
-    CharC = (CHAR8) fgetc (In);
-    if (feof (In)) {
-      break;
-    }
-
-    fputc (CharC, Out1);
-  }
-
-  for (;;) {
-    CharC = (CHAR8) fgetc (In);
-    if (feof (In)) {
-      break;
-    }
-
-    fputc (CharC, Out2);
-  }
-
-Finish:
-  if (OutName1 != NULL) {
-    free(OutName1);
-  }
-  if (OutName2 != NULL) {
-    free(OutName2);
-  }
-  if (In != NULL) {
-    fclose (In);
-  }
-  if (Out1 != NULL) {
-    fclose (Out1);
-  }
-  if (Out2 != NULL) {
-    fclose (Out2);
-  }
-
-  return ReturnStatus;
-}
diff --git a/BaseTools/BinWrappers/PosixLike/Split b/BaseTools/BinWrappers/PosixLike/Split
index 0945d86d9209..f3770eed42b4 100755
--- a/BaseTools/BinWrappers/PosixLike/Split
+++ b/BaseTools/BinWrappers/PosixLike/Split
@@ -1,29 +1,14 @@
 #!/usr/bin/env bash
+#python `dirname $0`/RunToolFromSource.py `basename $0` $*
+
+# If a ${PYTHON_COMMAND} command is available, use it in preference to python
+if command -v ${PYTHON_COMMAND} >/dev/null 2>&1; then
+    python_exe=${PYTHON_COMMAND}
+fi
 
 full_cmd=${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028 for a discussion of why $0 is not a good choice here
 dir=$(dirname "$full_cmd")
 cmd=${full_cmd##*/}
 
-if [ -n "$WORKSPACE" ] && [ -e "$WORKSPACE/Conf/BaseToolsCBinaries" ]
-then
-  exec "$WORKSPACE/Conf/BaseToolsCBinaries/$cmd"
-elif [ -n "$WORKSPACE" ] && [ -e "$EDK_TOOLS_PATH/Source/C" ]
-then
-  if [ ! -e "$EDK_TOOLS_PATH/Source/C/bin/$cmd" ]
-  then
-    echo "BaseTools C Tool binary was not found ($cmd)"
-    echo "You may need to run:"
-    echo "  make -C $EDK_TOOLS_PATH/Source/C"
-  else
-    exec "$EDK_TOOLS_PATH/Source/C/bin/$cmd" "$@"
-  fi
-elif [ -e "$dir/../../Source/C/bin/$cmd" ]
-then
-  exec "$dir/../../Source/C/bin/$cmd" "$@"
-else
-  echo "Unable to find the real '$cmd' to run"
-  echo "This message was printed by"
-  echo "  $0"
-  exit 127
-fi
-
+export PYTHONPATH="$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}"
+exec "${python_exe:-python}" "$dir/../../Source/Python/$cmd/$cmd.py" "$@"
diff --git a/BaseTools/BinWrappers/WindowsLike/Split.bat b/BaseTools/BinWrappers/WindowsLike/Split.bat
new file mode 100644
index 000000000000..9616cd893bec
--- /dev/null
+++ b/BaseTools/BinWrappers/WindowsLike/Split.bat
@@ -0,0 +1,3 @@
+@setlocal
+@set ToolName=%~n0%
+@%PYTHON_COMMAND% %BASE_TOOLS_PATH%\Source\Python\%ToolName%\%ToolName%.py %*
diff --git a/BaseTools/Source/C/GNUmakefile b/BaseTools/Source/C/GNUmakefile
index 464f43277455..8c191e0c3817 100644
--- a/BaseTools/Source/C/GNUmakefile
+++ b/BaseTools/Source/C/GNUmakefile
@@ -55,11 +55,10 @@ APPLICATIONS = \
   GenFv \
   GenFw \
   GenSec \
   GenCrc32 \
   LzmaCompress \
-  Split \
   TianoCompress \
   VolInfo \
   DevicePath
 
 SUBDIRS := $(LIBRARIES) $(APPLICATIONS)
diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile
index e8f8abe59a79..a376d32e220e 100644
--- a/BaseTools/Source/C/Makefile
+++ b/BaseTools/Source/C/Makefile
@@ -17,11 +17,10 @@ APPLICATIONS = \
   GenFfs \
   GenFv \
   GenFw \
   GenSec \
   LzmaCompress \
-  Split \
   TianoCompress \
   VolInfo \
   DevicePath
 
 all: libs apps install
diff --git a/BaseTools/Source/C/Split/GNUmakefile b/BaseTools/Source/C/Split/GNUmakefile
deleted file mode 100644
index b3d4dff51ac1..000000000000
--- a/BaseTools/Source/C/Split/GNUmakefile
+++ /dev/null
@@ -1,17 +0,0 @@
-## @file
-# GNU/Linux makefile for 'Split' module build.
-#
-# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-#
-MAKEROOT ?= ..
-
-APPNAME = Split
-
-OBJECTS = Split.o
-
-include $(MAKEROOT)/Makefiles/app.makefile
-
-LIBS = -lCommon
-
-
diff --git a/BaseTools/Source/C/Split/Makefile b/BaseTools/Source/C/Split/Makefile
deleted file mode 100644
index 19d3e31a7624..000000000000
--- a/BaseTools/Source/C/Split/Makefile
+++ /dev/null
@@ -1,16 +0,0 @@
-## @file
-# Windows makefile for 'Split' module build.
-#
-# Copyright (c) 2007 - 2010, Intel Corporation. All rights reserved.<BR>
-# SPDX-License-Identifier: BSD-2-Clause-Patent
-#
-!INCLUDE ..\Makefiles\ms.common
-
-APPNAME = Split
-
-LIBS = $(LIB_PATH)\Common.lib
-
-OBJECTS = Split.obj
-
-!INCLUDE ..\Makefiles\ms.app
-
diff --git a/BaseTools/Source/Python/Split/Split.py b/BaseTools/Source/Python/Split/Split.py
new file mode 100644
index 000000000000..39d4ee7689d6
--- /dev/null
+++ b/BaseTools/Source/Python/Split/Split.py
@@ -0,0 +1,203 @@
+
+# @file
+#  Split a file into two pieces at the request offset.
+#
+#  Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+# Import Modules
+#
+import argparse
+import os
+import io
+import shutil
+import logging
+import sys
+import tempfile
+
+parser = argparse.ArgumentParser(description='''
+SplitFile creates two Binary files either in the same directory as the current working directory or in the specified directory.
+''')
+parser.add_argument("-f", "--filename", dest="inputfile",
+                    required=True, help="The input file to split tool.")
+parser.add_argument("-s", "--split", dest="position",
+                    required=True, help="The number of bytes in the first file. The valid format are HEX, Decimal and Decimal[KMG].")
+parser.add_argument("-p", "--prefix",  dest="output",
+                    help="The output folder.")
+parser.add_argument("-o", "--firstfile",  help="The first file name")
+parser.add_argument("-t", "--secondfile",  help="The second file name")
+parser.add_argument("--version", action="version", version='%(prog)s Version 1.0',
+                    help="Print debug information.")
+
+group = parser.add_mutually_exclusive_group()
+group.add_argument("-v", "--verbose", action="store_true",
+                   help="Print debug information.")
+group.add_argument("-q", "--quiet", action="store_true",
+                   help="Disable all messages except fatal errors")
+
+SizeDict = {
+    "K": 1024,
+    "M": 1024*1024,
+    "G": 1024*1024*1024
+}
+
+
+def GetPositionValue(position):
+    '''
+    Parse the string of the argument position and return a decimal number.
+    The valid position formats are
+    1. HEX
+    e.g. 0x1000 or 0X1000
+    2. Decimal
+    e.g. 100
+    3. Decimal[KMG]
+    e.g. 100K or 100M or 100G or 100k or 100m or 100g
+    '''
+    logger = logging.getLogger('Split')
+    PosVal = 0
+    header = position[:2].upper()
+    tailer = position[-1].upper()
+
+    try:
+        if tailer in SizeDict:
+            PosVal = int(position[:-1]) * SizeDict[tailer]
+        else:
+            if header == "0X":
+                PosVal = int(position, 16)
+            else:
+                PosVal = int(position)
+    except Exception as e:
+        logger.error(
+            "The parameter %s format is incorrect. The valid format is HEX, Decimal and Decimal[KMG]." % position)
+        raise(e)
+
+    return PosVal
+
+
+def getFileSize(filename):
+    '''
+    Read the input file and return the file size.
+    '''
+    logger = logging.getLogger('Split')
+    length = 0
+    try:
+        with open(filename, "rb") as fin:
+            fin.seek(0, io.SEEK_END)
+            length = fin.tell()
+    except Exception as e:
+        logger.error("Access file failed: %s", filename)
+        raise(e)
+
+    return length
+
+
+def splitFile(inputfile, position, outputdir=None, outputfile1=None, outputfile2=None):
+    '''
+    Split the inputfile into outputfile1 and outputfile2 from the position.
+    '''
+    logger = logging.getLogger('Split')
+
+    inputfile = os.path.abspath(inputfile)
+    workspace = os.path.dirname(inputfile)
+    if not os.path.exists(inputfile):
+        logger.error("File Not Found: %s" % inputfile)
+        raise(Exception)
+
+    if outputfile1 and outputfile2 and outputfile1 == outputfile2:
+        logger.error(
+            "The firstfile and the secondfile can't be the same: %s" % outputfile1)
+        raise(Exception)
+
+    if not outputdir:
+        outputdir = workspace
+    elif not os.path.isabs(outputdir):
+        outputdir = os.path.join(workspace, outputdir)
+
+    # Create dir for the output files
+    try:
+        if not outputfile1:
+            outputfile1 = os.path.abspath(os.path.join(
+                outputdir, "{}1".format(os.path.basename(inputfile))))
+        else:
+            outputfile1 = os.path.abspath(os.path.join(outputdir, outputfile1))
+        outputdir = os.path.dirname(outputfile1)
+        if not os.path.exists(outputdir):
+            os.makedirs(outputdir)
+
+        if not outputfile2:
+            outputfile2 = os.path.abspath(os.path.join(
+                outputdir, "{}2".format(os.path.basename(inputfile))))
+        else:
+            outputfile2 = os.path.abspath(os.path.join(outputdir, outputfile2))
+        outputdir = os.path.dirname(outputfile2)
+        if not os.path.exists(outputdir):
+            os.makedirs(outputdir)
+    except Exception as e:
+        logger.error("Can't make dir: %s" % outputdir)
+        raise(e)
+
+    if position <= 0:
+        if outputfile2 != inputfile:
+            shutil.copy2(inputfile, outputfile2)
+        with open(outputfile1, "wb") as fout:
+            fout.write(b'')
+    else:
+        inputfilesize = getFileSize(inputfile)
+        if position >= inputfilesize:
+            if outputfile1 != inputfile:
+                shutil.copy2(inputfile, outputfile1)
+            with open(outputfile2, "wb") as fout:
+                fout.write(b'')
+        else:
+            try:
+                tempdir = tempfile.mkdtemp()
+                tempfile1 = os.path.join(tempdir, "file1.bin")
+                tempfile2 = os.path.join(tempdir, "file2.bin")
+                with open(inputfile, "rb") as fin:
+                    content1 = fin.read(position)
+                    with open(tempfile1, "wb") as fout1:
+                        fout1.write(content1)
+
+                    content2 = fin.read(inputfilesize - position)
+                    with open(tempfile2, "wb") as fout2:
+                        fout2.write(content2)
+                shutil.copy2(tempfile1, outputfile1)
+                shutil.copy2(tempfile2, outputfile2)
+            except Exception as e:
+                logger.error("Split file failed")
+                raise(e)
+            finally:
+                if os.path.exists(tempdir):
+                    shutil.rmtree(tempdir)
+
+
+def main():
+    args = parser.parse_args()
+    status = 0
+
+    logger = logging.getLogger('Split')
+    if args.quiet:
+        logger.setLevel(logging.CRITICAL)
+    if args.verbose:
+        logger.setLevel(logging.DEBUG)
+
+    lh = logging.StreamHandler(sys.stdout)
+    lf = logging.Formatter("%(levelname)-8s: %(message)s")
+    lh.setFormatter(lf)
+    logger.addHandler(lh)
+
+    try:
+        position = GetPositionValue(args.position)
+        splitFile(args.inputfile, position, args.output,
+                  args.firstfile, args.secondfile)
+    except Exception as e:
+        status = 1
+
+    return status
+
+
+if __name__ == "__main__":
+    exit(main())
diff --git a/BaseTools/Source/Python/Split/__init__.py b/BaseTools/Source/Python/Split/__init__.py
new file mode 100644
index 000000000000..8f4daf86e84a
--- /dev/null
+++ b/BaseTools/Source/Python/Split/__init__.py
@@ -0,0 +1,10 @@
+# @file
+#  Split a file into two pieces at the request offset.
+#
+#  Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+# Import Modules
-- 
2.29.1.windows.1

[Patch 2/2] BaseTools: Add unittest for Split tool

[Bob Feng]

This patch is to add the unit test for Split python tool

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>

---
 BaseTools/Source/Python/tests/test_split.py | 101 ++++++++++++++++++++
 1 file changed, 101 insertions(+)
 create mode 100644 BaseTools/Source/Python/tests/test_split.py

diff --git a/BaseTools/Source/Python/tests/test_split.py b/BaseTools/Source/Python/tests/test_split.py
new file mode 100644
index 000000000000..0d44946d7010
--- /dev/null
+++ b/BaseTools/Source/Python/tests/test_split.py
@@ -0,0 +1,101 @@
+import unittest
+import tempfile
+import os
+import shutil
+import Split.Split as sp
+import struct as st
+
+
+class TestSplit(unittest.TestCase):
+    def setUp(self):
+        self.WORKSPACE = tempfile.mkdtemp()
+        self.binary_file = os.path.join(self.WORKSPACE, "Binary.bin")
+        self.create_inputfile()
+
+    def tearDown(self):
+        if os.path.exists(self.WORKSPACE):
+            shutil.rmtree(self.WORKSPACE)
+
+    def test_splitFile_position(self):
+        position = [-1, 0, 256, 512, 700, 1024, 2048]
+        result = [(0, 1024), (0, 1024), (256, 768),
+                  (512, 512), (700, 324), (1024, 0), (1024, 0)]
+        for index, po in enumerate(position):
+            try:
+                sp.splitFile(self.binary_file, po)
+            except Exception as e:
+                self.assertTrue(False, msg="splitFile function error")
+
+            output1 = os.path.join(self.WORKSPACE, "Binary.bin1")
+            output2 = os.path.join(self.WORKSPACE, "Binary.bin2")
+            with open(output1, "rb") as f1:
+                size1 = len(f1.read())
+            with open(output2, "rb") as f2:
+                size2 = len(f2.read())
+
+            ex_result = result[index]
+            self.assertEqual(size1, ex_result[0])
+            self.assertEqual(size2, ex_result[1])
+
+    def create_inputfile(self):
+        with open(self.binary_file, "wb") as fout:
+            for i in range(512):
+                fout.write(st.pack("<H", i))
+
+    def test_splitFile_outputfile(self):
+        output = [None, "Binary.bin", "Binary1.bin", r"output/Binary1.bin",
+                  os.path.join(self.WORKSPACE, r"output/Binary1.bin")]
+        for o in output:
+            try:
+                sp.splitFile(self.binary_file, 123, outputfile1=o)
+            except Exception as e:
+                self.assertTrue(False, msg="splitFile function error")
+            if o is None:
+                self.assertTrue(os.path.exists(
+                    os.path.join(self.WORKSPACE, "Binary.bin1")))
+            else:
+                if os.path.isabs(o):
+                    self.assertTrue(os.path.exists(o))
+                else:
+                    self.assertTrue(os.path.exists(
+                        os.path.join(self.WORKSPACE, o)))
+            self.create_inputfile()
+
+            try:
+                sp.splitFile(self.binary_file, 123, outputfile2=o)
+            except Exception as e:
+                self.assertTrue(False, msg="splitFile function error")
+            if o is None:
+                self.assertTrue(os.path.exists(
+                    os.path.join(self.WORKSPACE, "Binary.bin2")))
+            else:
+                if os.path.isabs(o):
+                    self.assertTrue(os.path.exists(o))
+                else:
+                    self.assertTrue(os.path.exists(
+                        os.path.join(self.WORKSPACE, o)))
+            self.create_inputfile()
+
+    def test_splitFile_outputfolder(self):
+        outputfolder = [None, "output", r"output1/output2",
+                        os.path.join(self.WORKSPACE, "output")]
+        for o in outputfolder:
+            try:
+                sp.splitFile(self.binary_file, 123, outputdir=o)
+            except Exception as e:
+                self.assertTrue(False, msg="splitFile function error")
+
+            if o is None:
+                self.assertTrue(os.path.exists(
+                    os.path.join(self.WORKSPACE, "Binary.bin1")))
+            else:
+                if os.path.isabs(o):
+                    self.assertTrue(os.path.exists(
+                        os.path.join(o, "Binary.bin1")))
+                else:
+                    self.assertTrue(os.path.exists(
+                        os.path.join(self.WORKSPACE, o, "Binary.bin1")))
+
+
+if __name__ == '__main__':
+    unittest.main()
-- 
2.29.1.windows.1

回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[gaoliming]

Bob:
  Have you plan to convert all C tools to Python tools? Or only some ones?
Can you share your planning for the changes in BaseTools?
  
  And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest. 

Thanks
Liming
> -----邮件原件-----
> 发件人: bounce+27952+70232+4905953+8761045@groups.io
> <bounce+27952+70232+4905953+8761045@groups.io> 代表 Bob Feng
> 发送时间: 2021年1月13日 17:56
> 收件人: devel@edk2.groups.io
> 主题: [edk2-devel] [Patch 0/2] Convert Split tool to python
> 
> There are 2 reasons to convert Split tool from C to Python.
> 
> 1. We are in the process of moving the Basetools Python code
> 
> to a separate repository. But there still are many C tools under
> 
> edk2/BaseTools. To make all Basetools be in the separate repo,
> 
> we can convert the C tools to Python tools.
> 
> 2. The original Split tool is very slow. This python tool can reduce
> 
> 90% time
> 
> Bob Feng (2):
>   BaseTools: Convert Split tool to python
>   BaseTools: Add unittest for Split tool
> 
>  BaseTools/Source/C/Split/Split.c            | 466 --------------------
>  BaseTools/BinWrappers/PosixLike/Split       |  31 +-
>  BaseTools/BinWrappers/WindowsLike/Split.bat |   3 +
>  BaseTools/Source/C/GNUmakefile              |   1 -
>  BaseTools/Source/C/Makefile                 |   1 -
>  BaseTools/Source/C/Split/GNUmakefile        |  17 -
>  BaseTools/Source/C/Split/Makefile           |  16 -
>  BaseTools/Source/Python/Split/Split.py      | 203 +++++++++
>  BaseTools/Source/Python/Split/__init__.py   |  10 +
>  BaseTools/Source/Python/tests/test_split.py | 101 +++++
>  10 files changed, 325 insertions(+), 524 deletions(-)
>  delete mode 100644 BaseTools/Source/C/Split/Split.c
>  create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
>  delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
>  delete mode 100644 BaseTools/Source/C/Split/Makefile
>  create mode 100644 BaseTools/Source/Python/Split/Split.py
>  create mode 100644 BaseTools/Source/Python/Split/__init__.py
>  create mode 100644 BaseTools/Source/Python/tests/test_split.py
> 
> --
> 2.29.1.windows.1
> 
> 
> 
> -=-=-=-=-=-=
> Groups.io Links: You receive all messages sent to this group.
> View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
> Mute This Topic: https://groups.io/mt/79647273/4905953
> Group Owner: devel+owner@edk2.groups.io
> Unsubscribe: https://edk2.groups.io/g/devel/unsub
> [gaoliming@byosoft.com.cn]
> -=-=-=-=-=-=
> 

Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:

> 
> Bob:
> Have you plan to convert all C tools to Python tools? Or only some ones?

I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048

> 
> Can you share your planning for the changes in BaseTools?

Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.

> 
> And, I see you also add unittest for new split tool. Can you introduce the
> 
> unittest framework for python tool? If so, new added tool can follow the
> same way to add unittest.

The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

> 
> 
> Thanks
> Liming
> 
>> -----邮件原件-----
>> 发件人: bounce+27952+70232+4905953+8761045@groups.io
>> <bounce+27952+70232+4905953+8761045@groups.io> 代表 Bob Feng
>> 发送时间: 2021年1月13日 17:56
>> 收件人: devel@edk2.groups.io
>> 主题: [edk2-devel] [Patch 0/2] Convert Split tool to python
>> 
>> There are 2 reasons to convert Split tool from C to Python.
>> 
>> 1. We are in the process of moving the Basetools Python code
>> 
>> to a separate repository. But there still are many C tools under
>> 
>> edk2/BaseTools. To make all Basetools be in the separate repo,
>> 
>> we can convert the C tools to Python tools.
>> 
>> 2. The original Split tool is very slow. This python tool can reduce
>> 
>> 90% time
>> 
>> Bob Feng (2):
>> BaseTools: Convert Split tool to python
>> BaseTools: Add unittest for Split tool
>> 
>> BaseTools/Source/C/Split/Split.c | 466 --------------------
>> BaseTools/BinWrappers/PosixLike/Split | 31 +-
>> BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
>> BaseTools/Source/C/GNUmakefile | 1 -
>> BaseTools/Source/C/Makefile | 1 -
>> BaseTools/Source/C/Split/GNUmakefile | 17 -
>> BaseTools/Source/C/Split/Makefile | 16 -
>> BaseTools/Source/Python/Split/Split.py | 203 +++++++++
>> BaseTools/Source/Python/Split/__init__.py | 10 +
>> BaseTools/Source/Python/tests/test_split.py | 101 +++++
>> 10 files changed, 325 insertions(+), 524 deletions(-)
>> delete mode 100644 BaseTools/Source/C/Split/Split.c
>> create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
>> delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
>> delete mode 100644 BaseTools/Source/C/Split/Makefile
>> create mode 100644 BaseTools/Source/Python/Split/Split.py
>> create mode 100644 BaseTools/Source/Python/Split/__init__.py
>> create mode 100644 BaseTools/Source/Python/tests/test_split.py
>> 
>> --
>> 2.29.1.windows.1
>> 
>> 
>> 
>> -=-=-=-=-=-=
>> Groups.io Links: You receive all messages sent to this group.
>> View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
>> Mute This Topic: https://groups.io/mt/79647273/4905953
>> Group Owner: devel+owner@edk2.groups.io
>> Unsubscribe: https://edk2.groups.io/g/devel/unsub
>> [gaoliming@byosoft.com.cn]
>> -=-=-=-=-=-=
> 
>

[-- Attachment #2: Type: text/html, Size: 5196 bytes --]

回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[gaoliming]

[-- Attachment #1: Type: text/plain, Size: 5328 bytes --]

Bob:

 Thanks for your planning sharing. For BaseTools test, I still have some questions. 

 

1.      For test framework, is it the incremental test framework or the functionality test framework or both? 

2.      For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory. 

 

Thanks

Liming

发件人: tosubbounce+27952+0+0+0@groups.io <tosubbounce+27952+0+0+0@groups.io> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn>; devel@edk2.groups.io
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

 

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:

Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?

I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.

https://edk2.groups.io/g/devel/topic/76572200#64993 

https://edk2.groups.io/g/rfc/topic/74009714#270 

https://edk2.groups.io/g/devel/topic/73069134#58048

Can you share your planning for the changes in BaseTools?

Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.

And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.

The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html 
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected. 

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/ 


Thanks
Liming

-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io <mailto:bounce+27952+70232+4905953+8761045@groups.io> 
<bounce+27952+70232+4905953+8761045@groups.io <mailto:bounce+27952+70232+4905953+8761045@groups.io> > 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> 
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=


[-- Attachment #2: Type: text/html, Size: 13418 bytes --]

Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 6016 bytes --]

Hi Liming,

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.
2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

Is there any other comments before I send the patch V2?

Thanks,
Bob

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 Thanks for your planning sharing. For BaseTools test, I still have some questions.


1.     For test framework, is it the incremental test framework or the functionality test framework or both?

2.     For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory.

Thanks
Liming
发件人: tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io> <tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io>> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:
Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?
I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048
Can you share your planning for the changes in BaseTools?
Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.
And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.
The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

Thanks
Liming
-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>
<bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io>
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=


[-- Attachment #2: Type: text/html, Size: 16496 bytes --]

Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 6534 bytes --]

Liming,

I have sent the patch V3 that is to move the unit test to tests/Split folder.

Do you have any other concerns about this patch?

Thanks,
Bob

From: Feng, Bob C
Sent: Friday, January 15, 2021 9:50 AM
To: devel@edk2.groups.io; gaoliming@byosoft.com.cn
Cc: Feng, Bob C <bob.c.feng@intel.com>
Subject: RE: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Hi Liming,

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.
2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

Is there any other comments before I send the patch V2?

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 Thanks for your planning sharing. For BaseTools test, I still have some questions.


1.     For test framework, is it the incremental test framework or the functionality test framework or both?

2.     For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory.

Thanks
Liming
发件人: tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io> <tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io>> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:
Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?
I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048
Can you share your planning for the changes in BaseTools?
Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.
And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.
The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

Thanks
Liming
-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>
<bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io>
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=


[-- Attachment #2: Type: text/html, Size: 18772 bytes --]

回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[gaoliming]

[-- Attachment #1: Type: text/plain, Size: 7280 bytes --]

Bob:

 The code logic is fine. This is a big change. I suggest to update tool version from 1.0 to 2.0. With this change, Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>

 

Thanks

Liming

发件人: bounce+27952+70509+4905953+8761045@groups.io <bounce+27952+70509+4905953+8761045@groups.io> 代表 Bob Feng
发送时间: 2021年1月19日 7:57
收件人: devel@edk2.groups.io; gaoliming@byosoft.com.cn
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

 

Liming,

 

I have sent the patch V3 that is to move the unit test to tests/Split folder.

 

Do you have any other concerns about this patch?

 

Thanks,

Bob

 

From: Feng, Bob C 
Sent: Friday, January 15, 2021 9:50 AM
To: devel@edk2.groups.io <mailto:devel@edk2.groups.io> ; gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn> 
Cc: Feng, Bob C <bob.c.feng@intel.com <mailto:bob.c.feng@intel.com> >
Subject: RE: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

 

Hi Liming,

 

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.

2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

 

Is there any other comments before I send the patch V2?

 

Thanks,

Bob

 

From: devel@edk2.groups.io <mailto:devel@edk2.groups.io>  <devel@edk2.groups.io <mailto:devel@edk2.groups.io> > On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com <mailto:bob.c.feng@intel.com> >; devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

 

Bob:

 Thanks for your planning sharing. For BaseTools test, I still have some questions. 

 

1.      For test framework, is it the incremental test framework or the functionality test framework or both? 

2.      For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory. 

 

Thanks

Liming

发件人: tosubbounce+27952+0+0+0@groups.io <mailto:tosubbounce+27952+0+0+0@groups.io>  <tosubbounce+27952+0+0+0@groups.io <mailto:tosubbounce+27952+0+0+0@groups.io> > 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn> >; devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

 

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:

Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?

I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.

https://edk2.groups.io/g/devel/topic/76572200#64993 

https://edk2.groups.io/g/rfc/topic/74009714#270 

https://edk2.groups.io/g/devel/topic/73069134#58048

Can you share your planning for the changes in BaseTools?

Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.

And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.

The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html 
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected. 

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/ 


Thanks
Liming

-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io <mailto:bounce+27952+70232+4905953+8761045@groups.io> 
<bounce+27952+70232+4905953+8761045@groups.io <mailto:bounce+27952+70232+4905953+8761045@groups.io> > 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> 
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=




[-- Attachment #2: Type: text/html, Size: 20973 bytes --]

Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 7643 bytes --]

That’s is good suggestion.

Thanks,
Bob

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of gaoliming
Sent: Tuesday, January 19, 2021 9:10 AM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 The code logic is fine. This is a big change. I suggest to update tool version from 1.0 to 2.0. With this change, Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>

Thanks
Liming
发件人: bounce+27952+70509+4905953+8761045@groups.io<mailto:bounce+27952+70509+4905953+8761045@groups.io> <bounce+27952+70509+4905953+8761045@groups.io<mailto:bounce+27952+70509+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月19日 7:57
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Liming,

I have sent the patch V3 that is to move the unit test to tests/Split folder.

Do you have any other concerns about this patch?

Thanks,
Bob

From: Feng, Bob C
Sent: Friday, January 15, 2021 9:50 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: RE: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Hi Liming,

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.
2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

Is there any other comments before I send the patch V2?

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 Thanks for your planning sharing. For BaseTools test, I still have some questions.


1.     For test framework, is it the incremental test framework or the functionality test framework or both?

2.     For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory.

Thanks
Liming
发件人: tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io> <tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io>> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:
Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?
I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048
Can you share your planning for the changes in BaseTools?
Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.
And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.
The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

Thanks
Liming
-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>
<bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io>
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=


[-- Attachment #2: Type: text/html, Size: 24731 bytes --]

Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 8215 bytes --]

A reminder that if the user has the windows bat script that calls Split in it，it needs to change to "call Split" because Split will be a bat script but not an executable file.

Thanks,
Bob

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob Feng
Sent: Tuesday, January 19, 2021 9:29 AM
To: devel@edk2.groups.io; gaoliming@byosoft.com.cn
Subject: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

That’s is good suggestion.

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of gaoliming
Sent: Tuesday, January 19, 2021 9:10 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 The code logic is fine. This is a big change. I suggest to update tool version from 1.0 to 2.0. With this change, Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>

Thanks
Liming
发件人: bounce+27952+70509+4905953+8761045@groups.io<mailto:bounce+27952+70509+4905953+8761045@groups.io> <bounce+27952+70509+4905953+8761045@groups.io<mailto:bounce+27952+70509+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月19日 7:57
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Liming,

I have sent the patch V3 that is to move the unit test to tests/Split folder.

Do you have any other concerns about this patch?

Thanks,
Bob

From: Feng, Bob C
Sent: Friday, January 15, 2021 9:50 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: RE: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Hi Liming,

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.
2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

Is there any other comments before I send the patch V2?

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 Thanks for your planning sharing. For BaseTools test, I still have some questions.


1.      For test framework, is it the incremental test framework or the functionality test framework or both?

2.      For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory.

Thanks
Liming
发件人: tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io> <tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io>> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:
Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?
I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048
Can you share your planning for the changes in BaseTools?
Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.
And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.
The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

Thanks
Liming
-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>
<bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io>
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=


[-- Attachment #2: Type: text/html, Size: 26691 bytes --]

Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 8690 bytes --]

A PR is created. https://github.com/tianocore/edk2/pull/1376

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob Feng
Sent: Tuesday, January 19, 2021 9:58 PM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>; gaoliming@byosoft.com.cn
Subject: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

A reminder that if the user has the windows bat script that calls Split in it，it needs to change to "call Split" because Split will be a bat script but not an executable file.

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Bob Feng
Sent: Tuesday, January 19, 2021 9:29 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
Subject: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

That’s is good suggestion.

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of gaoliming
Sent: Tuesday, January 19, 2021 9:10 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 The code logic is fine. This is a big change. I suggest to update tool version from 1.0 to 2.0. With this change, Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>

Thanks
Liming
发件人: bounce+27952+70509+4905953+8761045@groups.io<mailto:bounce+27952+70509+4905953+8761045@groups.io> <bounce+27952+70509+4905953+8761045@groups.io<mailto:bounce+27952+70509+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月19日 7:57
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Liming,

I have sent the patch V3 that is to move the unit test to tests/Split folder.

Do you have any other concerns about this patch?

Thanks,
Bob

From: Feng, Bob C
Sent: Friday, January 15, 2021 9:50 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: RE: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Hi Liming,

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.
2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

Is there any other comments before I send the patch V2?

Thanks,
Bob

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 Thanks for your planning sharing. For BaseTools test, I still have some questions.


1.     For test framework, is it the incremental test framework or the functionality test framework or both?

2.     For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory.

Thanks
Liming
发件人: tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io> <tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io>> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:
Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?
I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048
Can you share your planning for the changes in BaseTools?
Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.
And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.
The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

Thanks
Liming
-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>
<bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io>
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=


[-- Attachment #2: Type: text/html, Size: 28102 bytes --]