From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mx.groups.io with SMTP id smtpd.web10.133.1649188825827398065 for ; Tue, 05 Apr 2022 13:00:26 -0700 Authentication-Results: mx.groups.io; dkim=fail reason="unable to parse pub key" header.i=@intel.com header.s=intel header.b=UC9MgLf4; spf=pass (domain: intel.com, ip: 192.55.52.151, mailfrom: artem.khakimov@intel.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1649188832; x=1680724832; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=E3VJ995s7KHPvQU90g2b03IqyOJI+V/ulQiwA0ccvGE=; b=UC9MgLf4C8pwrbpZQsT2i2BX3XsfccKJ+qg0SF7zWkFsI0MIHn5nIkdJ uvXTyGOgcmKA67RvbrzZoyUIWmmLFUyWTF265lZq2mvLn9bO/tm4QMoJv 3yY8PSCelIU5EucsvXMdWeGJwzK2El9fUWRYarzCCGBWlvOR93PMDXkpG Gwqt0ajBrdippL7AgG1Fuf6CgNyNWtel/UVUTwgBsMX8ODC5tGcKkoEvc g16TCBiJj8zGujtEjJfZwpiK7H3NbOHUoLTdrRRgCT7Z6RIcOHe0l/I1d l+ywjL9R8j/zxAWQctIssF2VQHRJM212aacFJSxkfZI+bGOAQMsErVB13 A==; X-IronPort-AV: E=McAfee;i="6200,9189,10308"; a="241438054" X-IronPort-AV: E=Sophos;i="5.90,238,1643702400"; d="scan'208";a="241438054" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2022 13:00:31 -0700 X-IronPort-AV: E=Sophos;i="5.90,238,1643702400"; d="scan'208";a="570061651" Received: from plxcad1029.pdx.intel.com ([10.39.237.164]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2022 13:00:30 -0700 From: artem.khakimov@intel.com To: devel@edk2.groups.io Cc: Artem Khakimov , Bob Feng , Olga Artemeva , Vadim Mikheev , Alexander Larionov Subject: [PATCH] BaseTools: Add python version of GenCrc32 Date: Tue, 5 Apr 2022 12:59:31 -0700 Message-Id: <86c94d6a11c45be6173bfd185d87158e08569746.1649188308.git.artem.khakimov@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable From: Artem Khakimov Add GenCrc32 python tool, add tests, switch BinWrappers and BinPipWrappers to GenCrc32.py Cc: Bob Feng Cc: Olga Artemeva Cc: Vadim Mikheev Cc: Alexander Larionov Signed-off-by: Artem Khakimov --- BaseTools/BinPipWrappers/PosixLike/GenCrc32 | 41 ++-- .../BinPipWrappers/WindowsLike/GenCrc32.bat | 3 + BaseTools/BinWrappers/PosixLike/GenCrc32 | 43 ++-- .../BinWrappers/WindowsLike/GenCrc32.bat | 4 + BaseTools/Source/Python/GenCrc32/GenCrc32.py | 185 ++++++++++++++++++ BaseTools/Tests/PythonToolsTests.py | 2 + BaseTools/Tests/TestGenCrc32.py | 64 ++++++ 7 files changed, 284 insertions(+), 58 deletions(-) create mode 100644 BaseTools/BinPipWrappers/WindowsLike/GenCrc32.bat create mode 100644 BaseTools/BinWrappers/WindowsLike/GenCrc32.bat create mode 100644 BaseTools/Source/Python/GenCrc32/GenCrc32.py create mode 100644 BaseTools/Tests/TestGenCrc32.py diff --git a/BaseTools/BinPipWrappers/PosixLike/GenCrc32 b/BaseTools/BinPip= Wrappers/PosixLike/GenCrc32 index 0945d86d92..14e8d8af71 100755 --- a/BaseTools/BinPipWrappers/PosixLike/GenCrc32 +++ b/BaseTools/BinPipWrappers/PosixLike/GenCrc32 @@ -1,29 +1,12 @@ -#!/usr/bin/env bash - -full_cmd=3D${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028= for a discussion of why $0 is not a good choice here -dir=3D$(dirname "$full_cmd") -cmd=3D${full_cmd##*/} - -if [ -n "$WORKSPACE" ] && [ -e "$WORKSPACE/Conf/BaseToolsCBinaries" ] -then - exec "$WORKSPACE/Conf/BaseToolsCBinaries/$cmd" -elif [ -n "$WORKSPACE" ] && [ -e "$EDK_TOOLS_PATH/Source/C" ] -then - if [ ! -e "$EDK_TOOLS_PATH/Source/C/bin/$cmd" ] - then - echo "BaseTools C Tool binary was not found ($cmd)" - echo "You may need to run:" - echo " make -C $EDK_TOOLS_PATH/Source/C" - else - exec "$EDK_TOOLS_PATH/Source/C/bin/$cmd" "$@" - fi -elif [ -e "$dir/../../Source/C/bin/$cmd" ] -then - exec "$dir/../../Source/C/bin/$cmd" "$@" -else - echo "Unable to find the real '$cmd' to run" - echo "This message was printed by" - echo " $0" - exit 127 -fi - +#!/usr/bin/env bash +#python `dirname $0`/RunToolFromSource.py `basename $0` $* + +# If a ${PYTHON_COMMAND} command is available, use it in preference to pyt= hon +if command -v ${PYTHON_COMMAND} >/dev/null 2>&1; then + python_exe=3D${PYTHON_COMMAND} +fi + +full_cmd=3D${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028= for a discussion of why $0 is not a good choice here +cmd=3D${full_cmd##*/} + +exec "${python_exe:-python}" -m edk2basetools.$cmd "$@" diff --git a/BaseTools/BinPipWrappers/WindowsLike/GenCrc32.bat b/BaseTools/= BinPipWrappers/WindowsLike/GenCrc32.bat new file mode 100644 index 0000000000..d347d64844 --- /dev/null +++ b/BaseTools/BinPipWrappers/WindowsLike/GenCrc32.bat @@ -0,0 +1,3 @@ +@setlocal +@set ToolName=3D%~n0% +@%PYTHON_COMMAND% -m edk2basetools.%ToolName%.%ToolName% %* diff --git a/BaseTools/BinWrappers/PosixLike/GenCrc32 b/BaseTools/BinWrappe= rs/PosixLike/GenCrc32 index 0945d86d92..39483a9869 100755 --- a/BaseTools/BinWrappers/PosixLike/GenCrc32 +++ b/BaseTools/BinWrappers/PosixLike/GenCrc32 @@ -1,29 +1,14 @@ -#!/usr/bin/env bash - -full_cmd=3D${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028= for a discussion of why $0 is not a good choice here -dir=3D$(dirname "$full_cmd") -cmd=3D${full_cmd##*/} - -if [ -n "$WORKSPACE" ] && [ -e "$WORKSPACE/Conf/BaseToolsCBinaries" ] -then - exec "$WORKSPACE/Conf/BaseToolsCBinaries/$cmd" -elif [ -n "$WORKSPACE" ] && [ -e "$EDK_TOOLS_PATH/Source/C" ] -then - if [ ! -e "$EDK_TOOLS_PATH/Source/C/bin/$cmd" ] - then - echo "BaseTools C Tool binary was not found ($cmd)" - echo "You may need to run:" - echo " make -C $EDK_TOOLS_PATH/Source/C" - else - exec "$EDK_TOOLS_PATH/Source/C/bin/$cmd" "$@" - fi -elif [ -e "$dir/../../Source/C/bin/$cmd" ] -then - exec "$dir/../../Source/C/bin/$cmd" "$@" -else - echo "Unable to find the real '$cmd' to run" - echo "This message was printed by" - echo " $0" - exit 127 -fi - +#!/usr/bin/env bash +#python `dirname $0`/RunToolFromSource.py `basename $0` $* + +# If a ${PYTHON_COMMAND} command is available, use it in preference to pyt= hon +if command -v ${PYTHON_COMMAND} >/dev/null 2>&1; then + python_exe=3D${PYTHON_COMMAND} +fi + +full_cmd=3D${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028= for a discussion of why $0 is not a good choice here +dir=3D$(dirname "$full_cmd") +cmd=3D${full_cmd##*/} + +export PYTHONPATH=3D"$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}" +exec "${python_exe:-python}" -m $cmd.$cmd "$@" diff --git a/BaseTools/BinWrappers/WindowsLike/GenCrc32.bat b/BaseTools/Bin= Wrappers/WindowsLike/GenCrc32.bat new file mode 100644 index 0000000000..f43dba81f1 --- /dev/null +++ b/BaseTools/BinWrappers/WindowsLike/GenCrc32.bat @@ -0,0 +1,4 @@ +@setlocal +@set ToolName=3D%~n0% +@set PYTHONPATH=3D%PYTHONPATH%;%BASE_TOOLS_PATH%\Source\Python +@%PYTHON_COMMAND% -m %ToolName%.%ToolName% %* diff --git a/BaseTools/Source/Python/GenCrc32/GenCrc32.py b/BaseTools/Sourc= e/Python/GenCrc32/GenCrc32.py new file mode 100644 index 0000000000..ef1f2a5b0a --- /dev/null +++ b/BaseTools/Source/Python/GenCrc32/GenCrc32.py @@ -0,0 +1,185 @@ +# +# @file +# Calculate Crc32 value and Verify Crc32 value for input data. +# +# Copyright (c) 2022, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +import sys +import argparse +import zlib + +__prog__ =3D 'GenCrc32' +__description__ =3D 'Tool has two modes of operation: Encode and Decode. \ + Encode calculates CRC32 of a file and appends checksum \ + to the beginning of file. Decode checks stored CRC32 \ + and removes it from file.' +__version__ =3D '0.3' + +MSG_INFO =3D 0 +MSG_ERROR =3D 1 +MSG_DEBUG =3D 2 + +STATUS_SUCCESS =3D 0 +STATUS_ERROR =3D 2 + + +def log(args, message_type, message): + if (((args.verbose and + (message_type =3D=3D MSG_INFO or + message_type =3D=3D MSG_ERROR)) or + (args.debug and + (message_type =3D=3D MSG_DEBUG or + message_type =3D=3D MSG_INFO or + message_type =3D=3D MSG_ERROR))) and + not args.quiet): + print("%s: %s" % (__prog__, message)) + + +# replacement for int.from_bytes() +# for compatability with python 2 +def uint32_from_bytes(data): + barray =3D bytearray(data) + return ( + (barray[3] << 24) | + (barray[2] << 16) | + (barray[1] << 8) | + (barray[0] << 0)) + + +# replacement for int.to_bytes() +# for compatability with python 2 +def uint32_to_bytes(uint): + return bytearray([ + (uint >> 0) & 0xff, + (uint >> 8) & 0xff, + (uint >> 16) & 0xff, + (uint >> 24) & 0xff]) + + +def parse_args(): + parser =3D argparse.ArgumentParser(prog=3D__prog__, + description=3D__description__, + conflict_handler=3D'resolve') + parser.add_argument('--version', + action=3D'version', + version=3D__version__, + help=3D'Show version number and exit') + parser.add_argument('--debug', + type=3Dint, + metavar=3D'[0-9]', + choices=3Drange(0, 10), + default=3D0, + help=3D'Output DEBUG statements') + logging =3D parser.add_mutually_exclusive_group(required=3DFalse) + logging.add_argument('-v', + '--verbose', + action=3D'store_true', + help=3D'Print informational statements') + logging.add_argument('-q', + '--quiet', + action=3D'store_true', + help=3D'Returns the exit code, \ + error messages will be displayed') + operation =3D parser.add_mutually_exclusive_group(required=3DTrue) + operation.add_argument('-e', + '--encode', + action=3D'store_true', + help=3D'Calculate CRC32 value for the input fil= e') + operation.add_argument('-d', + '--decode', + action=3D'store_true', + help=3D'Verify CRC32 value for the input file') + parser.add_argument('-o', + '--output', + dest=3D'output_file', + type=3Dstr, + help=3D'Output file name', + required=3DTrue) + parser.add_argument(dest=3D'input_file', + type=3Dargparse.FileType('rb'), + help=3D'Input file name') + return parser.parse_args() + + +def main(): + args =3D parse_args() + log(args, MSG_DEBUG, "Input file name: %s" % args.input_file.name) + log(args, MSG_DEBUG, "Output file name: %s" % args.output_file) + if args.encode: + log(args, MSG_INFO, "Operation: Encode") + if args.decode: + log(args, MSG_INFO, "Operation: Decode") + + # Reading input file to buffer + data =3D bytearray() + header =3D bytearray() + log(args, MSG_INFO, "Reading input file") + try: + if args.decode: + header =3D args.input_file.read(4) + data =3D args.input_file.read() + args.input_file.close() + except MemoryError as e: + log(args, MSG_ERROR, ("Error reading input file, " + "cannot allocate memory")) + sys.exit(STATUS_ERROR) + log(args, MSG_INFO, ("Input file length: %d bytes " + "(%d bytes crc32 header and %d bytes data)") % + ((len(header) + len(data)), len(header), len(data))) + + # Calculating CRC32 + log(args, MSG_INFO, "Calculating CRC32") + data_crc32 =3D zlib.crc32(data) & 0xffffffff + if args.decode: + if len(header) >=3D 4: + header_crc32 =3D uint32_from_bytes(header) + else: + log(args, MSG_ERROR, ("Input file size is smaller " + "than 4 bytes, invalid file format")) + sys.exit(STATUS_ERROR) + log(args, MSG_INFO, "CRC32 in input file header is %s" % + hex(header_crc32)) + log(args, MSG_INFO, "CRC32 of input file data is %s" % + hex(data_crc32)) + + # Perform selected operation by modifying header/data + + # Encoding + if args.encode: + log(args, MSG_DEBUG, "Encoding header") + header =3D uint32_to_bytes(data_crc32) + + # Verification and decoding + if args.decode: + log(args, MSG_INFO, "Verifying checksum") + header =3D bytearray() + if data_crc32 =3D=3D header_crc32: + log(args, MSG_INFO, "Checksum verification succeeded") + else: + log(args, MSG_ERROR, ("Checksum verification failed, " + "CRC32 mismatch")) + sys.exit(STATUS_ERROR) + + # Write output file + log(args, MSG_INFO, "Writing output file") + try: + with open(args.output_file, 'wb') as output_file: + output_file.write(header) + output_file.write(data) + except OSError as e: + log(args, MSG_ERROR, "Error writing output file: %s", + format(e.errno, e.strerror)) + sys.exit(STATUS_ERROR) + log(args, MSG_INFO, ("Output file length: %d bytes " + "(%d bytes crc32 header and %d bytes data)") % + ((len(header) + len(data)), len(header), len(data))) + + # Finish + log(args, MSG_INFO, "Success") + sys.exit(STATUS_SUCCESS) + + +if __name__ =3D=3D "__main__": + main() diff --git a/BaseTools/Tests/PythonToolsTests.py b/BaseTools/Tests/PythonTo= olsTests.py index 05b27ab033..4d0ae89508 100644 --- a/BaseTools/Tests/PythonToolsTests.py +++ b/BaseTools/Tests/PythonToolsTests.py @@ -20,6 +20,8 @@ def TheTestSuite(): suites.append(CheckPythonSyntax.TheTestSuite()) import CheckUnicodeSourceFiles suites.append(CheckUnicodeSourceFiles.TheTestSuite()) + import TestGenCrc32 + suites.append(TestGenCrc32.TheTestSuite()) return unittest.TestSuite(suites) = if __name__ =3D=3D '__main__': diff --git a/BaseTools/Tests/TestGenCrc32.py b/BaseTools/Tests/TestGenCrc32= .py new file mode 100644 index 0000000000..937d0b14d5 --- /dev/null +++ b/BaseTools/Tests/TestGenCrc32.py @@ -0,0 +1,64 @@ +# @file +# Unit tests for GenCrc32 utility +# +# Copyright (c) 2008, Intel Corporation. All rights reserved.
+# +# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +import unittest +import TestTools + + +class Tests(TestTools.BaseToolsTest): + + def setUp(self): + TestTools.BaseToolsTest.setUp(self) + self.toolName =3D 'GenCrc32' + + def test_display_help(self): + result =3D self.RunTool( + '--help', + logFile=3D'help' + ) + self.assertTrue(result =3D=3D 0) + + def do_encode_decode(self, data): + path =3D self.GetTmpFilePath('input') + self.WriteTmpFile('input', data) + result =3D self.RunTool( + '--verbose', + '-e', + '-o', self.GetTmpFilePath('output1'), + self.GetTmpFilePath('input'), + ) + self.assertTrue(result =3D=3D 0) + result =3D self.RunTool( + '-d', + '-o', self.GetTmpFilePath('output2'), + self.GetTmpFilePath('output1') + ) + self.assertTrue(result =3D=3D 0) + start =3D self.ReadTmpFile('input') + finish =3D self.ReadTmpFile('output2') + start_equals_finish =3D start =3D=3D finish + if not start_equals_finish: + print('Original data did not match decode(encode(data))') + self.DisplayBinaryData('original data', start) + self.DisplayBinaryData('after encoding', + self.ReadTmpFile('output1')) + self.DisplayBinaryData('after decoding', finish) + self.assertTrue(start_equals_finish) + + def test_encode_decode_various_sizes(self): + for i in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 100, 1000]: + data =3D self.GetRandomString(i, i) + self.do_encode_decode(data) + self.CleanUpTmpDir() + + +TheTestSuite =3D TestTools.MakeTheTestSuite(locals()) + +if __name__ =3D=3D '__main__': + allTests =3D TheTestSuite() + unittest.TextTestRunner().run(allTests) -- = 2.25.1 -------------------------------------------------------------------- Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.