From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 757008042A for ; Wed, 22 Mar 2017 19:16:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1490235385; x=1521771385; h=from:to:cc:subject:date:message-id:mime-version; bh=jBmrGSydU9TWiSEZ/05vGrXVUW/Rder4VwJsAkJdNHo=; b=xJshBTYQRJwW9Cqua45yVb5OPIW8dlJG9u7yOzTsqCpvjEDAjGbchOum OEHOgUx7gzHP5oo6rLGVTOjsDqn2Pw==; Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2017 19:16:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,207,1486454400"; d="dat'59?scan'59,208,59";a="837610087" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by FMSMGA003.fm.intel.com with ESMTP; 22 Mar 2017 19:16:24 -0700 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 22 Mar 2017 19:16:23 -0700 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.88]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.204]) with mapi id 14.03.0248.002; Thu, 23 Mar 2017 10:16:20 +0800 From: "Song, BinX" To: "edk2-devel@lists.01.org" CC: "Gao, Liming" Thread-Topic: [PATCH 4/4] BaseTools: Add Brotli algorithm tool Thread-Index: AdKje3MVqvVryOCpQYaxNMHzkvXsIw== Date: Thu, 23 Mar 2017 02:16:16 +0000 Message-ID: <559D2DF22BC9A3468B4FA1AA547F0EF102545F60@shsmsx102.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: <559D2DF22BC9A3468B4FA1AA547F0EF102545F60@shsmsx102.ccr.corp.intel.com> x-originating-ip: [10.239.127.40] MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 Subject: [PATCH 4/4] BaseTools: Add Brotli algorithm tool X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 02:16:25 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable - Add Brotli algorithm tool support Cc: Liming Gao Contributed-under: TianoCore Contribution Agreement 1.0 Signed-off-by: Bell Song --- BaseTools/BinWrappers/PosixLike/Brotli | 29 ++++++ BaseTools/BinWrappers/PosixLike/BrotliCompress | 42 ++++++++ BaseTools/Conf/tools_def.template | 6 ++ .../Source/C/BrotliCompress/BrotliCompress.bat | 48 ++++++++++ BaseTools/Source/C/BrotliCompress/GNUmakefile | 43 +++++++++ BaseTools/Source/C/BrotliCompress/Makefile | 60 ++++++++++++ BaseTools/Source/C/BrotliCompress/ReadMe.txt | 2 + BaseTools/Source/C/BrotliCompress/tools/bro.c | 106 +++++++++++++++++= ---- BaseTools/Source/C/GNUmakefile | 1 + BaseTools/Source/C/Makefile | 1 + 10 files changed, 318 insertions(+), 20 deletions(-) create mode 100644 BaseTools/BinWrappers/PosixLike/Brotli create mode 100644 BaseTools/BinWrappers/PosixLike/BrotliCompress create mode 100644 BaseTools/Source/C/BrotliCompress/BrotliCompress.bat create mode 100644 BaseTools/Source/C/BrotliCompress/GNUmakefile create mode 100644 BaseTools/Source/C/BrotliCompress/Makefile create mode 100644 BaseTools/Source/C/BrotliCompress/ReadMe.txt diff --git a/BaseTools/BinWrappers/PosixLike/Brotli b/BaseTools/BinWrappers= /PosixLike/Brotli new file mode 100644 index 0000000..0945d86 --- /dev/null +++ b/BaseTools/BinWrappers/PosixLike/Brotli @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +full_cmd=3D${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028= for a discussion of why $0 is not a good choice here +dir=3D$(dirname "$full_cmd") +cmd=3D${full_cmd##*/} + +if [ -n "$WORKSPACE" ] && [ -e "$WORKSPACE/Conf/BaseToolsCBinaries" ] +then + exec "$WORKSPACE/Conf/BaseToolsCBinaries/$cmd" +elif [ -n "$WORKSPACE" ] && [ -e "$EDK_TOOLS_PATH/Source/C" ] +then + if [ ! -e "$EDK_TOOLS_PATH/Source/C/bin/$cmd" ] + then + echo "BaseTools C Tool binary was not found ($cmd)" + echo "You may need to run:" + echo " make -C $EDK_TOOLS_PATH/Source/C" + else + exec "$EDK_TOOLS_PATH/Source/C/bin/$cmd" "$@" + fi +elif [ -e "$dir/../../Source/C/bin/$cmd" ] +then + exec "$dir/../../Source/C/bin/$cmd" "$@" +else + echo "Unable to find the real '$cmd' to run" + echo "This message was printed by" + echo " $0" + exit 127 +fi + diff --git a/BaseTools/BinWrappers/PosixLike/BrotliCompress b/BaseTools/Bin= Wrappers/PosixLike/BrotliCompress new file mode 100644 index 0000000..59c6465 --- /dev/null +++ b/BaseTools/BinWrappers/PosixLike/BrotliCompress @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# +# This script will exec Brotli tool. +# +# Copyright (c) 2017, Intel Corporation. All rights reserved.
+# This program and the accompanying materials +# are licensed and made available under the terms and conditions of the BS= D License +# which accompanies this distribution. The full text of the license may b= e found at +# http://opensource.org/licenses/bsd-license.php +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, +# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMP= LIED. +# +LVL=3D"--quality 9" + +while [ $# !=3D 0 ];do + case $1 in + -d) + ARGS+=3D"--decompress " + ;; + -e) + ;; + -g) + ARGS+=3D"--gap $2 " + shift + ;; + -l) + LVL=3D"--quality $2 " + shift + ;; + -o) + ARGS+=3D"--output $2 " + shift + ;; + *) + ARGS+=3D"--input $1 " + esac + +shift +done + +exec Brotli $ARGS $LVL diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.t= emplate index aaae4fc..168e42a 100755 --- a/BaseTools/Conf/tools_def.template +++ b/BaseTools/Conf/tools_def.template @@ -7691,6 +7691,12 @@ RELEASE_RVCTCYGWIN_ARM_CC_FLAGS =3D "$(CCPATH_FLAG)= " $(ARCHCC_FLAGS) $(PLATFORM_F *_*_*_RSA2048SHA256SIGN_GUID =3D A7717414-C616-4977-9420-844712A735BF =20 ################## +# BrotliCompress tool definitions +################## +*_*_*_BROTLI_PATH =3D BrotliCompress +*_*_*_BROTLI_GUID =3D 3D532050-5CDA-4FD0-879E-0F7F630D5AFB + +################## # LzmaCompress tool definitions ################## *_*_*_LZMA_PATH =3D LzmaCompress diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.bat b/BaseToo= ls/Source/C/BrotliCompress/BrotliCompress.bat new file mode 100644 index 0000000..62d10f7 --- /dev/null +++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.bat @@ -0,0 +1,48 @@ +@echo off +@setlocal + +set LVL=3D--quality 9 + +:Begin +if "%1"=3D=3D"" goto End + +if "%1"=3D=3D"-d" ( + set ARGS=3D%ARGS% --decompress + shift + goto Begin +) + +if "%1"=3D=3D"-e" ( + shift + goto Begin +) + +if "%1"=3D=3D"-g" ( + set ARGS=3D%ARGS% --gap %2 + shift + shift + goto Begin +) + +if "%1"=3D=3D"-l" ( + set LVL=3D--quality %2 + shift + shift + goto Begin +) + +if "%1"=3D=3D"-o" ( + set ARGS=3D%ARGS% --output %2 + set INTMP=3D%2 + shift + shift + goto Begin +) + +set ARGS=3D%ARGS% --input %1 +shift +goto Begin + +:End +Brotli %ARGS% %LVL% +@echo on \ No newline at end of file diff --git a/BaseTools/Source/C/BrotliCompress/GNUmakefile b/BaseTools/Sour= ce/C/BrotliCompress/GNUmakefile new file mode 100644 index 0000000..89203c8 --- /dev/null +++ b/BaseTools/Source/C/BrotliCompress/GNUmakefile @@ -0,0 +1,43 @@ +## @file +# GNU/Linux makefile for 'Brotli' module build. +# +# Copyright (c) 2017, Intel Corporation. All rights reserved.
+# This program and the accompanying materials +# are licensed and made available under the terms and conditions of the BS= D License +# which accompanies this distribution. The full text of the license may b= e found at +# http://opensource.org/licenses/bsd-license.php +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, +# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMP= LIED. +# +ARCH ?=3D IA32 +MAKEROOT ?=3D .. + +APPNAME =3D Brotli + +OBJECTS =3D \ + tools/bro.o \ + common/dictionary.o \ + dec/bit_reader.o \ + dec/decode.o \ + dec/huffman.o \ + dec/state.o \ + enc/backward_references.o \ + enc/bit_cost.o \ + enc/block_splitter.o \ + enc/brotli_bit_stream.o \ + enc/cluster.o \ + enc/compress_fragment.o \ + enc/compress_fragment_two_pass.o \ + enc/encode.o \ + enc/entropy_encode.o \ + enc/histogram.o \ + enc/literal_cost.o \ + enc/memory.o \ + enc/metablock.o \ + enc/static_dict.o \ + enc/utf8_util.o + +include $(MAKEROOT)/Makefiles/app.makefile + +LIBS +=3D -lm diff --git a/BaseTools/Source/C/BrotliCompress/Makefile b/BaseTools/Source/= C/BrotliCompress/Makefile new file mode 100644 index 0000000..9d45ea5 --- /dev/null +++ b/BaseTools/Source/C/BrotliCompress/Makefile @@ -0,0 +1,60 @@ +## @file +# Windows makefile for 'Brotli' module build. +# +# Copyright (c) 2017, Intel Corporation. All rights reserved.
+# This program and the accompanying materials +# are licensed and made available under the terms and conditions of the BS= D License +# which accompanies this distribution. The full text of the license may b= e found at +# http://opensource.org/licenses/bsd-license.php +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, +# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMP= LIED. +# +!INCLUDE ..\Makefiles\ms.common + +CFLAGS =3D $(CFLAGS) /W2 + +APPNAME =3D Brotli + +#LIBS =3D $(LIB_PATH)\Common.lib + +COMMON_OBJ =3D common\dictionary.obj +DEC_OBJ =3D \ + dec\bit_reader.obj \ + dec\decode.obj \ + dec\huffman.obj \ + dec\state.obj +ENC_OBJ =3D \ + enc\backward_references.obj \ + enc\bit_cost.obj \ + enc\block_splitter.obj \ + enc\brotli_bit_stream.obj \ + enc\cluster.obj \ + enc\compress_fragment.obj \ + enc\compress_fragment_two_pass.obj \ + enc\encode.obj \ + enc\entropy_encode.obj \ + enc\histogram.obj \ + enc\literal_cost.obj \ + enc\memory.obj \ + enc\metablock.obj \ + enc\static_dict.obj \ + enc\utf8_util.obj + +OBJECTS =3D \ + tools\bro.obj \ + $(COMMON_OBJ) \ + $(DEC_OBJ) \ + $(ENC_OBJ) + +!INCLUDE ..\Makefiles\ms.app + +all: $(BIN_PATH)\BrotliCompress.bat + +$(BIN_PATH)\BrotliCompress.bat: BrotliCompress.bat + copy BrotliCompress.bat $(BIN_PATH)\BrotliCompress.bat /Y + +cleanall: localCleanall + +localCleanall: + del /f /q $(BIN_PATH)\BrotliCompress.bat > nul diff --git a/BaseTools/Source/C/BrotliCompress/ReadMe.txt b/BaseTools/Sourc= e/C/BrotliCompress/ReadMe.txt new file mode 100644 index 0000000..c19c0a1 --- /dev/null +++ b/BaseTools/Source/C/BrotliCompress/ReadMe.txt @@ -0,0 +1,2 @@ +It is based on the Brotli v0.5.2. +Brotli was released on the website https://github.com/google/brotli. diff --git a/BaseTools/Source/C/BrotliCompress/tools/bro.c b/BaseTools/Sour= ce/C/BrotliCompress/tools/bro.c index 426c573..f11d192 100644 --- a/BaseTools/Source/C/BrotliCompress/tools/bro.c +++ b/BaseTools/Source/C/BrotliCompress/tools/bro.c @@ -73,6 +73,7 @@ static void ParseArgv(int argc, char **argv, char **dictionary_path, int *force, int *quality, + int *gapmem, int *decompress, int *repeat, int *verbose, @@ -160,6 +161,13 @@ static void ParseArgv(int argc, char **argv, } ++k; continue; + } else if (!strcmp("--gap", argv[k]) || + !strcmp("-g", argv[k])) { + if (!ParseQuality(argv[k + 1], gapmem)) { + goto error; + } + ++k; + continue; } } goto error; @@ -167,7 +175,7 @@ static void ParseArgv(int argc, char **argv, return; error: fprintf(stderr, - "Usage: %s [--force] [--quality n] [--decompress]" + "Usage: %s [--force] [--quality n] [--gap n] [--decompress]" " [--input filename] [--output filename] [--repeat iters]" " [--verbose] [--window n] [--custom-dictionary filename]\n", argv[0]); @@ -225,9 +233,20 @@ static int64_t FileSize(const char *path) { return retval; } =20 +/* Brotli specified memory allocate function */ +static void *BrAlloc (void *memsize, size_t size) { + *(int64_t *)memsize +=3D size; + return malloc(size); +} + +/* Brotli specified memory free function */ +static void BrFree (void *memsize, void *ptr) { + free(ptr); +} + /* Result ownersip is passed to caller. |*dictionary_size| is set to resulting buffer size. */ -static uint8_t* ReadDictionary(const char* path, size_t* dictionary_size) = { +static uint8_t* ReadDictionary(const char* path, size_t* dictionary_size, = void *memsize) { static const int kMaxDictionarySize =3D (1 << 24) - 16; FILE *f =3D fopen(path, "rb"); int64_t file_size_64; @@ -252,7 +271,7 @@ static uint8_t* ReadDictionary(const char* path, size_t= * dictionary_size) { } *dictionary_size =3D (size_t)file_size_64; =20 - buffer =3D (uint8_t*)malloc(*dictionary_size); + buffer =3D (uint8_t *)BrAlloc(memsize, *dictionary_size); if (!buffer) { fprintf(stderr, "could not read dictionary: out of memory\n"); exit(1); @@ -268,7 +287,7 @@ static uint8_t* ReadDictionary(const char* path, size_t= * dictionary_size) { =20 static const size_t kFileBufferSize =3D 65536; =20 -static int Decompress(FILE* fin, FILE* fout, const char* dictionary_path) = { +static int Decompress(FILE* fin, FILE* fout, const char* dictionary_path, = void *memsize) { /* Dictionary should be kept during first rounds of decompression. */ uint8_t* dictionary =3D NULL; uint8_t* input; @@ -279,18 +298,18 @@ static int Decompress(FILE* fin, FILE* fout, const ch= ar* dictionary_path) { size_t available_out =3D kFileBufferSize; uint8_t* next_out; BrotliResult result =3D BROTLI_RESULT_ERROR; - BrotliState* s =3D BrotliCreateState(NULL, NULL, NULL); + BrotliState* s =3D BrotliCreateState(BrAlloc, BrFree, memsize); if (!s) { fprintf(stderr, "out of memory\n"); return 0; } if (dictionary_path !=3D NULL) { size_t dictionary_size =3D 0; - dictionary =3D ReadDictionary(dictionary_path, &dictionary_size); + dictionary =3D ReadDictionary(dictionary_path, &dictionary_size, memsi= ze); BrotliSetCustomDictionary(dictionary_size, dictionary, s); } - input =3D (uint8_t*)malloc(kFileBufferSize); - output =3D (uint8_t*)malloc(kFileBufferSize); + input =3D (uint8_t*)BrAlloc(memsize, kFileBufferSize); + output =3D (uint8_t*)BrAlloc(memsize, kFileBufferSize); if (!input || !output) { fprintf(stderr, "out of memory\n"); goto end; @@ -331,17 +350,17 @@ static int Decompress(FILE* fin, FILE* fout, const ch= ar* dictionary_path) { } =20 end: - free(dictionary); - free(input); - free(output); + BrFree(memsize, dictionary); + BrFree(memsize, input); + BrFree(memsize, output); BrotliDestroyState(s); return (result =3D=3D BROTLI_RESULT_SUCCESS) ? 1 : 0; } =20 static int Compress(int quality, int lgwin, FILE* fin, FILE* fout, - const char *dictionary_path) { - BrotliEncoderState* s =3D BrotliEncoderCreateInstance(0, 0, 0); - uint8_t* buffer =3D (uint8_t*)malloc(kFileBufferSize << 1); + const char *dictionary_path, void *memsize) { + BrotliEncoderState* s =3D BrotliEncoderCreateInstance(BrAlloc, BrFree, m= emsize); + uint8_t* buffer =3D (uint8_t*)BrAlloc(memsize, kFileBufferSize << 1); uint8_t* input =3D buffer; uint8_t* output =3D buffer + kFileBufferSize; size_t available_in =3D 0; @@ -360,9 +379,9 @@ static int Compress(int quality, int lgwin, FILE* fin, = FILE* fout, BrotliEncoderSetParameter(s, BROTLI_PARAM_LGWIN, (uint32_t)lgwin); if (dictionary_path !=3D NULL) { size_t dictionary_size =3D 0; - uint8_t* dictionary =3D ReadDictionary(dictionary_path, &dictionary_si= ze); + uint8_t* dictionary =3D ReadDictionary(dictionary_path, &dictionary_si= ze, memsize); BrotliEncoderSetCustomDictionary(s, dictionary_size, dictionary); - free(dictionary); + BrFree(memsize, dictionary); } =20 while (1) { @@ -392,7 +411,7 @@ static int Compress(int quality, int lgwin, FILE* fin, = FILE* fout, } =20 finish: - free(buffer); + BrFree(memsize, buffer); BrotliEncoderDestroyInstance(s); =20 if (!is_ok) { @@ -409,29 +428,41 @@ finish: return 1; } =20 +#define GAP_MEM_BLOCK 4096 + int main(int argc, char** argv) { char *input_path =3D 0; char *output_path =3D 0; char *dictionary_path =3D 0; int force =3D 0; int quality =3D 11; + int gmem =3D 1; int decompress =3D 0; int repeat =3D 1; int verbose =3D 0; int lgwin =3D 0; clock_t clock_start; int i; + int64_t originsize =3D 0; + int64_t msize =3D 0; ParseArgv(argc, argv, &input_path, &output_path, &dictionary_path, &forc= e, - &quality, &decompress, &repeat, &verbose, &lgwin); + &quality, &gmem, &decompress, &repeat, &verbose, &lgwin); clock_start =3D clock(); for (i =3D 0; i < repeat; ++i) { FILE* fin =3D OpenInputFile(input_path); FILE* fout =3D OpenOutputFile(output_path, force || repeat); int is_ok =3D 0; if (decompress) { - is_ok =3D Decompress(fin, fout, dictionary_path); + if (fseek(fin, 16, SEEK_SET) !=3D 0) { + fclose(fin); + return -1; + } + is_ok =3D Decompress(fin, fout, dictionary_path, (void *)&msize); } else { - is_ok =3D Compress(quality, lgwin, fin, fout, dictionary_path); + originsize =3D FileSize(input_path); /* get original file size */ + fwrite(&originsize, 1, sizeof(int64_t), fout); /* add in original bi= nary file size */ + fwrite(&msize, 1, sizeof(int64_t), fout); /* add in dummy decompress= ion required memory size */ + is_ok =3D Compress(quality, lgwin, fin, fout, dictionary_path, (void= *)&msize); } if (!is_ok) { unlink(output_path); @@ -445,6 +476,41 @@ int main(int argc, char** argv) { perror("fclose"); exit(1); } + /* after compression operation then execute decompression operation + to get decompression required memory size. */ + if (decompress =3D=3D 0) { + fin =3D OpenInputFile(output_path); + fout =3D tmpfile (); + msize =3D 0; + if (fseek(fin, 16, SEEK_SET) !=3D 0) { + fclose(fin); + return -1; + } + is_ok =3D Decompress(fin, fout, dictionary_path, (void *)&msize); + if (!is_ok) { + exit(1); + } + if (fclose(fin) !=3D 0) { + perror("fclose"); + exit(1); + } + if (fclose(fout) !=3D 0) { + perror("fclose"); + exit(1); + } + fout =3D fopen(output_path, "rb+"); /* open output_path file and ad= d in head info */ + /* seek to the offset of decompression required memory size */ + if (fseek(fout, 8, SEEK_SET) !=3D 0) { + fclose(fout); + return -1; + } + msize +=3D gmem * GAP_MEM_BLOCK; /* there is a memory gap between I= A32 and X64 environment*/ + fwrite(&msize, 1, sizeof(int64_t), fout); /* update final decompress= ion required memory size */ + if (fclose(fout) !=3D 0) { + perror("fclose"); + exit(1); + } + } } if (verbose) { clock_t clock_end =3D clock(); diff --git a/BaseTools/Source/C/GNUmakefile b/BaseTools/Source/C/GNUmakefil= e index 66baabb..83e188c 100644 --- a/BaseTools/Source/C/GNUmakefile +++ b/BaseTools/Source/C/GNUmakefile @@ -52,6 +52,7 @@ LIBRARIES =3D Common APPLICATIONS =3D \ GnuGenBootSector \ BootSectImage \ + BrotliCompress \ EfiLdrImage \ EfiRom \ GenFfs \ diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile index 3005a17..77d3b97 100644 --- a/BaseTools/Source/C/Makefile +++ b/BaseTools/Source/C/Makefile @@ -17,6 +17,7 @@ ARCH =3D IA32 LIBRARIES =3D Common APPLICATIONS =3D \ BootSectImage \ + BrotliCompress \ EfiLdrImage \ EfiRom \ GenBootSector \ --=20 2.10.2.windows.1