From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mx.groups.io with SMTP id smtpd.web10.5005.1620972287793639670 for ; Thu, 13 May 2021 23:04:47 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.65, mailfrom: yuwei.chen@intel.com) IronPort-SDR: NRAFhA1lNLnEO4rIW58XxGwJs0UqhECnIdcaiJ05iL+Qqrbb9Vi+xpoQ5cNFAqprTNp6Zfxptp o1xCP6nsUUYg== X-IronPort-AV: E=McAfee;i="6200,9189,9983"; a="200172979" X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="200172979" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2021 23:04:46 -0700 IronPort-SDR: W0mz6e32zQXPUCAlMfqVT2NYXLKU8lHH7D2M/Ql/ffhQ8NMF16TyB0pG2wMwJ44kVZc3nr3jeO reEFRgQ/h5Mw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,299,1613462400"; d="scan'208";a="538704229" Received: from yuweipc.ccr.corp.intel.com ([10.239.158.34]) by fmsmga001.fm.intel.com with ESMTP; 13 May 2021 23:04:45 -0700 From: "Yuwei Chen" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao Subject: [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue Date: Fri, 14 May 2021 14:04:44 +0800 Message-Id: <20210514060444.1224-1-yuwei.chen@intel.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2946 Currenly, when using the Brotli tool to compress data, the output compressed binary file does not record complete compressed data when size of input file is too large, which makes the data loss and will trigger decompress-check issue. The Brotli document mentioned: The brotli tool use BrotliEncoderCompressStream method to compresses input stream to output stream. Under some circumstances (e.g. lack of output stream capacity) the BrotliEncoderOperation would require several calls to BrotliEncoderCompressStream. The method must be called again until both input stream is depleted and encoder has no more output after the method is called. This patch fixes this issue based on the Brotli document. Cc: Bob Feng Cc: Liming Gao Signed-off-by: Yuwei Chen --- .../Source/C/BrotliCompress/BrotliCompress.c | 61 ++++++++++++------- 1 file changed, 38 insertions(+), 23 deletions(-) diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c index 5a1400fda310..62a6aed3dbd0 100644 --- a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c +++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c @@ -156,6 +156,7 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_ uint8_t *NextOut; uint8_t *Input; uint8_t *Output; + size_t TotalOut; size_t OutSize; uint32_t SizeHint; BROTLI_BOOL IsOk; @@ -214,39 +215,53 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_ IsEof = !HasMoreInput(InputFileHandle); } - if (!BrotliEncoderCompressStream(EncodeState, - IsEof ? BROTLI_OPERATION_FINISH : BROTLI_OPERATION_PROCESS, - &AvailableIn, &NextIn, &AvailableOut, &NextOut, NULL)) { - printf("Failed to compress data [%s]\n", InputFile); - IsOk = BROTLI_FALSE; - goto Finish; - } - if (AvailableOut == 0) { - OutSize = (size_t)(NextOut - Output); - if (OutSize > 0) { - fwrite(Output, 1, OutSize, OutputFileHandle); - if (ferror(OutputFileHandle)) { - printf("Failed to write output [%s]\n", OutputFile); + if (!IsEof){ + do{ + if (!BrotliEncoderCompressStream(EncodeState, + BROTLI_OPERATION_FLUSH, + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) { + printf("Failed to compress data [%s]\n", InputFile); IsOk = BROTLI_FALSE; goto Finish; } + OutSize = (size_t)(NextOut - Output); + if (OutSize > 0) { + fwrite(Output, 1, OutSize, OutputFileHandle); + if (ferror(OutputFileHandle)) { + printf("Failed to write output [%s]\n", OutputFile); + IsOk = BROTLI_FALSE; + goto Finish; + } + } + NextOut = Output; + AvailableOut = kFileBufferSize; } - AvailableOut = kFileBufferSize; - NextOut = Output; + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState)); } - if (BrotliEncoderIsFinished(EncodeState)) { - OutSize = (size_t)(NextOut - Output); - if (OutSize > 0) { - fwrite(Output, 1, OutSize, OutputFileHandle); - if (ferror(OutputFileHandle)) { - printf("Failed to write output [%s]\n", OutputFile); + else{ + do{ + if (!BrotliEncoderCompressStream(EncodeState, + BROTLI_OPERATION_FINISH, + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) { + printf("Failed to compress data [%s]\n", InputFile); IsOk = BROTLI_FALSE; goto Finish; } - AvailableOut = 0; + OutSize = (size_t)(NextOut - Output); + if (OutSize > 0) { + fwrite(Output, 1, OutSize, OutputFileHandle); + if (ferror(OutputFileHandle)) { + printf("Failed to write output [%s]\n", OutputFile); + IsOk = BROTLI_FALSE; + goto Finish; + } + } + NextOut = Output; + AvailableOut = kFileBufferSize; } + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState)); } - if (IsEof) { + if (BrotliEncoderIsFinished(EncodeState)){ break; } } -- 2.27.0.windows.1