Re: [edk2-devel] [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue

["Bob Feng" <bob.c.feng@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4254 bytes --]

Reviewed-by: Bob Feng <bob.c.feng@intel.com>

On Fri, May 14, 2021 at 02:04 PM, Yuwei Chen wrote:

> 
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2946
> 
> Currenly, when using the Brotli tool to compress data, the output
> compressed binary file does not record complete compressed data
> when size of input file is too large, which makes the data loss and
> will trigger decompress-check issue.
> 
> The Brotli document mentioned:
> The brotli tool use BrotliEncoderCompressStream method to compresses
> input stream to output stream. Under some circumstances (e.g. lack of
> output stream capacity) the BrotliEncoderOperation would require
> several calls to BrotliEncoderCompressStream. The method must be
> called again until both input stream is depleted and encoder has no
> more output after the method is called.
> 
> This patch fixes this issue based on the Brotli document.
> 
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> ---
> .../Source/C/BrotliCompress/BrotliCompress.c | 61 ++++++++++++-------
> 1 file changed, 38 insertions(+), 23 deletions(-)
> 
> diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> index 5a1400fda310..62a6aed3dbd0 100644
> --- a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> +++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> @@ -156,6 +156,7 @@ int CompressFile(char *InputFile, uint8_t
> *InputBuffer, char *OutputFile, uint8_
> uint8_t *NextOut;
> uint8_t *Input;
> uint8_t *Output;
> + size_t TotalOut;
> size_t OutSize;
> uint32_t SizeHint;
> BROTLI_BOOL IsOk;
> @@ -214,39 +215,53 @@ int CompressFile(char *InputFile, uint8_t
> *InputBuffer, char *OutputFile, uint8_
> IsEof = !HasMoreInput(InputFileHandle);
> }
> 
> - if (!BrotliEncoderCompressStream(EncodeState,
> - IsEof ? BROTLI_OPERATION_FINISH : BROTLI_OPERATION_PROCESS,
> - &AvailableIn, &NextIn, &AvailableOut, &NextOut, NULL)) {
> - printf("Failed to compress data [%s]\n", InputFile);
> - IsOk = BROTLI_FALSE;
> - goto Finish;
> - }
> - if (AvailableOut == 0) {
> - OutSize = (size_t)(NextOut - Output);
> - if (OutSize > 0) {
> - fwrite(Output, 1, OutSize, OutputFileHandle);
> - if (ferror(OutputFileHandle)) {
> - printf("Failed to write output [%s]\n", OutputFile);
> + if (!IsEof){
> + do{
> + if (!BrotliEncoderCompressStream(EncodeState,
> + BROTLI_OPERATION_FLUSH,
> + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
> + printf("Failed to compress data [%s]\n", InputFile);
> IsOk = BROTLI_FALSE;
> goto Finish;
> }
> + OutSize = (size_t)(NextOut - Output);
> + if (OutSize > 0) {
> + fwrite(Output, 1, OutSize, OutputFileHandle);
> + if (ferror(OutputFileHandle)) {
> + printf("Failed to write output [%s]\n", OutputFile);
> + IsOk = BROTLI_FALSE;
> + goto Finish;
> + }
> + }
> + NextOut = Output;
> + AvailableOut = kFileBufferSize;
> }
> - AvailableOut = kFileBufferSize;
> - NextOut = Output;
> + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
> }
> - if (BrotliEncoderIsFinished(EncodeState)) {
> - OutSize = (size_t)(NextOut - Output);
> - if (OutSize > 0) {
> - fwrite(Output, 1, OutSize, OutputFileHandle);
> - if (ferror(OutputFileHandle)) {
> - printf("Failed to write output [%s]\n", OutputFile);
> + else{
> + do{
> + if (!BrotliEncoderCompressStream(EncodeState,
> + BROTLI_OPERATION_FINISH,
> + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
> + printf("Failed to compress data [%s]\n", InputFile);
> IsOk = BROTLI_FALSE;
> goto Finish;
> }
> - AvailableOut = 0;
> + OutSize = (size_t)(NextOut - Output);
> + if (OutSize > 0) {
> + fwrite(Output, 1, OutSize, OutputFileHandle);
> + if (ferror(OutputFileHandle)) {
> + printf("Failed to write output [%s]\n", OutputFile);
> + IsOk = BROTLI_FALSE;
> + goto Finish;
> + }
> + }
> + NextOut = Output;
> + AvailableOut = kFileBufferSize;
> }
> + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
> }
> - if (IsEof) {
> + if (BrotliEncoderIsFinished(EncodeState)){
> break;
> }
> }
> --
> 2.27.0.windows.1

[-- Attachment #2: Type: text/html, Size: 4700 bytes --]