* [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue
@ 2021-05-14 6:04 Yuwei Chen
2021-05-14 7:56 ` [edk2-devel] " Bob Feng
2021-05-14 7:58 ` Bob Feng
0 siblings, 2 replies; 4+ messages in thread
From: Yuwei Chen @ 2021-05-14 6:04 UTC (permalink / raw)
To: devel; +Cc: Bob Feng, Liming Gao
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2946
Currenly, when using the Brotli tool to compress data, the output
compressed binary file does not record complete compressed data
when size of input file is too large, which makes the data loss and
will trigger decompress-check issue.
The Brotli document mentioned:
The brotli tool use BrotliEncoderCompressStream method to compresses
input stream to output stream. Under some circumstances (e.g. lack of
output stream capacity) the BrotliEncoderOperation would require
several calls to BrotliEncoderCompressStream. The method must be
called again until both input stream is depleted and encoder has no
more output after the method is called.
This patch fixes this issue based on the Brotli document.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
---
.../Source/C/BrotliCompress/BrotliCompress.c | 61 ++++++++++++-------
1 file changed, 38 insertions(+), 23 deletions(-)
diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
index 5a1400fda310..62a6aed3dbd0 100644
--- a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
+++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
@@ -156,6 +156,7 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_
uint8_t *NextOut;
uint8_t *Input;
uint8_t *Output;
+ size_t TotalOut;
size_t OutSize;
uint32_t SizeHint;
BROTLI_BOOL IsOk;
@@ -214,39 +215,53 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_
IsEof = !HasMoreInput(InputFileHandle);
}
- if (!BrotliEncoderCompressStream(EncodeState,
- IsEof ? BROTLI_OPERATION_FINISH : BROTLI_OPERATION_PROCESS,
- &AvailableIn, &NextIn, &AvailableOut, &NextOut, NULL)) {
- printf("Failed to compress data [%s]\n", InputFile);
- IsOk = BROTLI_FALSE;
- goto Finish;
- }
- if (AvailableOut == 0) {
- OutSize = (size_t)(NextOut - Output);
- if (OutSize > 0) {
- fwrite(Output, 1, OutSize, OutputFileHandle);
- if (ferror(OutputFileHandle)) {
- printf("Failed to write output [%s]\n", OutputFile);
+ if (!IsEof){
+ do{
+ if (!BrotliEncoderCompressStream(EncodeState,
+ BROTLI_OPERATION_FLUSH,
+ &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
+ printf("Failed to compress data [%s]\n", InputFile);
IsOk = BROTLI_FALSE;
goto Finish;
}
+ OutSize = (size_t)(NextOut - Output);
+ if (OutSize > 0) {
+ fwrite(Output, 1, OutSize, OutputFileHandle);
+ if (ferror(OutputFileHandle)) {
+ printf("Failed to write output [%s]\n", OutputFile);
+ IsOk = BROTLI_FALSE;
+ goto Finish;
+ }
+ }
+ NextOut = Output;
+ AvailableOut = kFileBufferSize;
}
- AvailableOut = kFileBufferSize;
- NextOut = Output;
+ while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
}
- if (BrotliEncoderIsFinished(EncodeState)) {
- OutSize = (size_t)(NextOut - Output);
- if (OutSize > 0) {
- fwrite(Output, 1, OutSize, OutputFileHandle);
- if (ferror(OutputFileHandle)) {
- printf("Failed to write output [%s]\n", OutputFile);
+ else{
+ do{
+ if (!BrotliEncoderCompressStream(EncodeState,
+ BROTLI_OPERATION_FINISH,
+ &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
+ printf("Failed to compress data [%s]\n", InputFile);
IsOk = BROTLI_FALSE;
goto Finish;
}
- AvailableOut = 0;
+ OutSize = (size_t)(NextOut - Output);
+ if (OutSize > 0) {
+ fwrite(Output, 1, OutSize, OutputFileHandle);
+ if (ferror(OutputFileHandle)) {
+ printf("Failed to write output [%s]\n", OutputFile);
+ IsOk = BROTLI_FALSE;
+ goto Finish;
+ }
+ }
+ NextOut = Output;
+ AvailableOut = kFileBufferSize;
}
+ while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
}
- if (IsEof) {
+ if (BrotliEncoderIsFinished(EncodeState)){
break;
}
}
--
2.27.0.windows.1
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [edk2-devel] [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue
2021-05-14 6:04 [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue Yuwei Chen
@ 2021-05-14 7:56 ` Bob Feng
2021-05-14 7:58 ` Bob Feng
1 sibling, 0 replies; 4+ messages in thread
From: Bob Feng @ 2021-05-14 7:56 UTC (permalink / raw)
To: Yuwei Chen, devel
[-- Attachment #1: Type: text/plain, Size: 46 bytes --]
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
[-- Attachment #2: Type: text/html, Size: 52 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [edk2-devel] [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue
2021-05-14 6:04 [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue Yuwei Chen
2021-05-14 7:56 ` [edk2-devel] " Bob Feng
@ 2021-05-14 7:58 ` Bob Feng
2021-05-17 3:18 ` Bob Feng
1 sibling, 1 reply; 4+ messages in thread
From: Bob Feng @ 2021-05-14 7:58 UTC (permalink / raw)
To: Yuwei Chen, devel
[-- Attachment #1: Type: text/plain, Size: 4254 bytes --]
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
On Fri, May 14, 2021 at 02:04 PM, Yuwei Chen wrote:
>
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2946
>
> Currenly, when using the Brotli tool to compress data, the output
> compressed binary file does not record complete compressed data
> when size of input file is too large, which makes the data loss and
> will trigger decompress-check issue.
>
> The Brotli document mentioned:
> The brotli tool use BrotliEncoderCompressStream method to compresses
> input stream to output stream. Under some circumstances (e.g. lack of
> output stream capacity) the BrotliEncoderOperation would require
> several calls to BrotliEncoderCompressStream. The method must be
> called again until both input stream is depleted and encoder has no
> more output after the method is called.
>
> This patch fixes this issue based on the Brotli document.
>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
> ---
> .../Source/C/BrotliCompress/BrotliCompress.c | 61 ++++++++++++-------
> 1 file changed, 38 insertions(+), 23 deletions(-)
>
> diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> index 5a1400fda310..62a6aed3dbd0 100644
> --- a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> +++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
> @@ -156,6 +156,7 @@ int CompressFile(char *InputFile, uint8_t
> *InputBuffer, char *OutputFile, uint8_
> uint8_t *NextOut;
> uint8_t *Input;
> uint8_t *Output;
> + size_t TotalOut;
> size_t OutSize;
> uint32_t SizeHint;
> BROTLI_BOOL IsOk;
> @@ -214,39 +215,53 @@ int CompressFile(char *InputFile, uint8_t
> *InputBuffer, char *OutputFile, uint8_
> IsEof = !HasMoreInput(InputFileHandle);
> }
>
> - if (!BrotliEncoderCompressStream(EncodeState,
> - IsEof ? BROTLI_OPERATION_FINISH : BROTLI_OPERATION_PROCESS,
> - &AvailableIn, &NextIn, &AvailableOut, &NextOut, NULL)) {
> - printf("Failed to compress data [%s]\n", InputFile);
> - IsOk = BROTLI_FALSE;
> - goto Finish;
> - }
> - if (AvailableOut == 0) {
> - OutSize = (size_t)(NextOut - Output);
> - if (OutSize > 0) {
> - fwrite(Output, 1, OutSize, OutputFileHandle);
> - if (ferror(OutputFileHandle)) {
> - printf("Failed to write output [%s]\n", OutputFile);
> + if (!IsEof){
> + do{
> + if (!BrotliEncoderCompressStream(EncodeState,
> + BROTLI_OPERATION_FLUSH,
> + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
> + printf("Failed to compress data [%s]\n", InputFile);
> IsOk = BROTLI_FALSE;
> goto Finish;
> }
> + OutSize = (size_t)(NextOut - Output);
> + if (OutSize > 0) {
> + fwrite(Output, 1, OutSize, OutputFileHandle);
> + if (ferror(OutputFileHandle)) {
> + printf("Failed to write output [%s]\n", OutputFile);
> + IsOk = BROTLI_FALSE;
> + goto Finish;
> + }
> + }
> + NextOut = Output;
> + AvailableOut = kFileBufferSize;
> }
> - AvailableOut = kFileBufferSize;
> - NextOut = Output;
> + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
> }
> - if (BrotliEncoderIsFinished(EncodeState)) {
> - OutSize = (size_t)(NextOut - Output);
> - if (OutSize > 0) {
> - fwrite(Output, 1, OutSize, OutputFileHandle);
> - if (ferror(OutputFileHandle)) {
> - printf("Failed to write output [%s]\n", OutputFile);
> + else{
> + do{
> + if (!BrotliEncoderCompressStream(EncodeState,
> + BROTLI_OPERATION_FINISH,
> + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
> + printf("Failed to compress data [%s]\n", InputFile);
> IsOk = BROTLI_FALSE;
> goto Finish;
> }
> - AvailableOut = 0;
> + OutSize = (size_t)(NextOut - Output);
> + if (OutSize > 0) {
> + fwrite(Output, 1, OutSize, OutputFileHandle);
> + if (ferror(OutputFileHandle)) {
> + printf("Failed to write output [%s]\n", OutputFile);
> + IsOk = BROTLI_FALSE;
> + goto Finish;
> + }
> + }
> + NextOut = Output;
> + AvailableOut = kFileBufferSize;
> }
> + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
> }
> - if (IsEof) {
> + if (BrotliEncoderIsFinished(EncodeState)){
> break;
> }
> }
> --
> 2.27.0.windows.1
[-- Attachment #2: Type: text/html, Size: 4700 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [edk2-devel] [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue
2021-05-14 7:58 ` Bob Feng
@ 2021-05-17 3:18 ` Bob Feng
0 siblings, 0 replies; 4+ messages in thread
From: Bob Feng @ 2021-05-17 3:18 UTC (permalink / raw)
To: devel@edk2.groups.io, Feng, Bob C, Chen, Christine
[-- Attachment #1: Type: text/plain, Size: 4488 bytes --]
Created a PR for this patch. https://github.com/tianocore/edk2/pull/1648
Thanks,
Bob
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob Feng
Sent: Friday, May 14, 2021 3:58 PM
To: Chen, Christine <yuwei.chen@intel.com>; devel@edk2.groups.io
Subject: Re: [edk2-devel] [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue
Reviewed-by: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
On Fri, May 14, 2021 at 02:04 PM, Yuwei Chen wrote:
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2946
Currenly, when using the Brotli tool to compress data, the output
compressed binary file does not record complete compressed data
when size of input file is too large, which makes the data loss and
will trigger decompress-check issue.
The Brotli document mentioned:
The brotli tool use BrotliEncoderCompressStream method to compresses
input stream to output stream. Under some circumstances (e.g. lack of
output stream capacity) the BrotliEncoderOperation would require
several calls to BrotliEncoderCompressStream. The method must be
called again until both input stream is depleted and encoder has no
more output after the method is called.
This patch fixes this issue based on the Brotli document.
Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Cc: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com<mailto:yuwei.chen@intel.com>>
---
.../Source/C/BrotliCompress/BrotliCompress.c | 61 ++++++++++++-------
1 file changed, 38 insertions(+), 23 deletions(-)
diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
index 5a1400fda310..62a6aed3dbd0 100644
--- a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
+++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c
@@ -156,6 +156,7 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_
uint8_t *NextOut;
uint8_t *Input;
uint8_t *Output;
+ size_t TotalOut;
size_t OutSize;
uint32_t SizeHint;
BROTLI_BOOL IsOk;
@@ -214,39 +215,53 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_
IsEof = !HasMoreInput(InputFileHandle);
}
- if (!BrotliEncoderCompressStream(EncodeState,
- IsEof ? BROTLI_OPERATION_FINISH : BROTLI_OPERATION_PROCESS,
- &AvailableIn, &NextIn, &AvailableOut, &NextOut, NULL)) {
- printf("Failed to compress data [%s]\n", InputFile);
- IsOk = BROTLI_FALSE;
- goto Finish;
- }
- if (AvailableOut == 0) {
- OutSize = (size_t)(NextOut - Output);
- if (OutSize > 0) {
- fwrite(Output, 1, OutSize, OutputFileHandle);
- if (ferror(OutputFileHandle)) {
- printf("Failed to write output [%s]\n", OutputFile);
+ if (!IsEof){
+ do{
+ if (!BrotliEncoderCompressStream(EncodeState,
+ BROTLI_OPERATION_FLUSH,
+ &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
+ printf("Failed to compress data [%s]\n", InputFile);
IsOk = BROTLI_FALSE;
goto Finish;
}
+ OutSize = (size_t)(NextOut - Output);
+ if (OutSize > 0) {
+ fwrite(Output, 1, OutSize, OutputFileHandle);
+ if (ferror(OutputFileHandle)) {
+ printf("Failed to write output [%s]\n", OutputFile);
+ IsOk = BROTLI_FALSE;
+ goto Finish;
+ }
+ }
+ NextOut = Output;
+ AvailableOut = kFileBufferSize;
}
- AvailableOut = kFileBufferSize;
- NextOut = Output;
+ while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
}
- if (BrotliEncoderIsFinished(EncodeState)) {
- OutSize = (size_t)(NextOut - Output);
- if (OutSize > 0) {
- fwrite(Output, 1, OutSize, OutputFileHandle);
- if (ferror(OutputFileHandle)) {
- printf("Failed to write output [%s]\n", OutputFile);
+ else{
+ do{
+ if (!BrotliEncoderCompressStream(EncodeState,
+ BROTLI_OPERATION_FINISH,
+ &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) {
+ printf("Failed to compress data [%s]\n", InputFile);
IsOk = BROTLI_FALSE;
goto Finish;
}
- AvailableOut = 0;
+ OutSize = (size_t)(NextOut - Output);
+ if (OutSize > 0) {
+ fwrite(Output, 1, OutSize, OutputFileHandle);
+ if (ferror(OutputFileHandle)) {
+ printf("Failed to write output [%s]\n", OutputFile);
+ IsOk = BROTLI_FALSE;
+ goto Finish;
+ }
+ }
+ NextOut = Output;
+ AvailableOut = kFileBufferSize;
}
+ while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState));
}
- if (IsEof) {
+ if (BrotliEncoderIsFinished(EncodeState)){
break;
}
}
--
2.27.0.windows.1
[-- Attachment #2: Type: text/html, Size: 7651 bytes --]
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-05-17 3:18 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-05-14 6:04 [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue Yuwei Chen
2021-05-14 7:56 ` [edk2-devel] " Bob Feng
2021-05-14 7:58 ` Bob Feng
2021-05-17 3:18 ` Bob Feng
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox