Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library

["Lin, Kuei-Hung (Timothy)" <Kuei-Hung.Lin@amd.com>]

[-- Attachment #1.1: Type: text/plain, Size: 16777 bytes --]

[AMD Official Use Only - General]

Hi Michael,

>> I am wondering if we need to expand package DSC builds to make sure all libraries that are built are also used in at least one module with that package build.
Agreed.
We may consider using a placeholder/dummy driver to link those NULL libraries to avoid accidentally "changing" the default behavior of DxeIpl including build-time linkage setting and run-time functionality.
Devising the functionality test cases for those NULL libraries can be another topic.


One more thing.
I noticed that I overlooked that VS2019 may need additional header files underMdeModulePkg/Library/BrotliCustomDecompressLib to fix the missing logic of the intrinsic functions: memset() and free().
I am attaching the two files here.

GCC build has no such problem.

--
Timothy
________________________________
From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Monday, January 30, 2023 5:15 PM
To: Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com>; Gao, Liming <gaoliming@byosoft.com.cn>; devel@edk2.groups.io <devel@edk2.groups.io>; He, Jiangang <Jiangang.He@amd.com>; Chang, Abner <Abner.Chang@amd.com>
Cc: Bi, Dandan <dandan.bi@intel.com>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Subject: RE: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library


[AMD Official Use Only - General]

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


Hi Timothy,



This is an interesting bug report because it looks like something that should be caught by EDK II CI



MdeModulePkg.dsc does list the Brotli lib in a [Components] section, so it will always be built.



[Components.IA32, Components.X64, Components.ARM, Components.AARCH64]

  MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf

  MdeModulePkg/Library/LzmaCustomDecompressLib/LzmaCustomDecompressLib.inf

  MdeModulePkg/Library/VarCheckUefiLib/VarCheckUefiLib.inf

  MdeModulePkg/Core/Dxe/DxeMain.inf {

    <LibraryClasses>

      NULL|MdeModulePkg/Library/DxeCrc32GuidedSectionExtractLib/DxeCrc32GuidedSectionExtractLib.inf

  }



However, a lib is allowed to have unresolved externals such as calling other libraries.  So unless the library is linked to a module, we do

not know if all the externals are resolved.



I am wondering if we need to expand package DSC builds to make sure all libraries that are built are also used in

at least one module with that package build.



You were able to produce this failure by attempting to use Brotli with the DxeIpl module, so the unresolved

externals that are resolved by the missing source files were not noticed until this used case.  For example MdeModulePkg.dsc

could be extended to DxeIpl with different NULL library instances for the different decompressors.



Mike



From: Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com>
Sent: Monday, January 30, 2023 4:50 PM
To: Gao, Liming <gaoliming@byosoft.com.cn>; devel@edk2.groups.io; He, Jiangang <Jiangang.He@amd.com>; Chang, Abner <Abner.Chang@amd.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Bi, Dandan <dandan.bi@intel.com>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com>
Subject: Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]





The failure can be replicated with VS2019 and GCC5.

  *   GCC5:

     *   /usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: warning: relocation against `_kBrotliPrefixCodeRanges' in read-only section `.text.BrotliUefiDecompress'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `DecodeLiteralBlockSwitch':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `DecodeCommandBlockSwitch':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `DecodeDistanceBlockSwitch':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `SafeDecodeLiteralBlockSwitch':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `SafeDecodeCommandBlockSwitch':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `SafeDecodeDistanceBlockSwitch':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `BrotliDecoderDestroyInstance':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/state.c:136: undefined reference to `BrotliSharedDictionaryDestroyInstance'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `BrotliUefiDecompress':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/state.c:87: undefined reference to `BrotliSharedDictionaryCreateInstance'

/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `BrotliUefiDecompress':

/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'

/usr/bin/ld: warning: creating DT_TEXTREL in a PIE

collect2: error: ld returned 1 exit status

...



build.py...

 : error F002: Failed to build module

      /home/XXX/edk2-stable202211/MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf [X64, GCC5, RELEASE]

  *   VS2019:

     *   BBrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliDefaultFreeFunc

BrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliDefaultAllocFunc

BrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliSharedDictionaryCreateInstance

BrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliSharedDictionaryDestroyInstance

BrotliDecompressLib.lib(decode.obj) : error LNK2001: unresolved external symbol _kBrotliPrefixCodeRanges

BrotliDecompressLib.lib(decode.obj) : error LNK2001: unresolved external symbol BrotliSharedDictionaryAttach

BrotliDecompressLib.lib(decode.obj) : error LNK2001: unresolved external symbol _kBrotliContextLookupTable

d:\XXX\Build\OvmfX64\RELEASE_VS2019\X64\MdeModulePkg\Core\DxeIplPeim\DxeIpl\DEBUG\DxeIpl.dll : fatal error LNK1120: 7 unresolved externals

...

build.py...

 : error 7000: Failed to execute command

        C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\Hostx86\x86\nmake.exe /nologo tbuild [d:\XXX\Build\OvmfX64\RELEASE_VS2019\X64\MdeModulePkg\Core\DxeIplPeim\DxeIpl]





--

Timothy

________________________________

From: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
Sent: Saturday, January 28, 2023 11:35 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com<mailto:Kuei-Hung.Lin@amd.com>>; He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>; Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>; 'Kinney, Michael D' <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: 'Bi, Dandan' <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>
Subject: 回复: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.



Timothy:

 Thanks for your report. This is a real issue in Brotli decompress library. Which tool chain has been verified? GCC5 or VS2019 or Both?



Thanks

Liming

发件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> 代表 Lin, Kuei-Hung (Timothy) via groups.io
发送时间: 2023年1月13日 6:18
收件人: He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>; Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
抄送: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>
主题: Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]



This patch tried to fix following build failure when linking the Brotli decompress lib.



Steps to replicate the issue:

1 - Attach a NULL lib to DxeIpl.inf in any project's .DSC, say OvmfPkgX64.dsc

  MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf {

    <LibraryClasses>

      NULL|MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf

  }

2 - Follow the EDKII'2 package build steps to build that project. Build.py would compliains with following errors.

...

/home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'

/usr/bin/ld: /tmp/ccKzzHN0.ltrans0.ltrans.o: in function `DecodeCommandBlockSwitch':

...

/usr/bin/ld: /tmp/ccKzzHN0.ltrans0.ltrans.o: in function `BrotliUefiDecompress':

/home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'



--

Timothy









________________________________

From: He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>
Sent: Thursday, January 12, 2023 8:08 AM
To: Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com<mailto:Kuei-Hung.Lin@amd.com>>
Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: RE: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]



Timothy will answer the question.



From: Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>
Sent: Thursday, January 12, 2023 10:05 AM
To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>
Subject: Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]



Hi Mike,

CC Jiangang, he can give the clear answer.



Thanks

Abner





Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________

From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Sent: Thursday, January 12, 2023 11:23:48 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>
Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: RE: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


Hi Abner,

What was the failure before this change?

This looks like something that would always break the build of source files were missing.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Chang, Abner via groups.io
> Sent: Wednesday, January 11, 2023 9:08 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
> Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Garrett Kirkendall <garrett.kirkendall@amd.com<mailto:garrett.kirkendall@amd.com>>;
> Abner Chang <abner.chang@amd.com<mailto:abner.chang@amd.com>>
> Subject: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library
>
> From: Abner Chang <abner.chang@amd.com<mailto:abner.chang@amd.com>>
>
> Add missing source files to Brotli library INF.
>
> Signed-off-by: Jiangang He <jiangang.he@amd.com<mailto:jiangang.he@amd.com>>
> Cc: Dandan Bi <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>
> Cc: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Cc: Garrett Kirkendall <garrett.kirkendall@amd.com<mailto:garrett.kirkendall@amd.com>>
> Cc: Abner Chang <abner.chang@amd.com<mailto:abner.chang@amd.com>>
> ---
>  .../BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf  | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> b/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> index 525e92408d6..244c2e35023 100644
> --- a/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> +++ b/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> @@ -5,6 +5,7 @@
>  #  Brotli was released on the website https://github.com/google/brotli.
>  #
>  #  Copyright (c) 2017 - 2020, Intel Corporation. All rights reserved.<BR>
> +#  Copyright (C) 2023 Advanced Micro Devices, Inc. All rights reserved.<BR>
>  #
>  #  SPDX-License-Identifier: BSD-2-Clause-Patent
>  #
> @@ -41,6 +42,10 @@
>    # Wrapper header files end #
>    brotli/c/common/dictionary.c
>    brotli/c/common/transform.c
> +  brotli/c/common/context.c
> +  brotli/c/common/platform.c
> +  brotli/c/common/constants.c
> +  brotli/c/common/shared_dictionary.c
>    brotli/c/dec/bit_reader.c
>    brotli/c/dec/decode.c
>    brotli/c/dec/huffman.c
> --
> 2.37.1.windows.1
>
>
>
>
>



[-- Attachment #1.2: Type: text/html, Size: 58850 bytes --]

[-- Attachment #2: memory.h --]
[-- Type: text/plain, Size: 271 bytes --]

/*****************************************************************************
 * Copyright (C) 2022 Advanced Micro Devices, Inc. All rights reserved.
 *****************************************************************************/

#include <BrotliDecUefiSupport.h>

[-- Attachment #3: intrin.h --]
[-- Type: text/plain, Size: 271 bytes --]

/*****************************************************************************
 * Copyright (C) 2022 Advanced Micro Devices, Inc. All rights reserved.
 *****************************************************************************/

#include <BrotliDecUefiSupport.h>