Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library

["Michael D Kinney" <michael.d.kinney@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 15059 bytes --]

Hi Timothy,

This is an interesting bug report because it looks like something that should be caught by EDK II CI

MdeModulePkg.dsc does list the Brotli lib in a [Components] section, so it will always be built.

[Components.IA32, Components.X64, Components.ARM, Components.AARCH64]
  MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
  MdeModulePkg/Library/LzmaCustomDecompressLib/LzmaCustomDecompressLib.inf
  MdeModulePkg/Library/VarCheckUefiLib/VarCheckUefiLib.inf
  MdeModulePkg/Core/Dxe/DxeMain.inf {
    <LibraryClasses>
      NULL|MdeModulePkg/Library/DxeCrc32GuidedSectionExtractLib/DxeCrc32GuidedSectionExtractLib.inf
  }

However, a lib is allowed to have unresolved externals such as calling other libraries.  So unless the library is linked to a module, we do
not know if all the externals are resolved.

I am wondering if we need to expand package DSC builds to make sure all libraries that are built are also used in
at least one module with that package build.

You were able to produce this failure by attempting to use Brotli with the DxeIpl module, so the unresolved
externals that are resolved by the missing source files were not noticed until this used case.  For example MdeModulePkg.dsc
could be extended to DxeIpl with different NULL library instances for the different decompressors.

Mike

From: Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com>
Sent: Monday, January 30, 2023 4:50 PM
To: Gao, Liming <gaoliming@byosoft.com.cn>; devel@edk2.groups.io; He, Jiangang <Jiangang.He@amd.com>; Chang, Abner <Abner.Chang@amd.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Bi, Dandan <dandan.bi@intel.com>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com>
Subject: Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library


[AMD Official Use Only - General]


The failure can be replicated with VS2019 and GCC5.

  *   GCC5:

     *   /usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: warning: relocation against `_kBrotliPrefixCodeRanges' in read-only section `.text.BrotliUefiDecompress'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `DecodeLiteralBlockSwitch':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `DecodeCommandBlockSwitch':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `DecodeDistanceBlockSwitch':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `SafeDecodeLiteralBlockSwitch':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `SafeDecodeCommandBlockSwitch':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `SafeDecodeDistanceBlockSwitch':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `BrotliDecoderDestroyInstance':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/state.c:136: undefined reference to `BrotliSharedDictionaryDestroyInstance'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `BrotliUefiDecompress':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/state.c:87: undefined reference to `BrotliSharedDictionaryCreateInstance'
/usr/bin/ld: /tmp/cc7sinRE.ltrans0.ltrans.o: in function `BrotliUefiDecompress':
/home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'
/usr/bin/ld: /home/XXX/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'
/usr/bin/ld: warning: creating DT_TEXTREL in a PIE
collect2: error: ld returned 1 exit status
...

build.py...
 : error F002: Failed to build module
      /home/XXX/edk2-stable202211/MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf [X64, GCC5, RELEASE]

  *   VS2019:

     *   BBrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliDefaultFreeFunc

BrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliDefaultAllocFunc

BrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliSharedDictionaryCreateInstance

BrotliDecompressLib.lib(state.obj) : error LNK2001: unresolved external symbol BrotliSharedDictionaryDestroyInstance

BrotliDecompressLib.lib(decode.obj) : error LNK2001: unresolved external symbol _kBrotliPrefixCodeRanges

BrotliDecompressLib.lib(decode.obj) : error LNK2001: unresolved external symbol BrotliSharedDictionaryAttach

BrotliDecompressLib.lib(decode.obj) : error LNK2001: unresolved external symbol _kBrotliContextLookupTable

d:\XXX\Build\OvmfX64\RELEASE_VS2019\X64\MdeModulePkg\Core\DxeIplPeim\DxeIpl\DEBUG\DxeIpl.dll : fatal error LNK1120: 7 unresolved externals

...

build.py...

 : error 7000: Failed to execute command

        C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\Hostx86\x86\nmake.exe /nologo tbuild [d:\XXX\Build\OvmfX64\RELEASE_VS2019\X64\MdeModulePkg\Core\DxeIplPeim\DxeIpl]



--
Timothy
________________________________
From: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
Sent: Saturday, January 28, 2023 11:35 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com<mailto:Kuei-Hung.Lin@amd.com>>; He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>; Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>; 'Kinney, Michael D' <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: 'Bi, Dandan' <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>
Subject: 回复: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


Timothy:

 Thanks for your report. This is a real issue in Brotli decompress library. Which tool chain has been verified? GCC5 or VS2019 or Both?



Thanks

Liming

发件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> 代表 Lin, Kuei-Hung (Timothy) via groups.io
发送时间: 2023年1月13日 6:18
收件人: He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>; Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
抄送: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>
主题: Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]



This patch tried to fix following build failure when linking the Brotli decompress lib.



Steps to replicate the issue:

1 - Attach a NULL lib to DxeIpl.inf in any project's .DSC, say OvmfPkgX64.dsc

  MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf {

    <LibraryClasses>

      NULL|MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf

  }

2 - Follow the EDKII'2 package build steps to build that project. Build.py would compliains with following errors.

...

/home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:879: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'

/usr/bin/ld: /tmp/ccKzzHN0.ltrans0.ltrans.o: in function `DecodeCommandBlockSwitch':

...

/usr/bin/ld: /tmp/ccKzzHN0.ltrans0.ltrans.o: in function `BrotliUefiDecompress':

/home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:898: undefined reference to `_kBrotliPrefixCodeRanges'

/usr/bin/ld: /home/edk2/edk2-stable202211/MdeModulePkg/Library/BrotliCustomDecompressLib/brotli/c/dec/decode.c:1193: undefined reference to `_kBrotliContextLookupTable'



--

Timothy









________________________________

From: He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>
Sent: Thursday, January 12, 2023 8:08 AM
To: Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Lin, Kuei-Hung (Timothy) <Kuei-Hung.Lin@amd.com<mailto:Kuei-Hung.Lin@amd.com>>
Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: RE: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]



Timothy will answer the question.



From: Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>
Sent: Thursday, January 12, 2023 10:05 AM
To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; He, Jiangang <Jiangang.He@amd.com<mailto:Jiangang.He@amd.com>>
Subject: Re: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



[AMD Official Use Only - General]



Hi Mike,

CC Jiangang, he can give the clear answer.



Thanks

Abner





Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________

From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Sent: Thursday, January 12, 2023 11:23:48 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Chang, Abner <Abner.Chang@amd.com<mailto:Abner.Chang@amd.com>>
Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kirkendall, Garrett <Garrett.Kirkendall@amd.com<mailto:Garrett.Kirkendall@amd.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: RE: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library



Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


Hi Abner,

What was the failure before this change?

This looks like something that would always break the build of source files were missing.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Chang, Abner via groups.io
> Sent: Wednesday, January 11, 2023 9:08 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
> Cc: Bi, Dandan <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Garrett Kirkendall <garrett.kirkendall@amd.com<mailto:garrett.kirkendall@amd.com>>;
> Abner Chang <abner.chang@amd.com<mailto:abner.chang@amd.com>>
> Subject: [edk2-devel] [PATCH V2] MdeModulePkg/Decompress: Add missing source file to Brotli library
>
> From: Abner Chang <abner.chang@amd.com<mailto:abner.chang@amd.com>>
>
> Add missing source files to Brotli library INF.
>
> Signed-off-by: Jiangang He <jiangang.he@amd.com<mailto:jiangang.he@amd.com>>
> Cc: Dandan Bi <dandan.bi@intel.com<mailto:dandan.bi@intel.com>>
> Cc: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Cc: Garrett Kirkendall <garrett.kirkendall@amd.com<mailto:garrett.kirkendall@amd.com>>
> Cc: Abner Chang <abner.chang@amd.com<mailto:abner.chang@amd.com>>
> ---
>  .../BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf  | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> b/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> index 525e92408d6..244c2e35023 100644
> --- a/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> +++ b/MdeModulePkg/Library/BrotliCustomDecompressLib/BrotliCustomDecompressLib.inf
> @@ -5,6 +5,7 @@
>  #  Brotli was released on the website https://github.com/google/brotli.
>  #
>  #  Copyright (c) 2017 - 2020, Intel Corporation. All rights reserved.<BR>
> +#  Copyright (C) 2023 Advanced Micro Devices, Inc. All rights reserved.<BR>
>  #
>  #  SPDX-License-Identifier: BSD-2-Clause-Patent
>  #
> @@ -41,6 +42,10 @@
>    # Wrapper header files end #
>    brotli/c/common/dictionary.c
>    brotli/c/common/transform.c
> +  brotli/c/common/context.c
> +  brotli/c/common/platform.c
> +  brotli/c/common/constants.c
> +  brotli/c/common/shared_dictionary.c
>    brotli/c/dec/bit_reader.c
>    brotli/c/dec/decode.c
>    brotli/c/dec/huffman.c
> --
> 2.37.1.windows.1
>
>
>
>
>



[-- Attachment #2: Type: text/html, Size: 93032 bytes --]