Re: Pull in pre-built library during edk2 build?

[Laszlo Ersek <lersek@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5280 bytes --]

On 04/06/17 20:07, Peter Hornyack wrote:
> I'd like to make an adjustment to the edk2 build (locally, not for
> upstream) and I'm hoping someone can offer some guidance.
> 
> My goal is to pre-build an edk2 library in a separate build process,
> then pull that library into the full build later on. Specifically I'm
> building my firmware image using OvmfPkgX64.dsc, but I want to build
> OpensslLib (CryptoPkg/Library/OpensslLib/OpensslLib.inf) in advance,
> then pull the resulting lib into the full build later. How can I
> achieve this?
> 
> In my build output I can see that when OpensslLib.inf is built, all of
> the openssl .c files are compiled into .obj files, then an ar command
> wraps those up into OpensslLib.lib. I want to pull those steps out and
> pre-build OpensslLib.lib, but I've been unable to find where/how the
> edk2 build grabs that .lib file and turns it into the final firmware
> image. I've reviewed the edk2 build documentation but still can't
> figure this out. Can anyone point me to the right place in the edk2
> build files where I can make this happen? Or perhaps is there an
> example of this already in the edk2 build that I can imitate?

(1) Apply the attached patch

  OvmfPkg: add USE_EXTERNAL_OPENSSL for external, binary OpenSSL

in your edk2 clone.

(Don't forget to pass --keep-cr to git-am.)

(2) Create a new directory somewhere, initialize it as a git repository,
and apply the attached patch

  ExternalSslPkg: create INF files for OpensslLib binaries

in it:

  export EXTERNAL_OPENSSL_DIR=...
  mkdir -pv -- "$EXTERNAL_OPENSSL_DIR"
  cd "$EXTERNAL_OPENSSL_DIR"
  git init
  git am --keep-cr ...

(3) Set another variable (customize as necessary):

  export TOOLCHAIN=GCC48

(4) Go to your normal edk2 clone, and run the following commands, for
initially building the OpenSSL binaries only, and installing them. This
assumes that OpenSSL has been embedded in the edk2 tree, as directed by

  CryptoPkg/Library/OpensslLib/OpenSSL-HOWTO.txt

This step has to be done only once.

  (
    ARCH_ID=(IA32 X64)
    ARCH_NAME=(Ia32 X64)

    SSL_FLAVOR=("" Crypto)
    SSL_TLS=(TRUE FALSE)

    SSLD=CryptoPkg/Library/OpensslLib
    source edksetup.sh
    set -e -u -C -x
    make -C "$EDK_TOOLS_PATH"

    for TARGET in DEBUG NOOPT RELEASE; do
      for ((ARCH_IDX=0; ARCH_IDX < ${#ARCH_ID[@]}; ARCH_IDX++)); do
        for ((SSL_IDX=0; SSL_IDX < ${#SSL_FLAVOR[@]}; SSL_IDX++)); do

          build \
            --arch="${ARCH_ID[ARCH_IDX]}" \
            --platform="OvmfPkg/OvmfPkg${ARCH_NAME[ARCH_IDX]}.dsc" \
            --module="$SSLD/OpensslLib${SSL_FLAVOR[SSL_IDX]}.inf" \
            --buildtarget="$TARGET" \
            --tagname="$TOOLCHAIN" \
            --define=TLS_ENABLE="${SSL_TLS[SSL_IDX]}"

          LIBF="Build/Ovmf${ARCH_NAME[ARCH_IDX]}/${TARGET}_${TOOLCHAIN}"
          LIBF="$LIBF/${ARCH_ID[ARCH_IDX]}"
          LIBF="$LIBF/$SSLD/OpensslLib${SSL_FLAVOR[SSL_IDX]}"
          LIBF="$LIBF/OUTPUT/OpensslLib${SSL_FLAVOR[SSL_IDX]}.lib"

          DSTF="$EXTERNAL_OPENSSL_DIR/ExternalSslPkg/Library/OpensslLib"
          DSTF="$DSTF/$TARGET/${ARCH_ID[ARCH_IDX]}"
          DSTF="$DSTF/OpensslLib${SSL_FLAVOR[SSL_IDX]}.lib"

          install -m 0644 -D -- "$LIBF" "$DSTF"
        done
      done
    done

    rm -r Build
  )

At the end of this step, the binaries will be available under
$EXTERNAL_OPENSSL_DIR, in addition to the INF files from step (2):

  ExternalSslPkg/Library/OpensslLib/DEBUG/IA32/OpensslLib.lib
  ExternalSslPkg/Library/OpensslLib/DEBUG/IA32/OpensslLibCrypto.lib
  ExternalSslPkg/Library/OpensslLib/DEBUG/X64/OpensslLib.lib
  ExternalSslPkg/Library/OpensslLib/DEBUG/X64/OpensslLibCrypto.lib
  ExternalSslPkg/Library/OpensslLib/NOOPT/IA32/OpensslLib.lib
  ExternalSslPkg/Library/OpensslLib/NOOPT/IA32/OpensslLibCrypto.lib
  ExternalSslPkg/Library/OpensslLib/NOOPT/X64/OpensslLib.lib
  ExternalSslPkg/Library/OpensslLib/NOOPT/X64/OpensslLibCrypto.lib
  ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
  ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf
  ExternalSslPkg/Library/OpensslLib/RELEASE/IA32/OpensslLib.lib
  ExternalSslPkg/Library/OpensslLib/RELEASE/IA32/OpensslLibCrypto.lib
  ExternalSslPkg/Library/OpensslLib/RELEASE/X64/OpensslLib.lib
  ExternalSslPkg/Library/OpensslLib/RELEASE/X64/OpensslLibCrypto.lib

If you wish, you can add and commit the lib files to the repository, or
save them for later by other means. Either case they have to stick
together with the INF files.

(5) Now build OVMF against the external libraries.

For this, we use PACKAGES_PATH (see
<https://github.com/tianocore/tianocore.github.io/wiki/Multiple_Workspace>).

Also note -D USE_EXTERNAL_OPENSSL in COMMON_OPTS, which utilizes the
patch from step (1).

This step can be repeated as many times as necessary. Two builds are
included below as examples.

  (
    export PACKAGES_PATH="$EXTERNAL_OPENSSL_DIR"
    source edksetup.sh

    COMMON_OPTS="-t GCC48 -n 12 -D USE_EXTERNAL_OPENSSL -D SMM_REQUIRE"
    COMMON_OPTS="$COMMON_OPTS -D SECURE_BOOT_ENABLE -D HTTP_BOOT_ENABLE"

    build $COMMON_OPTS \
      -a IA32 -p OvmfPkg/OvmfPkgIa32.dsc \
      -b NOOPT

    build $COMMON_OPTS \
      -a IA32 -a X64 -p OvmfPkg/OvmfPkgIa32X64.dsc \
      -D TLS_ENABLE \
      -b DEBUG
  )

Thanks
Laszlo

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-OvmfPkg-add-USE_EXTERNAL_OPENSSL-for-external-binary.patch --]
[-- Type: text/x-patch; name="0001-OvmfPkg-add-USE_EXTERNAL_OPENSSL-for-external-binary.patch", Size: 3197 bytes --]

From bd7fecb526fba849cd0fb0702f2614fe60649836 Mon Sep 17 00:00:00 2001
From: Laszlo Ersek <lersek@redhat.com>
Date: Thu, 6 Apr 2017 22:23:35 +0200
Subject: [PATCH] OvmfPkg: add USE_EXTERNAL_OPENSSL for external, binary
 OpenSSL

Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
---
 OvmfPkg/OvmfPkgIa32.dsc    | 8 ++++++++
 OvmfPkg/OvmfPkgIa32X64.dsc | 8 ++++++++
 OvmfPkg/OvmfPkgX64.dsc     | 8 ++++++++
 3 files changed, 24 insertions(+)

diff --git a/OvmfPkg/OvmfPkgIa32.dsc b/OvmfPkg/OvmfPkgIa32.dsc
index 6f515c9aa1aa..1eb635a68250 100644
--- a/OvmfPkg/OvmfPkgIa32.dsc
+++ b/OvmfPkg/OvmfPkgIa32.dsc
@@ -145,11 +145,19 @@ [LibraryClasses]
   DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
 
   IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+!ifdef $(USE_EXTERNAL_OPENSSL)
+!if $(TLS_ENABLE) == TRUE
+  OpensslLib|ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
+!else
+  OpensslLib|ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf
+!endif
+!else
 !if $(TLS_ENABLE) == TRUE
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
 !else
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf
 !endif
+!endif
 
 !if $(SECURE_BOOT_ENABLE) == TRUE
   PlatformSecureLib|OvmfPkg/Library/PlatformSecureLib/PlatformSecureLib.inf
diff --git a/OvmfPkg/OvmfPkgIa32X64.dsc b/OvmfPkg/OvmfPkgIa32X64.dsc
index 7d1676d122b7..06d29a73313f 100644
--- a/OvmfPkg/OvmfPkgIa32X64.dsc
+++ b/OvmfPkg/OvmfPkgIa32X64.dsc
@@ -150,11 +150,19 @@ [LibraryClasses]
   DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
 
   IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+!ifdef $(USE_EXTERNAL_OPENSSL)
+!if $(TLS_ENABLE) == TRUE
+  OpensslLib|ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
+!else
+  OpensslLib|ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf
+!endif
+!else
 !if $(TLS_ENABLE) == TRUE
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
 !else
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf
 !endif
+!endif
 
 !if $(SECURE_BOOT_ENABLE) == TRUE
   PlatformSecureLib|OvmfPkg/Library/PlatformSecureLib/PlatformSecureLib.inf
diff --git a/OvmfPkg/OvmfPkgX64.dsc b/OvmfPkg/OvmfPkgX64.dsc
index f288c8c98536..94b70290fcf7 100644
--- a/OvmfPkg/OvmfPkgX64.dsc
+++ b/OvmfPkg/OvmfPkgX64.dsc
@@ -150,11 +150,19 @@ [LibraryClasses]
   DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
 
   IntrinsicLib|CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf
+!ifdef $(USE_EXTERNAL_OPENSSL)
+!if $(TLS_ENABLE) == TRUE
+  OpensslLib|ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
+!else
+  OpensslLib|ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf
+!endif
+!else
 !if $(TLS_ENABLE) == TRUE
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf
 !else
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf
 !endif
+!endif
 
 !if $(SECURE_BOOT_ENABLE) == TRUE
   PlatformSecureLib|OvmfPkg/Library/PlatformSecureLib/PlatformSecureLib.inf
-- 
2.9.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-ExternalSslPkg-create-INF-files-for-OpensslLib-binar.patch --]
[-- Type: text/x-patch; name="0001-ExternalSslPkg-create-INF-files-for-OpensslLib-binar.patch", Size: 3687 bytes --]

From 4672a027f54c54574129f9c9cc947c80f7bc4d9f Mon Sep 17 00:00:00 2001
From: Laszlo Ersek <lersek@redhat.com>
Date: Thu, 6 Apr 2017 22:56:04 +0200
Subject: [PATCH] ExternalSslPkg: create INF files for OpensslLib binaries

Signed-off-by: Laszlo Ersek
---
 .../Library/OpensslLib/OpensslLibBin.inf           | 33 ++++++++++++++++++++++
 .../Library/OpensslLib/OpensslLibBinCrypto.inf     | 33 ++++++++++++++++++++++
 2 files changed, 66 insertions(+)
 create mode 100644 ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
 create mode 100644 ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf

diff --git a/ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf b/ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
new file mode 100644
index 000000000000..703fddb47606
--- /dev/null
+++ b/ExternalSslPkg/Library/OpensslLib/OpensslLibBin.inf
@@ -0,0 +1,33 @@
+## @file
+#  This module provides binary OpenSSL Library implementation.
+#
+#  Copyright (C) 2017, Red Hat, Inc.
+#  Copyright (c) 2010 - 2017, Intel Corporation. All rights reserved.<BR>
+#  This program and the accompanying materials are licensed and made available
+#  under the terms and conditions of the BSD License which accompanies this
+#  distribution.  The full text of the license may be found at
+#  http://opensource.org/licenses/bsd-license.php
+#
+#  THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR
+#  IMPLIED.
+#
+##
+
+[Defines]
+  INF_VERSION                    = 1.25
+  BASE_NAME                      = OpensslLibBin
+  FILE_GUID                      = d0d4d4cf-460c-4752-9c9b-6d821b2ffe49
+  MODULE_TYPE                    = BASE
+  VERSION_STRING                 = 1.0
+  LIBRARY_CLASS                  = OpensslLib
+
+[Binaries.IA32]
+  LIB|DEBUG/IA32/OpensslLib.lib|DEBUG
+  LIB|NOOPT/IA32/OpensslLib.lib|NOOPT
+  LIB|RELEASE/IA32/OpensslLib.lib|RELEASE
+
+[Binaries.X64]
+  LIB|DEBUG/X64/OpensslLib.lib|DEBUG
+  LIB|NOOPT/X64/OpensslLib.lib|NOOPT
+  LIB|RELEASE/X64/OpensslLib.lib|RELEASE
diff --git a/ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf b/ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf
new file mode 100644
index 000000000000..6848ea6f5037
--- /dev/null
+++ b/ExternalSslPkg/Library/OpensslLib/OpensslLibBinCrypto.inf
@@ -0,0 +1,33 @@
+## @file
+#  This module provides binary OpenSSL Library implementation.
+#
+#  Copyright (C) 2017, Red Hat, Inc.
+#  Copyright (c) 2010 - 2017, Intel Corporation. All rights reserved.<BR>
+#  This program and the accompanying materials are licensed and made available
+#  under the terms and conditions of the BSD License which accompanies this
+#  distribution.  The full text of the license may be found at
+#  http://opensource.org/licenses/bsd-license.php
+#
+#  THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR
+#  IMPLIED.
+#
+##
+
+[Defines]
+  INF_VERSION                    = 1.25
+  BASE_NAME                      = OpensslLibBinCrypto
+  FILE_GUID                      = 26788980-79b5-4e2f-8164-b4084f677c48
+  MODULE_TYPE                    = BASE
+  VERSION_STRING                 = 1.0
+  LIBRARY_CLASS                  = OpensslLib
+
+[Binaries.IA32]
+  LIB|DEBUG/IA32/OpensslLibCrypto.lib|DEBUG
+  LIB|NOOPT/IA32/OpensslLibCrypto.lib|NOOPT
+  LIB|RELEASE/IA32/OpensslLibCrypto.lib|RELEASE
+
+[Binaries.X64]
+  LIB|DEBUG/X64/OpensslLibCrypto.lib|DEBUG
+  LIB|NOOPT/X64/OpensslLibCrypto.lib|NOOPT
+  LIB|RELEASE/X64/OpensslLibCrypto.lib|RELEASE
-- 
2.9.3