Re: [edk2-devel] [PATCH v5 4/5] BaseTools: Compile AML bytecode arrays into .obj file

["Masahisa Kojima" <masahisa.kojima@linaro.org>]

Hi Leif,

> > Masahisa - since Ard is still on holiday, could you create a patch and
> > send out for me to review? Either one of the files needs to be
> > renamed, or we need to move the .asl files (Emmc.asl and Optee.asl)
> > into a subdirectory.

Probably I'm missing something, but my build for Developerbox platform is
successful, with the latest edk2/edk2-platforms as of today.

My build option is:
---
export TARGET=Platform/Socionext/DeveloperBox/DeveloperBox.dsc
export PROFILE=DEBUG
export ARCH=AARCH64
export TOOL_CHAIN_TAG=GCC5

build -n $NUM_CPUS -a $ARCH -b $PROFILE -t $TOOL_CHAIN_TAG -p $TARGET
-D SECURE_BOOT_ENABLE=TRUE -D X64EMU_ENABLE=TRUE -D TPM2_ENABLE=TRUE
---

GCC version is "gcc-linaro-6.4.1-2017.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-".

Some Output directory information:
~/src/uefi/Build/DeveloperBox$ find .|grep -i emmc
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.iii
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.obj.deps
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.iiii
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.obj
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.aml.deps
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.aml
./DEBUG_GCC5/AARCH64/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/PlatformDxe/OUTPUT/Emmc.i

Regards,
Masahisa

On Tue, 28 Jul 2020 at 09:40, Gao, Liming <liming.gao@intel.com> wrote:
>
> Pierre:
>   Thanks for your investigation. For now, the module will rename source file to avoid the file conflict. Can you submit one BZ for BaseTools to enhance the logic for the autogen source file? This needs more discussion.
>
>   Yes. I agree to place the generated source file into $(DEBUG_DIR). You can make another patch for this change.
>
> Thanks
> Liming
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois
> Sent: 2020年7月28日 1:10
> To: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Leif Lindholm <leif@nuviainc.com>; Masahisa Kojima <masahisa.kojima@linaro.org>
> Cc: Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Ard Biesheuvel <Ard.Biesheuvel@arm.com>
> Subject: Re: [edk2-devel] [PATCH v5 4/5] BaseTools: Compile AML bytecode arrays into .obj file
>
> Hello,
>
> Liming:
> I didn't find anything preventing from having a .vfr and .c file with the same name in the same scope.
>
> Everybody:
> The .c files generated (from either .vfr or .asl) are generated in the Build/..../OUTPUT/ directory, with their original subdirectory stripped. In the example below, the .c, .vfr and .asl files have been placed in subdirectories in edk2/edk2-platforms. The recipes that have been generated are:
> $(OUTPUT_DIR)/Emmc.obj : $(OUTPUT_DIR)/AslDir/Emmc.c (from Emmc.asl) $(OUTPUT_DIR)/Ip4Config2.obj : $(DEBUG_DIR)/VfrDir/Ip4Config2.c (from Ip4Config2.vfr) $(OUTPUT_DIR)/CDir/Emmc.obj : $(WORKSPACE)/edk2-platforms/Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/CDir/Emmc.c (from Emmc.c)
>
> The only subdirectory that remains is the one from the .c files. Indeed, the build system might detects subdirectories from the .inf file location. As the .c file is generated in the OUTPUT directory where there is no .inf file, it gets stripped.
> This means that if a subdirectory is created, it is for the .c file, but I don't think we should modify this.
>
> Adding a pre/postfix to the generated .c filename seems a better approach to me. It would require modifying build_rule.template, and choose which pre/postfix to add.
>
> Another point is that the .c file generated from the .c file is placed in the $(DEBUG_DIR), when the one coming from the .asl file is placed in $(OUTPUT_DIR). Maybe it should be modified to $(DEBUG_DIR) aswell.
>
> Regards,
> Pierre
>
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Liming Gao via groups.io
> Sent: Monday, July 27, 2020 3:21 PM
> To: Leif Lindholm <leif@nuviainc.com>; devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com>; Masahisa Kojima <masahisa.kojima@linaro.org>
> Cc: Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Ard Biesheuvel <Ard.Biesheuvel@arm.com>
> Subject: Re: [edk2-devel] [PATCH v5 4/5] BaseTools: Compile AML bytecode arrays into .obj file
>
> Leif:
>   VFR file has the similar case. VFR is converted to xxx.c, then compile it to obj file.
>
> Bob:
>   Has BaseTools such detection if VFR and C source file have the same file name?
>
> Thanks
> Liming
> > -----Original Message-----
> > From: Leif Lindholm <leif@nuviainc.com>
> > Sent: Monday, July 27, 2020 9:58 PM
> > To: devel@edk2.groups.io; pierre.gondois@arm.com; Masahisa Kojima
> > <masahisa.kojima@linaro.org>
> > Cc: sami.mujawar@arm.com; tomas.pilar@arm.com; Feng, Bob C
> > <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Ard
> > Biesheuvel <ard.biesheuvel@arm.com>
> > Subject: Re: [edk2-devel] [PATCH v5 4/5] BaseTools: Compile AML
> > bytecode arrays into .obj file
> >
> > Hi Pierre, (+Masahisa)
> >
> > This commit (0a4aa20e8d44) made for an exciting start to my week.
> >
> > Socionext's Developerbox failed to build for me, with the spectacular
> > error message:
> >
> > /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/bin/ld:
> > DWARF error: could not find abbrev number 5912
> > /tmp/ccKt4gaM.ltrans0.ltrans.o: in function `RegisterDevices':
> > <artificial>:(.text.RegisterDevices+0xb0): undefined reference to
> > `RegisterEmmc'
> >
> > GCC49 (without lto) and CLANG38 profiles give much the same result,
> > with slightly less esoteric messages.
> >
> > The reason for this turned out to be that edk2-platforms
> > Silicon/Socionext/SynQuacer/Drivers/PlatformDxe/ has both an Emmc.asl
> > and an Emmc.c file, which after this patch both generate an Emmc.obj
> > in the same output directory.
> >
> > I think the correct course of action is to fix this in the SynQuacer
> > driver, but I am reporting it here so we get it logged in the list
> > archives.
> >
> > It would of course be good if the build system could detect and warn
> > over cases like this, rather than silently overwriting existing object
> > files.
> >
> > Masahisa - since Ard is still on holiday, could you create a patch and
> > send out for me to review? Either one of the files needs to be
> > renamed, or we need to move the .asl files (Emmc.asl and Optee.asl)
> > into a subdirectory.
> >
> > Best Regards,
> >
> > Leif
> >
> > On Wed, Jul 01, 2020 at 15:06:03 +0100, PierreGondois wrote:
> > > From: Pierre Gondois <pierre.gondois@arm.com>
> > >
> > > The AmlToHex script and Posix/WindowsLike wrappers convert an AML
> > > file to a .hex file, containing a C array storing AML bytecode. This
> > > ".hex" file can then be included in a C file, allowing to access the
> > > AML bytecode from this C file.
> > >
> > > The EDK2 build system doesn't allow to a depict dependency orders
> > > between files of different languages. For instance, in a module
> > > containing a ".c" file and a ".asl", the ".c"
> > > file may or may not be built prior to the ".asl" file.
> > > This prevents any inclusion of a generated ".hex" in a ".c" file
> > > since this later ".hex" file may or may not have been created yet.
> > >
> > > This patch modifies the AmlToC script to generate a C file instead
> > > of a ".hex" file.
> > > It also adds the generation of an intermediate ".amli" file when
> > > compiling an ASL file, and adds a rule to convert this ".amli" to a
> > > C file.
> > >
> > > This allows to generate a C file containing the AML bytecode from an
> > > ASL file. This C file will then be handled by the EDK2 build system
> > > to generate an object file.
> > > Thus, no file inclusion will be required anymore. The C file
> > > requiring the AML bytecode as a C array, and the ASL file, will be
> > > compiled independently. The C array must be defined as an external
> > > symbol. The linker is resolving the reference to the C array symbol.
> > >
> > > To summarize, the flow goes as:
> > >  -1. ASL file is compiled to AML;
> > >  -2. AML file is copied to a ".amli" intermediate file;  -3. EDK2
> > > build system applies the rule relevant to ".amli"
> > >      files. This is, calling the "AmlToC" script, generating
> > >      a C file from the ".amli" file;  -4. EDK2 build system applies
> > > the rule relevant to C files.
> > >      This is creating an object file.
> > >  -5. EDK2 build system links the object file containing the
> > >      AML bytecode with the object file requiring it.
> > >
> > > Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com>
> > > Suggested-by: Tomas Pilar <Tomas.Pilar@arm.com>
> > > ---
> > >
> > > The changes can be seen at:
> > > https://github.com/PierreARM/edk2/commits/803_Compile_AML_bytecode_a
> > > rray_into_OBJ_file_v5
> > >
> > > Notes:
> > >     v1:
> > >      - Add a new rule to the build_rule.template file to
> > >        generate ".obj" files from .asl files, and modify
> > >        the AmlToC script accordingly. [Pierre]
> > >     v2:
> > >      - Restrict the rule to DXE_DRIVER. This allows to build
> > >        the OvmfPkg, which was not the case in v1. [Pierre]
> > >     v3:
> > >      - Changed "Signed-off-by" to "Suggested-by". [Bob]
> > >     v4:
> > >     - No modification. Re-sending the patch with base64
> > >       encoding to conserve the right line endings. [Bob]
> > >     v5:
> > >      - No modification. [Pierre]
> > >
> > >  BaseTools/Conf/build_rule.template       | 15 +++-
> > >  BaseTools/Source/Python/AmlToC/AmlToC.py | 82 ++++++++------------
> > >  2 files changed, 47 insertions(+), 50 deletions(-)
> > >
> > > diff --git a/BaseTools/Conf/build_rule.template
> > > b/BaseTools/Conf/build_rule.template
> > > index
> > > 0822b681fcd9f61c6508e6f93ffc31fa70fd7059..c034869915914936e28f64a6aa
> > > dba08e0169da44 100755
> > > --- a/BaseTools/Conf/build_rule.template
> > > +++ b/BaseTools/Conf/build_rule.template
> > > @@ -419,6 +419,7 @@
> > >
> > >      <OutputFile>
> > >          $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml
> > > +        $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.amli
> > >
> > >      <ExtraDependency>
> > >          $(MAKE_FILE)
> > > @@ -428,14 +429,24 @@
> > >          "$(ASLPP)" $(DEPS_FLAGS) $(ASLPP_FLAGS) $(INC) /I${s_path}
> > > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.i >
> > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii
> > >          Trim --source-code -l -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii
> > >          "$(ASL)" $(ASL_FLAGS) $(ASL_OUTFLAGS)${dst} $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii
> > > -        -AmlToHex $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml
> > > +        $(CP) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml
> > > + $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.amli
> > >
> > >      <Command.GCC>
> > >          Trim --asl-file --asl-deps -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.i -i $(INC_LIST) ${src}
> > >          "$(ASLPP)" $(DEPS_FLAGS) $(ASLPP_FLAGS) $(INC) -I${s_path}
> > > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.i >
> > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii
> > >          Trim --source-code -l -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii
> > >          "$(ASL)" $(ASL_FLAGS) $(ASL_OUTFLAGS)${dst} $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii
> > > -        -AmlToHex $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml
> > > +        $(CP) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml
> > > + $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.amli
> > > +
> > > +[Acpi-Machine-Language-File-to-C.DXE_DRIVER]
> > > +    <InputFile>
> > > +        ?.amli
> > > +
> > > +    <OutputFile>
> > > +        ${s_path}(+)${s_base}.c
> > > +
> > > +    <Command>
> > > +        -AmlToC ${src}
> > >
> > >  [C-Code-File.AcpiTable]
> > >      <InputFile>
> > > diff --git a/BaseTools/Source/Python/AmlToC/AmlToC.py
> > > b/BaseTools/Source/Python/AmlToC/AmlToC.py
> > > index
> > > 643db2910e37acfdd80ac18d288c921320a79ce1..346de7159de702d860bbd809dd
> > > be8175f1493cfb 100644
> > > --- a/BaseTools/Source/Python/AmlToC/AmlToC.py
> > > +++ b/BaseTools/Source/Python/AmlToC/AmlToC.py
> > > @@ -1,9 +1,9 @@
> > >  ## @file
> > >  #
> > > -# Convert an AML file to a .hex file containing the AML bytecode
> > > stored in a
> > > +# Convert an AML file to a .c file containing the AML bytecode
> > > +stored in a
> > >  # C array.
> > > -# By default, "Tables\Dsdt.aml" will generate "Tables\Dsdt.hex".
> > > -# "Tables\Dsdt.hex" will contain a C array named "dsdt_aml_code"
> > > that contains
> > > +# By default, "Tables\Dsdt.aml" will generate "Tables\Dsdt.c".
> > > +# "Tables\Dsdt.c" will contain a C array named "dsdt_aml_code" that
> > > +contains
> > >  # the AML bytecode.
> > >  #
> > >  # Copyright (c) 2020, ARM Limited. All rights reserved.<BR> @@
> > > -17,31 +17,26 @@ from Common.BuildToolError import *  import sys
> > > import os
> > >
> > > +__description__ = """
> > > +Convert an AML file to a .c file containing the AML bytecode stored
> > > +in a C array. By default, Tables\Dsdt.aml will generate Tables\Dsdt.c.
> > > +Tables\Dsdt.c will contain a C array named "dsdt_aml_code" that
> > > +contains the AML bytecode.
> > > +"""
> > > +
> > >  ## Parse the command line arguments.
> > >  #
> > >  # @retval A argparse.NameSpace instance, containing parsed values.
> > >  #
> > >  def ParseArgs():
> > >      # Initialize the parser.
> > > -    Parser = argparse.ArgumentParser(
> > > -      description="Convert an AML file to a .hex file containing the AML " + \
> > > -                  "bytecode stored in a C array. By default, " + \
> > > -                  "\"Tables\\Dsdt.aml\" will generate" + \
> > > -                  "\"Tables\\Dsdt.hex\". \"Tables\\Dsdt.hex\" will " + \
> > > -                  "contain a C array named \"dsdt_aml_code\" that " + \
> > > -                  "contains the AML bytecode."
> > > -      )
> > > +    Parser = argparse.ArgumentParser(description=__description__)
> > >
> > >      # Define the possible arguments.
> > > -    Parser.add_argument(
> > > -      dest="InputFile",
> > > -      help="Path to an input AML file to generate a .hex file from."
> > > -      )
> > > -    Parser.add_argument(
> > > -      "-o", "--out-dir", dest="OutDir",
> > > -      help="Output directory where the .hex file will be generated. " + \
> > > -           "Default is the input file's directory."
> > > -      )
> > > +    Parser.add_argument(dest="InputFile",
> > > +                        help="Path to an input AML file to generate a .c file from.")
> > > +    Parser.add_argument("-o", "--out-dir", dest="OutDir",
> > > +                        help="Output directory where the .c file
> > > + will be generated. Default is the input file's directory.")
> > >
> > >      # Parse the input arguments.
> > >      Args = Parser.parse_args()
> > > @@ -55,9 +50,7 @@ def ParseArgs():
> > >          with open(Args.InputFile, "rb") as fIn:
> > >              Signature = str(fIn.read(4))
> > >              if ("DSDT" not in Signature) and ("SSDT" not in Signature):
> > > -                EdkLogger.info("Invalid file type. " + \
> > > -                                "File does not have a valid " + \
> > > -                                "DSDT or SSDT signature: %s" % Args.InputFile)
> > > +                EdkLogger.info("Invalid file type. File does not
> > > + have a valid DSDT or SSDT signature: {}".format(Args.InputFile))
> > >                  return None
> > >
> > >      # Get the basename of the input file.
> > > @@ -66,42 +59,39 @@ def ParseArgs():
> > >
> > >      # If no output directory is specified, output to the input directory.
> > >      if not Args.OutDir:
> > > -        Args.OutputFile = os.path.join(
> > > -          os.path.dirname(Args.InputFile),
> > > -          BaseName + ".hex"
> > > -          )
> > > +        Args.OutputFile = os.path.join(os.path.dirname(Args.InputFile),
> > > +                                       BaseName + ".c")
> > >      else:
> > >          if not os.path.exists(Args.OutDir):
> > >              os.mkdir(Args.OutDir)
> > > -        Args.OutputFile = os.path.join(Args.OutDir, BaseName + ".hex")
> > > +        Args.OutputFile = os.path.join(Args.OutDir, BaseName +
> > > + ".c")
> > >
> > >      Args.BaseName = BaseName
> > >
> > >      return Args
> > >
> > > -## Convert an AML file to a .hex file containing the AML bytecode
> > > stored
> > > +## Convert an AML file to a .c file containing the AML bytecode
> > > +stored
> > >  #  in a C array.
> > >  #
> > >  # @param  InputFile     Path to the input AML file.
> > > -# @param  OutputFile    Path to the output .hex file to generate.
> > > +# @param  OutputFile    Path to the output .c file to generate.
> > >  # @param  BaseName      Base name of the input file.
> > > -#                       This is also the name of the generated .hex file.
> > > +#                       This is also the name of the generated .c file.
> > >  #
> > > -def AmlToHex(InputFile, OutputFile, BaseName):
> > > +def AmlToC(InputFile, OutputFile, BaseName):
> > >
> > > -    MacroName = "__{}_HEX__".format(BaseName.upper())
> > >      ArrayName = BaseName.lower() + "_aml_code"
> > > +    FileHeader =\
> > > +"""
> > > +// This file has been generated from:
> > > +//   -Python script: {}
> > > +//   -Input AML file: {}
> > > +
> > > +"""
> > >
> > >      with open(InputFile, "rb") as fIn, open(OutputFile, "w") as fOut:
> > >          # Write header.
> > > -        fOut.write("// This file has been generated from:\n" + \
> > > -                   "// \tPython script: " + \
> > > -                   os.path.abspath(__file__) + "\n" + \
> > > -                   "// \tInput AML file: " + \
> > > -                   os.path.abspath(InputFile) + "\n\n" + \
> > > -                   "#ifndef {}\n".format(MacroName) + \
> > > -                   "#define {}\n\n".format(MacroName)
> > > -                   )
> > > +        fOut.write(FileHeader.format(os.path.abspath(InputFile),
> > > + os.path.abspath(__file__)))
> > >
> > >          # Write the array and its content.
> > >          fOut.write("unsigned char {}[] = {{\n  ".format(ArrayName))
> > > @@ -115,15 +105,12 @@ def AmlToHex(InputFile, OutputFile, BaseName):
> > >              byte = fIn.read(1)
> > >          fOut.write("\n};\n")
> > >
> > > -        # Write footer.
> > > -        fOut.write("#endif // {}\n".format(MacroName))
> > > -
> > >  ## Main method
> > >  #
> > >  # This method:
> > >  #   1-  Initialize an EdkLogger instance.
> > >  #   2-  Parses the input arguments.
> > > -#   3-  Converts an AML file to a .hex file containing the AML bytecode stored
> > > +#   3-  Converts an AML file to a .c file containing the AML bytecode stored
> > >  #       in a C array.
> > >  #
> > >  # @retval 0     Success.
> > > @@ -139,10 +126,9 @@ def Main():
> > >          if not CommandArguments:
> > >              return 1
> > >
> > > -        # Convert an AML file to a .hex file containing the AML bytecode stored
> > > +        # Convert an AML file to a .c file containing the AML
> > > + bytecode stored
> > >          # in a C array.
> > > -        AmlToHex(CommandArguments.InputFile, CommandArguments.OutputFile,
> > > -                          CommandArguments.BaseName)
> > > +        AmlToC(CommandArguments.InputFile,
> > > + CommandArguments.OutputFile, CommandArguments.BaseName)
> > >      except Exception as e:
> > >          print(e)
> > >          return 1
> > > --
> > > 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)'
> > >
> > >
>
>
>
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
>
> 
>