EmulatorPkg build issue with macOS mojave and Xcode 10

[唐佳诚 <chasontang@warmbloom.com>]

Hi Dear EmulatorPkg Maintainer:
  I'm a newbie at this, I am sorry for the wrong format of the previous mail. There are some issue when building EmulatorPkg. Any inspiration or help will appreciated!

OS Environment: macOS mojave with Xcode 10 command line tool and installed macOS_SDK_headers_for_macOS_10.14.pkg
Download edk2-UDK2018.zip
Initializing workspace
> source ./edk2setup.sh

WORKSPACE: /Users/QianYun/Downloads/edk2-vUDK2018
EDK_TOOLS_PATH: /Users/QianYun/Downloads/edk2-vUDK2018/BaseTools
CONF_PATH: /Users/QianYun/Downloads/edk2-vUDK2018/Conf
Copying $EDK_TOOLS_PATH/Conf/build_rule.template
     to /Users/QianYun/Downloads/edk2-vUDK2018/Conf/build_rule.txt
Copying $EDK_TOOLS_PATH/Conf/tools_def.template
     to /Users/QianYun/Downloads/edk2-vUDK2018/Conf/tools_def.txt
Copying $EDK_TOOLS_PATH/Conf/target.template
     to /Users/QianYun/Downloads/edk2-vUDK2018/Conf/target.txt

> cd EmulatorPkg && ./build.sh

The BaseTools will build successfully, and EmulatorPkg will fail to build.

'import sitecustomize' failed; use -v for traceback
Build environment: Darwin-18.2.0-x86_64-i386-64bit
Build start time: 15:10:45, Jan.14 2019

WORKSPACE        = /Users/QianYun/Downloads/edk2-vUDK2018
ECP_SOURCE       = /Users/QianYun/Downloads/edk2-vUDK2018/EdkCompatibilityPkg
EDK_SOURCE       = /Users/QianYun/Downloads/edk2-vUDK2018/EdkCompatibilityPkg
EFI_SOURCE       = /Users/QianYun/Downloads/edk2-vUDK2018/EdkCompatibilityPkg
EDK_TOOLS_PATH   = /Users/QianYun/Downloads/edk2-vUDK2018/BaseTools
CONF_PATH        = /Users/QianYun/Downloads/edk2-vUDK2018/Conf
build: : warning: Tool chain [XCODE32] is not defined


build.py...
 : error 4000: Not available
	[XCODE32] not defined. No toolchain available for build!


- Failed -
Build end time: 15:10:46, Jan.14 2019
Build total time: 00:00:00

Edit EmulatorPkg/build.sh and ./build.sh again

diff --git a/build.sh.new b/build.sh
index 4f653d2..339c6b3 100755
--- a/build.sh.new
+++ b/build.sh
@@ -63,8 +63,17 @@ case `uname` in
         echo UnixPkg requires Snow Leopard or later OS
         exit 1
       else
-        HOST_TOOLS=XCODE5
-        TARGET_TOOLS=XCODE5
+        CLANG_VER=$(clang -ccc-host-triple x86_64-pc-win32-macho 2>&1 >/dev/null) || true
+        if [[ "$CLANG_VER" == *-ccc-host-triple* ]]
+        then
+        # only older versions of Xcode support -ccc-host-tripe, for newer versions
+        # it is -target
+          HOST_TOOLS=XCODE32
+          TARGET_TOOLS=XCODE5
+        else
+          HOST_TOOLS=XCODE32
+          TARGET_TOOLS=XCLANG
+        fi
       fi
       BUILD_NEW_SHELL="-D BUILD_NEW_SHELL"
       BUILD_FAT="-D BUILD_FAT"

However, this attempt still failed.

Building ... /Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/Host.inf [X64]
"clang" -target x86_64-pc-win32-macho -c -g -Os -Wall -Werror -Wextra -include AutoGen.h -funsigned-char -fno-ms-extensions -fno-stack-protector -fno-builtin -fshort-wchar -mno-implicit-float -mms-bitfields -Wno-unused-parameter -Wno-missing-braces -Wno-missing-field-initializers -Wno-tautological-compare -Wno-sign-compare -Wno-varargs -ftrap-function=undefined_behavior_has_been_optimized_away_by_clang -D NO_MSABI_VA_FUNCS  -I/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/X11IncludeHack -o /Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/EmulatorPkg/Unix/Host/Host/OUTPUT/./Host.obj -I/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/X64 -I/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host -I/Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/EmulatorPkg/Unix/Host/Host/DEBUG -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg/Include -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg/Include/X64 -I/Users/QianYun/Downloads/edk2-vUDK2018/MdeModulePkg -I/Users/QianYun/Downloads/edk2-vUDK2018/MdeModulePkg/Include -I/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg -I/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Include /Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/Host.c
"clang" -target x86_64-pc-win32-macho -c -g -Os -Wall -Werror -Wextra -include AutoGen.h -funsigned-char -fno-ms-extensions -fno-stack-protector -fno-builtin -fshort-wchar -mno-implicit-float -mms-bitfields -Wno-unused-parameter -Wno-missing-braces -Wno-missing-field-initializers -Wno-tautological-compare -Wno-sign-compare -Wno-varargs -ftrap-function=undefined_behavior_has_been_optimized_away_by_clang -D NO_MSABI_VA_FUNCS  -o /Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/ShellPkg/Library/UefiShellNetwork1CommandsLib/UefiShellNetwork1CommandsLib/OUTPUT/./AutoGen.obj -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Library/UefiShellNetwork1CommandsLib -I/Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/ShellPkg/Library/UefiShellNetwork1CommandsLib/UefiShellNetwork1CommandsLib/DEBUG -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg/Include -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg/Include/X64 -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Include -I/Users/QianYun/Downloads/edk2-vUDK2018/MdeModulePkg -I/Users/QianYun/Downloads/edk2-vUDK2018/MdeModulePkg/Include /Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/ShellPkg/Library/UefiShellNetwork1CommandsLib/UefiShellNetwork1CommandsLib/DEBUG/AutoGen.c
In file included from /Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/Host.c:15:
/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/Host.h:28:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
"clang" -target x86_64-pc-win32-macho -c -g -Os -Wall -Werror -Wextra -include AutoGen.h -funsigned-char -fno-ms-extensions -fno-stack-protector -fno-builtin -fshort-wchar -mno-implicit-float -mms-bitfields -Wno-unused-parameter -Wno-missing-braces -Wno-missing-field-initializers -Wno-tautological-compare -Wno-sign-compare -Wno-varargs -ftrap-function=undefined_behavior_has_been_optimized_away_by_clang -D NO_MSABI_VA_FUNCS  -o /Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/ShellPkg/Library/UefiShellDebug1CommandsLib/UefiShellDebug1CommandsLib/OUTPUT/./SerMode.obj -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Library/UefiShellDebug1CommandsLib/HexEdit -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Library/UefiShellDebug1CommandsLib/Edit -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Library/UefiShellDebug1CommandsLib/SmbiosView -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Library/UefiShellDebug1CommandsLib -I/Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/ShellPkg/Library/UefiShellDebug1CommandsLib/UefiShellDebug1CommandsLib/DEBUG -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg/Include -I/Users/QianYun/Downloads/edk2-vUDK2018/MdePkg/Include/X64 -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg -I/Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Include -I/Users/QianYun/Downloads/edk2-vUDK2018/MdeModulePkg -I/Users/QianYun/Downloads/edk2-vUDK2018/MdeModulePkg/Include /Users/QianYun/Downloads/edk2-vUDK2018/ShellPkg/Library/UefiShellDebug1CommandsLib/SerMode.c
1 error generated.
make: *** [/Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/EmulatorPkg/Unix/Host/Host/OUTPUT/Host.obj] Error 1


build.py...
 : error 7000: Failed to execute command
	make tbuild [/Users/QianYun/Downloads/edk2-vUDK2018/Build/Emulator/DEBUG_XCODE5/X64/EmulatorPkg/Unix/Host/Host]


build.py...
 : error F002: Failed to build module
	/Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg/Unix/Host/Host.inf [X64, XCODE5, DEBUG]

- Failed -
Build end time: 15:16:01, Jan.14 2019
Build total time: 00:00:39

After careful analysis, i find out that the include search path is wrong. clang will search some path below by default.

> clang++ -E -x c++ - -v < /dev/null

Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 409.12 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0 -stdlib=libc++ -fdeprecated-macro -fdebug-compilation-dir /Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg -ferror-limit 19 -fmessage-length 181 -stack-protector 1 -fblocks -fencode-extended-block-signature -fobjc-runtime=macosx-10.14.0 -fcxx-exceptions -fexceptions -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o - -x c++ -
clang -cc1 version 10.0.0 (clang-1000.11.45.5) default target x86_64-apple-darwin18.2.0
ignoring nonexistent directory "/usr/include/c++/v1"
#include "..." search starts here:
#include <...> search starts here:
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 373 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2

But -target x86_64-pc-win32-macho will working in cross-compilation mode. it will search this path and stdio.h will not found

> clang++ -target x86_64-pc-win32-macho -E -x c++ - -v < /dev/null

Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-pc-windows-macho
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-pc-windows-macho -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -target-linker-version 409.12 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0 -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/include -fdeprecated-macro -fdebug-compilation-dir /Users/QianYun/Downloads/edk2-vUDK2018/EmulatorPkg -ferror-limit 19 -fmessage-length 181 -fno-use-cxa-atexit -fms-extensions -fms-compatibility -fms-compatibility-version=19.11 -std=c++14 -fdelayed-template-parsing -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fseh-exceptions -fdiagnostics-show-option -fcolor-diagnostics -o - -x c++ -
clang -cc1 version 10.0.0 (clang-1000.11.45.5) default target x86_64-apple-darwin18.2.0
#include "..." search starts here:
#include <...> search starts here:
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 366 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2

We can obviously see that the XcodeDefault.xctoolchain/usr/lib/clang/10.0.0/include is not include the stdio.h, so that, preview build error is stdio.h not found.

And there is another thing that puzzles me. The EmulatorPkg/Unix/Host/Host.h define some name mangle, and include stdio.h(this file include endian.h), That will cause the macro definition to be redefined.
This thing made me spend a lot of energy.Is there are anyone can solve my confusion.
Thanks,
ChasonTang