From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.61]) by mx.groups.io with SMTP id smtpd.web10.15093.1594905515082672553 for ; Thu, 16 Jul 2020 06:18:35 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BkTVx8kM; spf=pass (domain: redhat.com, ip: 205.139.110.61, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594905514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cji+KWT6c0D0HTLi7D1Z9Lb4IpgQrqTdbFq4FgqLnPs=; b=BkTVx8kMLN4ICjBgOZ7vUL54LSQk73rZb4dzJJDTgBtFD0ryNv4m6kUFCfoz9OOos4OLFD XXL9eJ0yAnyeJVdbt87lfo0EJjjjNXcsWLeChCLqMN37poXCsA8QMNFIrj+tCgIq4lw3Et J0vMJMFX1LOOKuiicH0Q2kg9bN2qMo0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-112-KIQVgNahPcq7pDk9U_JlVg-1; Thu, 16 Jul 2020 09:18:15 -0400 X-MC-Unique: KIQVgNahPcq7pDk9U_JlVg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A9F108027F3; Thu, 16 Jul 2020 13:18:14 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-112-123.ams2.redhat.com [10.36.112.123]) by smtp.corp.redhat.com (Postfix) with ESMTP id E1A0A5C1C3; Thu, 16 Jul 2020 13:18:13 +0000 (UTC) Subject: Re: [edk2-devel] File in the build question. I'm working on a smart git grep To: devel@edk2.groups.io, afish@apple.com References: From: "Laszlo Ersek" Message-ID: <39884c36-c42d-8718-f14b-d638e0cd1d69@redhat.com> Date: Thu, 16 Jul 2020 15:18:13 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=lersek@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 07/16/20 03:41, Andrew Fish via groups.io wrote: > I think I mentioned before on the list that I think a smart git grep that only scopes to a give target would be useful. For our internal repo we have lots of targets and lots of generations of platforms so lots of code that hits in a git grep that is not in the platform. > > In thinking about this it looks to me like the Build/**/AutoGenTimeStamp (module build files) files and the Build/**/AutoGen (global build files) contain a list of all the files for a project. > > I wrote some python that takes the build target (OvmfX64 for example) parses all the AutoGenTimeStamp and AutoGen files merges then together and removes the workspace path. All the other arguments are passed to `git grep` and only lines that match the file set are displayed. > > My questions are: > 1) Did I extract all the files? I don't know enough of BaseTools to answer the question :) > 2) Are people interested in this feature? It seems useful. :) > 3) Any ideas on how to hook it in? I just dropped in the root of the git tree for development. We usually place such utilities under BaseTools/Scripts/, I think (e.g. PatchCheck.py, SetupGit.py, GetMaintainer.py). > 4) Any other cool ideas of what we could do with a file list of the project? Perhaps introduce a more focused utility that only produces the file list (on stdout) so people could pass the list to whatever commands they desire? Thanks! Laszlo > > Example (to give people a feel for how it works): > $ time ./ggrep.py OvmfX64 CC_FLAGS | wc -l > 42 > > real 0m0.318s > user 0m0.243s > sys 0m0.576s > $ time git grep CC_FLAGS | wc -l > 479 > > real 0m0.110s > user 0m0.148s > sys 0m0.525s > $ ./ggrep.py OvmfX64 CC_FLAGS > OvmfX64 only: git grep CC_FLAGS > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:96: MSFT:*_*_*_CC_FLAGS = /wd4090 > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:100: RVCT:*_*_ARM_CC_FLAGS = -JCryptoPkg/Include --diag_remark=1 > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:102: GCC:*_CLANG35_*_CC_FLAGS = -std=c99 > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:103: GCC:*_CLANG38_*_CC_FLAGS = -std=c99 > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:104: GCC:*_CLANGPDB_*_CC_FLAGS = -std=c99 -Wno-error=incompatible-pointer-types > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:106: XCODE:*_*_*_CC_FLAGS = -std=c99 > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:55: MSFT:DEBUG_VS2003_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:56: MSFT:RELEASE_VS2003_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:57: MSFT:DEBUG_VS2003xASL_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:58: MSFT:RELEASE_VS2003xASL_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:59: MSFT:DEBUG_DDK3790_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:60: MSFT:RELEASE_DDK3790_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:61: MSFT:DEBUG_DDK3790xASL_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:62: MSFT:RELEASE_DDK3790xASL_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:63: MSFT:DEBUG_*_IA32_CC_FLAGS == /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gm > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:64: MSFT:RELEASE_*_IA32_CC_FLAGS == /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:65: MSFT:DEBUG_*_X64_CC_FLAGS == /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:66: MSFT:RELEASE_*_X64_CC_FLAGS == /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF > CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:67: INTEL:*_*_*_CC_FLAGS = /Oi- > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:569: MSFT:*_*_IA32_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd4267 /wd4310 /wd4389 /wd4700 /wd4702 /wd4706 /wd4819 > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:570: MSFT:*_*_X64_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd4267 /wd4306 /wd4310 /wd4700 /wd4389 /wd4702 /wd4706 /wd4819 > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:572: INTEL:*_*_IA32_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:573: INTEL:*_*_X64_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:582: GCC:*_*_IA32_CC_FLAGS = -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=unused-but-set-variable > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:583: GCC:*_*_X64_CC_FLAGS = -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=format -Wno-format -Wno-error=unused-but-set-variable -DNO_MSABI_VA_FUNCS > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:584: GCC:*_*_ARM_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=unused-but-set-variable > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:585: GCC:*_*_AARCH64_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-format -Wno-error=unused-but-set-variable > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:586: GCC:*_*_RISCV64_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-format -Wno-error=unused-but-set-variable > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:587: GCC:*_CLANG35_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:588: GCC:*_CLANG38_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:589: GCC:*_CLANGPDB_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized -Wno-error=incompatible-pointer-types -Wno-error=pointer-sign -Wno-error=implicit-function-declaration -Wno-error=ignored-pragma-optimize > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:610: RVCT:*_*_ARM_CC_FLAGS = $(OPENSSL_FLAGS) --library_interface=aeabi_clib99 --diag_suppress=1296,1295,550,1293,111,68,177,223,144,513,188,128,546,1,3017 -JCryptoPkg/Include > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:611: XCODE:*_*_IA32_CC_FLAGS = -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=c99 -Wno-error=uninitialized > CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:612: XCODE:*_*_X64_CC_FLAGS = -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=c99 -Wno-error=uninitialized > OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS = -DMDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS = -mno-mmx -mno-sse > OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES > OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES > OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES > $./ggrep.py OvmfX64 CC_FLAG -- *.dsc > OvmfX64 only: git grep CC_FLAG -- *.dsc > OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS = -DMDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS = -mno-mmx -mno-sse > OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES > OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES > OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES > $ > > Note if we cache the combined AutoGenTimeStamp and AutoGen files into a single file in the Build direction and use that file on the next run the time goes down to: > real 0m0.177s > user 0m0.185s > sys 0m0.522s > > If you don’t pass any `git grep` commands you get the file list back: > $./ggrep.py OvmfX64 > Conf/build_rule.txt > Conf/tools_def.txt > CryptoPkg/CryptoPkg.dec > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf > CryptoPkg/Library/BaseCryptLib/Cipher/CryptAes.c > … > > > My prototype is around 135 lines of Python. > > Thanks, > > Andrew Fish > > > > >