[PATCH v1 1/1] BaseTools: Update Edk2ToolsBuild.py to use multiple threads on Linux

[PATCH v1 1/1] BaseTools: Update Edk2ToolsBuild.py to use multiple threads on Linux

[Michael Kubacki]

From: Sean Brogan <sean.brogan@microsoft.com>

REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2640

Azure Pipelines agents have 2 threads. This commit has been shown to
reduce the build time in half on those agents.

Cc: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 BaseTools/Edk2ToolsBuild.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/BaseTools/Edk2ToolsBuild.py b/BaseTools/Edk2ToolsBuild.py
index 057d2e9e0633..1ea8187de693 100644
--- a/BaseTools/Edk2ToolsBuild.py
+++ b/BaseTools/Edk2ToolsBuild.py
@@ -11,6 +11,7 @@ import os
 import sys
 import logging
 import argparse
+import multiprocessing
 from edk2toolext import edk2_logging
 from edk2toolext.environment import self_describing_environment
 from edk2toolext.base_abstract_invocable import BaseAbstractInvocable
@@ -141,7 +142,8 @@ class Edk2ToolsBuild(BaseAbstractInvocable):
             return ret
 
         elif self.tool_chain_tag.lower().startswith("gcc"):
-            ret = RunCmd("make", "-C .", workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
+            cpu_count = self.GetCpuThreads()
+            ret = RunCmd("make", f"-C .  -j {cpu_count}", workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
             if ret != 0:
                 raise Exception("Failed to build.")
 
@@ -154,6 +156,18 @@ class Edk2ToolsBuild(BaseAbstractInvocable):
         logging.critical("Tool Chain not supported")
         return -1
 
+    def GetCpuThreads(self) -> int:
+        ''' Function to return number of cpus. If error return 1'''
+        cpus = 1
+        try:
+            cpus = multiprocessing.cpu_count()
+        except:
+            # from the internet there are cases where cpu_count is not implemented.
+            # will handle error by just doing single proc build
+            pass
+        return cpus
+
+
 
 def main():
     Edk2ToolsBuild().Invoke()
-- 
2.16.3.windows.1

Re: [edk2-devel] [PATCH v1 1/1] BaseTools: Update Edk2ToolsBuild.py to use multiple threads on Linux

[Bob Feng]

This change is good to me.

Reviewed-by: Bob C Feng <bob.c.feng@intel.com>

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Michael Kubacki
Sent: Friday, April 3, 2020 1:05 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>
Subject: [edk2-devel] [PATCH v1 1/1] BaseTools: Update Edk2ToolsBuild.py to use multiple threads on Linux

From: Sean Brogan <sean.brogan@microsoft.com>

REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2640

Azure Pipelines agents have 2 threads. This commit has been shown to reduce the build time in half on those agents.

Cc: Bob C Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 BaseTools/Edk2ToolsBuild.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/BaseTools/Edk2ToolsBuild.py b/BaseTools/Edk2ToolsBuild.py index 057d2e9e0633..1ea8187de693 100644
--- a/BaseTools/Edk2ToolsBuild.py
+++ b/BaseTools/Edk2ToolsBuild.py
@@ -11,6 +11,7 @@ import os
 import sys
 import logging
 import argparse
+import multiprocessing
 from edk2toolext import edk2_logging
 from edk2toolext.environment import self_describing_environment  from edk2toolext.base_abstract_invocable import BaseAbstractInvocable @@ -141,7 +142,8 @@ class Edk2ToolsBuild(BaseAbstractInvocable):
             return ret
 
         elif self.tool_chain_tag.lower().startswith("gcc"):
-            ret = RunCmd("make", "-C .", workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
+            cpu_count = self.GetCpuThreads()
+            ret = RunCmd("make", f"-C .  -j {cpu_count}", 
+ workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
             if ret != 0:
                 raise Exception("Failed to build.")
 
@@ -154,6 +156,18 @@ class Edk2ToolsBuild(BaseAbstractInvocable):
         logging.critical("Tool Chain not supported")
         return -1
 
+    def GetCpuThreads(self) -> int:
+        ''' Function to return number of cpus. If error return 1'''
+        cpus = 1
+        try:
+            cpus = multiprocessing.cpu_count()
+        except:
+            # from the internet there are cases where cpu_count is not implemented.
+            # will handle error by just doing single proc build
+            pass
+        return cpus
+
+
 
 def main():
     Edk2ToolsBuild().Invoke()
--
2.16.3.windows.1

Re: [edk2-devel] [PATCH v1 1/1] BaseTools: Update Edk2ToolsBuild.py to use multiple threads on Linux

[Liming Gao]

Reviewed-by: Liming Gao <liming.gao@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael Kubacki
> Sent: Friday, April 3, 2020 1:05 AM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>
> Subject: [edk2-devel] [PATCH v1 1/1] BaseTools: Update Edk2ToolsBuild.py to use multiple threads on Linux
> 
> From: Sean Brogan <sean.brogan@microsoft.com>
> 
> REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2640
> 
> Azure Pipelines agents have 2 threads. This commit has been shown to
> reduce the build time in half on those agents.
> 
> Cc: Bob C Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
> ---
>  BaseTools/Edk2ToolsBuild.py | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/BaseTools/Edk2ToolsBuild.py b/BaseTools/Edk2ToolsBuild.py
> index 057d2e9e0633..1ea8187de693 100644
> --- a/BaseTools/Edk2ToolsBuild.py
> +++ b/BaseTools/Edk2ToolsBuild.py
> @@ -11,6 +11,7 @@ import os
>  import sys
>  import logging
>  import argparse
> +import multiprocessing
>  from edk2toolext import edk2_logging
>  from edk2toolext.environment import self_describing_environment
>  from edk2toolext.base_abstract_invocable import BaseAbstractInvocable
> @@ -141,7 +142,8 @@ class Edk2ToolsBuild(BaseAbstractInvocable):
>              return ret
> 
>          elif self.tool_chain_tag.lower().startswith("gcc"):
> -            ret = RunCmd("make", "-C .", workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
> +            cpu_count = self.GetCpuThreads()
> +            ret = RunCmd("make", f"-C .  -j {cpu_count}", workingdir=shell_env.get_shell_var("EDK_TOOLS_PATH"))
>              if ret != 0:
>                  raise Exception("Failed to build.")
> 
> @@ -154,6 +156,18 @@ class Edk2ToolsBuild(BaseAbstractInvocable):
>          logging.critical("Tool Chain not supported")
>          return -1
> 
> +    def GetCpuThreads(self) -> int:
> +        ''' Function to return number of cpus. If error return 1'''
> +        cpus = 1
> +        try:
> +            cpus = multiprocessing.cpu_count()
> +        except:
> +            # from the internet there are cases where cpu_count is not implemented.
> +            # will handle error by just doing single proc build
> +            pass
> +        return cpus
> +
> +
> 
>  def main():
>      Edk2ToolsBuild().Invoke()
> --
> 2.16.3.windows.1
> 
> 
>