[PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

[PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

[Chiu, Chasel]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1913

To improve build performance by default the maximum
processors should be used for building.
To support this, NUMBER_OF_PROCESSORS in build.cfg has
new definition:
  0 = AUTO. Maximum of processors will be used.
  non-zero = limit or force the number of processors.

Test: tried several NUMBER_OF_PROCESSORS settings and
      the build parameter can be updated accordingly
      also builds successfully.

Cc: Michael Kubacki <michael.a.kubacki@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
---
 Platform/Intel/build.cfg     | 2 +-
 Platform/Intel/build_bios.py | 8 +++++++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Platform/Intel/build.cfg b/Platform/Intel/build.cfg
index 6c23e5eabc..fc6e4fe824 100644
--- a/Platform/Intel/build.cfg
+++ b/Platform/Intel/build.cfg
@@ -46,7 +46,7 @@ FSP_TEST_RELEASE = FALSE
 SECURE_BOOT_ENABLE = FALSE
 REBUILD_MODE =
 BUILD_ROM_ONLY =
-NUMBER_OF_PROCESSORS = 1
+NUMBER_OF_PROCESSORS = 0
 
 
 [PLATFORMS]
diff --git a/Platform/Intel/build_bios.py b/Platform/Intel/build_bios.py
index 09eceddeff..864511ac4d 100644
--- a/Platform/Intel/build_bios.py
+++ b/Platform/Intel/build_bios.py
@@ -21,6 +21,7 @@ import shutil
 import argparse
 import traceback
 import subprocess
+import multiprocessing
 from importlib import import_module
 
 try:
@@ -345,7 +346,12 @@ def build(config):
     print(" BUILD_ROM_ONLY = ", config.get("BUILD_ROM_ONLY"))
     print("==========================================")
 
-    command = ["build", "-n", config["NUMBER_OF_PROCESSORS"]]
+    NumberOfProcessors = config["NUMBER_OF_PROCESSORS"]
+    if config["NUMBER_OF_PROCESSORS"] == "0":
+        # 0 means AUTO, maximum number of processors will be set
+        NumberOfProcessors = str(multiprocessing.cpu_count())
+    print ("Number of processors set to " + NumberOfProcessors)
+    command = ["build", "-n", NumberOfProcessors]
 
     if config["REBUILD_MODE"] and config["REBUILD_MODE"] != "":
         command.append(config["REBUILD_MODE"])
-- 
2.13.3.windows.1

Re: [edk2-devel] [PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

[Bob Feng]

Hi Chasel,

Patch looks good.

For the process number, build.py does the same thing as this patch. If pass -n 0 to build.py, build.py will set the processor number to multiprocessing.cpu_count()

Thanks,
Bob


-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Chiu, Chasel
Sent: Thursday, June 13, 2019 2:15 PM
To: devel@edk2.groups.io
Cc: Kubacki, Michael A <michael.a.kubacki@intel.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Gao, Liming <liming.gao@intel.com>
Subject: [edk2-devel] [PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1913

To improve build performance by default the maximum processors should be used for building.
To support this, NUMBER_OF_PROCESSORS in build.cfg has new definition:
  0 = AUTO. Maximum of processors will be used.
  non-zero = limit or force the number of processors.

Test: tried several NUMBER_OF_PROCESSORS settings and
      the build parameter can be updated accordingly
      also builds successfully.

Cc: Michael Kubacki <michael.a.kubacki@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
---
 Platform/Intel/build.cfg     | 2 +-
 Platform/Intel/build_bios.py | 8 +++++++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Platform/Intel/build.cfg b/Platform/Intel/build.cfg index 6c23e5eabc..fc6e4fe824 100644
--- a/Platform/Intel/build.cfg
+++ b/Platform/Intel/build.cfg
@@ -46,7 +46,7 @@ FSP_TEST_RELEASE = FALSE  SECURE_BOOT_ENABLE = FALSE  REBUILD_MODE =  BUILD_ROM_ONLY = -NUMBER_OF_PROCESSORS = 1
+NUMBER_OF_PROCESSORS = 0
 
 
 [PLATFORMS]
diff --git a/Platform/Intel/build_bios.py b/Platform/Intel/build_bios.py index 09eceddeff..864511ac4d 100644
--- a/Platform/Intel/build_bios.py
+++ b/Platform/Intel/build_bios.py
@@ -21,6 +21,7 @@ import shutil
 import argparse
 import traceback
 import subprocess
+import multiprocessing
 from importlib import import_module
 
 try:
@@ -345,7 +346,12 @@ def build(config):
     print(" BUILD_ROM_ONLY = ", config.get("BUILD_ROM_ONLY"))
     print("==========================================")
 
-    command = ["build", "-n", config["NUMBER_OF_PROCESSORS"]]
+    NumberOfProcessors = config["NUMBER_OF_PROCESSORS"]
+    if config["NUMBER_OF_PROCESSORS"] == "0":
+        # 0 means AUTO, maximum number of processors will be set
+        NumberOfProcessors = str(multiprocessing.cpu_count())
+    print ("Number of processors set to " + NumberOfProcessors)
+    command = ["build", "-n", NumberOfProcessors]
 
     if config["REBUILD_MODE"] and config["REBUILD_MODE"] != "":
         command.append(config["REBUILD_MODE"])
--
2.13.3.windows.1

Re: [PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

[Nate DeSimone]

Reviewed-by: Nate DeSimone <nathaniel.l.desimone@intel.com>

On 6/12/19, 11:17 PM, "Chiu, Chasel" <chasel.chiu@intel.com> wrote:

    REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1913
    
    To improve build performance by default the maximum
    processors should be used for building.
    To support this, NUMBER_OF_PROCESSORS in build.cfg has
    new definition:
      0 = AUTO. Maximum of processors will be used.
      non-zero = limit or force the number of processors.
    
    Test: tried several NUMBER_OF_PROCESSORS settings and
          the build parameter can be updated accordingly
          also builds successfully.
    
    Cc: Michael Kubacki <michael.a.kubacki@intel.com>
    Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
    Cc: Liming Gao <liming.gao@intel.com>
    Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
    ---
     Platform/Intel/build.cfg     | 2 +-
     Platform/Intel/build_bios.py | 8 +++++++-
     2 files changed, 8 insertions(+), 2 deletions(-)
    
    diff --git a/Platform/Intel/build.cfg b/Platform/Intel/build.cfg
    index 6c23e5eabc..fc6e4fe824 100644
    --- a/Platform/Intel/build.cfg
    +++ b/Platform/Intel/build.cfg
    @@ -46,7 +46,7 @@ FSP_TEST_RELEASE = FALSE
     SECURE_BOOT_ENABLE = FALSE
     REBUILD_MODE =
     BUILD_ROM_ONLY =
    -NUMBER_OF_PROCESSORS = 1
    +NUMBER_OF_PROCESSORS = 0
     
     
     [PLATFORMS]
    diff --git a/Platform/Intel/build_bios.py b/Platform/Intel/build_bios.py
    index 09eceddeff..864511ac4d 100644
    --- a/Platform/Intel/build_bios.py
    +++ b/Platform/Intel/build_bios.py
    @@ -21,6 +21,7 @@ import shutil
     import argparse
     import traceback
     import subprocess
    +import multiprocessing
     from importlib import import_module
     
     try:
    @@ -345,7 +346,12 @@ def build(config):
         print(" BUILD_ROM_ONLY = ", config.get("BUILD_ROM_ONLY"))
         print("==========================================")
     
    -    command = ["build", "-n", config["NUMBER_OF_PROCESSORS"]]
    +    NumberOfProcessors = config["NUMBER_OF_PROCESSORS"]
    +    if config["NUMBER_OF_PROCESSORS"] == "0":
    +        # 0 means AUTO, maximum number of processors will be set
    +        NumberOfProcessors = str(multiprocessing.cpu_count())
    +    print ("Number of processors set to " + NumberOfProcessors)
    +    command = ["build", "-n", NumberOfProcessors]
     
         if config["REBUILD_MODE"] and config["REBUILD_MODE"] != "":
             command.append(config["REBUILD_MODE"])
    -- 
    2.13.3.windows.1
    
    

Re: [edk2-devel] [PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

[Chiu, Chasel]

Thanks Bob! I was not aware that BaseTools already handled this.
I just verified and it works so I will update code review which will only touch build.cfg.

Thanks!
Chasel


> -----Original Message-----
> From: Feng, Bob C
> Sent: Thursday, June 13, 2019 3:39 PM
> To: devel@edk2.groups.io; Chiu, Chasel <chasel.chiu@intel.com>
> Cc: Kubacki, Michael A <michael.a.kubacki@intel.com>; Desimone, Nathaniel L
> <nathaniel.l.desimone@intel.com>; Gao, Liming <liming.gao@intel.com>
> Subject: RE: [edk2-devel] [PATCH] Platform/Intel: Dynamic
> NUMBER_OF_PROCESSORS in build.
> 
> Hi Chasel,
> 
> Patch looks good.
> 
> For the process number, build.py does the same thing as this patch. If pass -n 0 to
> build.py, build.py will set the processor number to multiprocessing.cpu_count()
> 
> Thanks,
> Bob
> 
> 
> -----Original Message-----
> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Chiu,
> Chasel
> Sent: Thursday, June 13, 2019 2:15 PM
> To: devel@edk2.groups.io
> Cc: Kubacki, Michael A <michael.a.kubacki@intel.com>; Desimone, Nathaniel L
> <nathaniel.l.desimone@intel.com>; Gao, Liming <liming.gao@intel.com>
> Subject: [edk2-devel] [PATCH] Platform/Intel: Dynamic
> NUMBER_OF_PROCESSORS in build.
> 
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1913
> 
> To improve build performance by default the maximum processors should be
> used for building.
> To support this, NUMBER_OF_PROCESSORS in build.cfg has new definition:
>   0 = AUTO. Maximum of processors will be used.
>   non-zero = limit or force the number of processors.
> 
> Test: tried several NUMBER_OF_PROCESSORS settings and
>       the build parameter can be updated accordingly
>       also builds successfully.
> 
> Cc: Michael Kubacki <michael.a.kubacki@intel.com>
> Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
> ---
>  Platform/Intel/build.cfg     | 2 +-
>  Platform/Intel/build_bios.py | 8 +++++++-
>  2 files changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/Platform/Intel/build.cfg b/Platform/Intel/build.cfg index
> 6c23e5eabc..fc6e4fe824 100644
> --- a/Platform/Intel/build.cfg
> +++ b/Platform/Intel/build.cfg
> @@ -46,7 +46,7 @@ FSP_TEST_RELEASE = FALSE  SECURE_BOOT_ENABLE =
> FALSE  REBUILD_MODE =  BUILD_ROM_ONLY = -NUMBER_OF_PROCESSORS =
> 1
> +NUMBER_OF_PROCESSORS = 0
> 
> 
>  [PLATFORMS]
> diff --git a/Platform/Intel/build_bios.py b/Platform/Intel/build_bios.py index
> 09eceddeff..864511ac4d 100644
> --- a/Platform/Intel/build_bios.py
> +++ b/Platform/Intel/build_bios.py
> @@ -21,6 +21,7 @@ import shutil
>  import argparse
>  import traceback
>  import subprocess
> +import multiprocessing
>  from importlib import import_module
> 
>  try:
> @@ -345,7 +346,12 @@ def build(config):
>      print(" BUILD_ROM_ONLY = ", config.get("BUILD_ROM_ONLY"))
>      print("==========================================")
> 
> -    command = ["build", "-n", config["NUMBER_OF_PROCESSORS"]]
> +    NumberOfProcessors = config["NUMBER_OF_PROCESSORS"]
> +    if config["NUMBER_OF_PROCESSORS"] == "0":
> +        # 0 means AUTO, maximum number of processors will be set
> +        NumberOfProcessors = str(multiprocessing.cpu_count())
> +    print ("Number of processors set to " + NumberOfProcessors)
> +    command = ["build", "-n", NumberOfProcessors]
> 
>      if config["REBUILD_MODE"] and config["REBUILD_MODE"] != "":
>          command.append(config["REBUILD_MODE"])
> --
> 2.13.3.windows.1
> 
> 
> 

[PATCH] Platform/Intel: Dynamic NUMBER_OF_PROCESSORS in build.

[Chiu, Chasel]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1913

To improve build performance by default the maximum
processors should be used for building.
NUMBER_OF_PROCESSORS in build.cfg is set to 0 so
BaseTools can utilize maximum processors for build.

Test: tried NUMBER_OF_PROCESSORS 0 and 1 cases and
      confirmed the build time is half with 0 case.

Cc: Michael Kubacki <michael.a.kubacki@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Chasel Chiu <chasel.chiu@intel.com>
---
 Platform/Intel/build.cfg | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Platform/Intel/build.cfg b/Platform/Intel/build.cfg
index 6c23e5eabc..fc6e4fe824 100644
--- a/Platform/Intel/build.cfg
+++ b/Platform/Intel/build.cfg
@@ -46,7 +46,7 @@ FSP_TEST_RELEASE = FALSE
 SECURE_BOOT_ENABLE = FALSE
 REBUILD_MODE =
 BUILD_ROM_ONLY =
-NUMBER_OF_PROCESSORS = 1
+NUMBER_OF_PROCESSORS = 0
 
 
 [PLATFORMS]
-- 
2.13.3.windows.1