From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mx.groups.io with SMTP id smtpd.web11.1616.1596531615308429827 for ; Tue, 04 Aug 2020 02:00:15 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=ayZb/hwT; spf=pass (domain: intel.com, ip: 192.55.52.115, mailfrom: liming.gao@intel.com) IronPort-SDR: 9i7n9oUKpWYoz7u9MrMd5QxWuycAiF8jnhitHaWlI58tAx3QH2hO7nlUZQaVl9Vz45x1F8FNoA JfzbEU1Yg0QQ== X-IronPort-AV: E=McAfee;i="6000,8403,9702"; a="151501190" X-IronPort-AV: E=Sophos;i="5.75,433,1589266800"; d="scan'208";a="151501190" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Aug 2020 02:00:14 -0700 IronPort-SDR: l8FjyT3OAgyFuNQa7IG2n526Sz3ky8gFC32FPBganqPLJc0sGlfB4Tn43PiQ2o2lWP8SGor+CK tPQPVOIrKcfA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,433,1589266800"; d="scan'208";a="436714615" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga004.jf.intel.com with ESMTP; 04 Aug 2020 02:00:13 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 4 Aug 2020 02:00:05 -0700 Received: from FMSEDG001.ED.cps.intel.com (10.1.192.133) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Tue, 4 Aug 2020 02:00:05 -0700 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.36.53) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 4 Aug 2020 02:00:02 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gbL5DR5o+sIJbMbDsYn9QJWrJ7dUgNTgtXEFfGiHlKUZ7vSX5NEAVdXIzVkQGogNlFosInq9YMRRDoJobnFWod4FhCiCBEtB2XTfqK4SRNSURcVvcZngQNnBmXXRLXdaLRbLCHwbCNbSNd0q1gYpAhutfPV7ViiA10mQj+sNUZBM2u1k6ojIk2mTYJUmRBD/D38jSYVyI4g8NVwXxGkQyDcbC9mBNWjWn9vjQfRZcGRl2pzICDdm9vu32GA2Q+UlG+onMmyWAFCD6bt9WAuxgTbVCnm/8NA5u7GXlLc7x7IMmby9lvkYmzl64N4tVR0bWjZgBXkTCSAqGgOqGN2JaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=acbNe/fecSZv0dsxXstVeeyS4ld6GDpqN2oMX0ovbuU=; b=fId19WI1MJ+HUA0mk0Unqy0OeVBfe+hHCqmHKf618r59PODtgVEFnJ4AaqXvA5WNppworgJE7ePQ9lWjUHHCmFE1DH5tDXSxHCca0GzBfO92D2Q7gL3dBkuQILDCa7kCa8c86Awhcj30EyLcLuO8C8tQmZ8J8ITFdM76NdABnndoHpfcN2nT26bUSP2oMpVpdoyDVrdXM0cDpjOAu1hTHp4CcqodEXW18xdUSdydsYjiVTK3mOw+sjRxaQ217lwJrnxORQM3/Lc++n/GpjlmEG7T19kW4KrZHrQfCyDQJ95cruDKy3Ev5E9naVPTcZod68+WmqdYyzpjrY0mwDsEmQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=acbNe/fecSZv0dsxXstVeeyS4ld6GDpqN2oMX0ovbuU=; b=ayZb/hwTYOP1SBRXPMMlH7SxaQdk2W7V9GRbv6RSKGLHG7d+5N+4UY4r+/VSktYTc2NpESHQ444J3wnyGuAgLbkukFMf0CUGI+NNH7jJT9niVw+hjKoXKhDWy9+hfuGdUha8uLSlXaZjl0M9jgNalaM/3LA1Wu1RWgY+Iywi53s= Received: from CY4PR11MB1526.namprd11.prod.outlook.com (2603:10b6:910:7::10) by CY4PR1101MB2149.namprd11.prod.outlook.com (2603:10b6:910:1a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3239.16; Tue, 4 Aug 2020 09:00:01 +0000 Received: from CY4PR11MB1526.namprd11.prod.outlook.com ([fe80::f8c4:711:84e3:4c97]) by CY4PR11MB1526.namprd11.prod.outlook.com ([fe80::f8c4:711:84e3:4c97%4]) with mapi id 15.20.3239.021; Tue, 4 Aug 2020 09:00:01 +0000 From: "Liming Gao" To: "devel@edk2.groups.io" , "Feng, Bob C" CC: "Chen, Christine" , Lucy Yan Subject: Re: [edk2-devel] [Patch] BaseTools: Improve the method of checking queue empty Thread-Topic: [edk2-devel] [Patch] BaseTools: Improve the method of checking queue empty Thread-Index: AQHWaTpYZwbLqck9ckS9kysqkqBi76knqM1g Date: Tue, 4 Aug 2020 09:00:01 +0000 Message-ID: References: <20200803020338.24712-1-bob.c.feng@intel.com> In-Reply-To: <20200803020338.24712-1-bob.c.feng@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-product: dlpe-windows dlp-reaction: no-action authentication-results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.198.147.194] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fea2a5ae-5325-4dd9-a151-08d83854c597 x-ms-traffictypediagnostic: CY4PR1101MB2149: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:510; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Vg033lwDeMFNrJ9uxyW0KAuM4oeCfLzrbo5dCldzecPcijQgyEwp5+QwEdq/gqJC/Sn9liBodK5YxQpOYeryLpEuid5FiZFmJY6G0O20i3KE5ET7Z2xcsrn0tl9CoU7wcfV6JT7Xfx2iMwKGZFXJFSkSNtKS26cknuWM9PVX9X/fI7KFAhEzkNLjkI7FEHvcZ4X0++cERwz6WmV+3Sg7MTfF+Sd+2SjUDSSrN7QCgp8d17P/GphEVOpaCkPUv37r7Qc5wP/0UIz3VCsJybmHySqu2jpZjmsqYE8Az8dqY4dhmRcHW+ej4lNUAUtvK6gP/sl7tDH7g9GYRX7UcjJ9nyUSVEZbVmoOxCc5tgz65ieKTpDn8pxLUA8IyBiHOi+Xp8hbInh+7+Bc+q3TywfNbQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY4PR11MB1526.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(396003)(346002)(366004)(376002)(39860400002)(136003)(110136005)(6506007)(53546011)(71200400001)(186003)(316002)(55016002)(66446008)(66946007)(66476007)(64756008)(6636002)(54906003)(5660300002)(66556008)(86362001)(76116006)(26005)(9686003)(7696005)(4326008)(478600001)(52536014)(8936002)(8676002)(966005)(33656002)(83380400001)(2906002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: eTn1ZLAX2af/w86iUZbOgLXjIHqCMNQicbZiRa4q6VyZbwyGMjeA3TWTrwmNlE38BXn0qEPIronZ06BIYaOaLP5LgMu5wskazB+l1LIFEl2xCK4Z5Ssy4JZFDsfRN1KoOwVQkrYX6V2AwG0+3ooyZOyxl1WqZQpI6UNGLSGjHvFcc6i1EmKtxDinTIaB3PKTBzNmJfd4fzHrWUEnPXOUf0SryCVy2yQhZW6Pq6zDuICnILwRT/PITImS/t/T7IAWZYjEEYNz2bhhOCc40dIA9Yqvou0Ejki7zIEw/oUnpXapaDBpZm8tu7Zq65+Tx6mPeQiAaBbSlG28MFRtIMk5SCqHU9G0Ncl0q9GplwrbHdeLXxJWtrEJxjMa5Jow7UlHYIOdACm8G0HNW8e8Q5D+wIoRGvB9ejM3xNh37mctxjfCEx7nFcA83pLlMwcGvxH59yEKYuS//3bnw/njwLAeZgkZhZU67zvzIZoktDa0dTfIlhnRVUQn744vgkJeCL13KagPc/gICLkNRMOPnktPD9FKIXjAwvOLQ2i00hmJUasKNH8UgUzqB0XCxNNVwPF4qXKcok63XQCGINwD0VJAl/pDHk89LgYfEGLN1X4jnrEC+Ap/K/8Fn0yvUgb5z9z2Ck+ef5FJYbDyLLzN3gDITg== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1526.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: fea2a5ae-5325-4dd9-a151-08d83854c597 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Aug 2020 09:00:01.5372 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: YvF8DfRPUqBEqsYXhBrtC8F/7nh2kQn0IynrRiGdepz6CBq58ZHW3ETfhD3/0+K+/m/iihSmw+Sz+MUu331n8w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1101MB2149 Return-Path: liming.gao@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable Reviewed-by: Liming Gao -----Original Message----- From: devel@edk2.groups.io On Behalf Of Bob Feng Sent: 2020=1B$BG/=1B(B8=1B$B7n=1B(B3=1B$BF|=1B(B 10:04 To: devel@edk2.groups.io Cc: Feng, Bob C ; Gao, Liming = ; Chen, Christine ; Lucy Yan Subject: [edk2-devel] [Patch] BaseTools: Improve the method of checking qu= eue empty From: "Feng, Bob C" Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2807 The Queue.empty() method is not reliable in the multiple process runtime e= nvironment. This patch uses a new method to check if all modules are proces= sed and workers need to be stopped. That is to add a None item at the botto= m of the queue. Worker check if it gets that None item to know if all the m= odule is processed. Signed-off-by: Bob Feng Cc: Liming Gao Cc: Yuwei Chen Cc: Lucy Yan --- .../Source/Python/AutoGen/AutoGenWorker.py | 26 ++++++++++++++----- BaseTools/Source/Python/build/build.py | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/= Source/Python/AutoGen/AutoGenWorker.py index 563d91b421..017f676399 100755 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -22,10 +22,11 @@ except: from Queue import Empty import traceback import sys from AutoGen.DataPipe import MemoryDataPipe import logging +import time =20 def clearQ(q): try: while True: q.get_nowait() @@ -109,11 +110,15 @@ class AutoGenManager(threading.Thread): badnews =3D self.feedback_q.get() if badnews is None: break if badnews =3D=3D "Done": fin_num +=3D 1 + elif badnews =3D=3D "QueueEmpty": + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % = (os.getpid(), badnews)) + self.TerminateWorkers() else: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s"=20 + % (os.getpid(), badnews)) self.Status =3D False self.TerminateWorkers() if fin_num =3D=3D len(self.autogen_workers): self.clearQueue() for w in self.autogen_workers: @@ -225,16 +230,25 @@ class AutoGenWorkerInProcess(mp.Process): FfsCmd =3D {} GlobalData.FfsCmd =3D FfsCmd PlatformMetaFile =3D self.GetPlatformMetaFile(self.data_pipe.= Get("P_Info").get("ActivePlatform"), self.data_pipe.Get("P_Info")= .get("WorkspaceDir")) while True: - if self.module_queue.empty(): - break if self.error_event.is_set(): break module_count +=3D 1 - module_file,module_root,module_path,module_basename,modul= e_originalpath,module_arch,IsLib =3D self.module_queue.get_nowait() + try: + module_file,module_root,module_path,module_basename,m= odule_originalpath,module_arch,IsLib =3D self.module_queue.get_nowait() + except Empty: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % = (os.getpid(), "Fake Empty.")) + time.sleep(0.01) + continue + if module_file is None: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % = (os.getpid(), "Worker get the last item in the queue.")) + self.feedback_q.put("QueueEmpty") + time.sleep(0.01) + continue + modulefullpath =3D os.path.join(module_root,module_file) taskname =3D " : ".join((modulefullpath,module_arch)) module_metafile =3D PathClass(module_file,module_root) if module_path: module_metafile.Path =3D module_path @@ -278,15 +292,= 15 @@ class AutoGenWorkerInProcess(mp.Process): self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, "Mak= eCache", True)) continue else: self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, "Mak= eCache", False)) =20 - except Empty: - pass - except: + except Exception as e: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" %=20 + (os.getpid(), str(e))) self.feedback_q.put(taskname) finally: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" %=20 + (os.getpid(), "Done")) self.feedback_q.put("Done") self.cache_q.put("CacheDone") =20 def printStatus(self): print("Processs ID: %d Run %d modules in AutoGen " % (os.getpid()= ,len(AutoGen.Cache()))) diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyt= hon/build/build.py index 1ab1e60a64..59ceacfed0 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -1215,11 +1215,11 @@ class Build(): if Target not in ['clean', 'cleanlib', 'cleanall', 'run', 'fds']: # for target which must generate AutoGen code and makefile mqueue =3D mp.Queue() for m in AutoGenObject.GetAllModuleInfo: mqueue.put(m) - + mqueue.put((None,None,None,None,None,None,None)) AutoGenObject.DataPipe.DataContainer =3D {"CommandTarget": se= lf.Target} AutoGenObject.DataPipe.DataContainer =3D {"Workspace_timestam= p": AutoGenObject.Workspace._SrcTimeStamp} AutoGenObject.CreateLibModuelDirs() AutoGenObject.DataPipe.DataContainer =3D {"LibraryBuildDirect= oryList":AutoGenObject.LibraryBuildDirectoryList} AutoGenObject.DataPipe.DataContainer =3D {"ModuleBuildDirecto= ryList":AutoGenObject.ModuleBuildDirectoryList} @@ -2172,10 +2172,11 @@ class Build(): ToolChain, Arch, self.PlatformFile,Pa.D= ataPipe) self.AllModules.add(Ma) data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVar_%s_%s= .bin" % (str(Pa.Guid),Pa.Arch)) Pa.DataPipe.dump(data_pipe_file) =20 + mqueue.put((None,None,None,None,None,None,None)) autogen_rt, errorcode =3D self.StartAutoGen(mqueue, Pa.DataPi= pe, self.SkipAutoGen, PcdMaList, cqueue) =20 if not autogen_rt: self.AutoGenMgr.TerminateWorkers() self.AutoGenMgr.join(1) -- 2.20.1.windows.1