From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by mx.groups.io with SMTP id smtpd.web11.2226.1668468589459393126 for ; Mon, 14 Nov 2022 15:29:49 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=eDfAoKSz; spf=pass (domain: gmail.com, ip: 209.85.216.47, mailfrom: pedro.falcato@gmail.com) Received: by mail-pj1-f47.google.com with SMTP id r61-20020a17090a43c300b00212f4e9cccdso15273590pjg.5 for ; Mon, 14 Nov 2022 15:29:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pwwfXA+rEU6CNw5yarG1rr6JabYCIJTkmwnZuo96VHY=; b=eDfAoKSzdpdh551D38xyOHl3voFiDIEXrHpPj0eNVcjTnlu8tiKJZXYNbXhT75FQQT Txh/PLgi3PK2wFyFn0QqWKw+1XJ3XigxXM/XhXDcUJLJtVYOSebUlYgpAO4rQRCYVOp9 zEyNamM8fjp5u/jj3tFDTpAdmcEav5I7klvIP8MaX0eAsG2gOrzp36xFTTMJo6dhKfLW RpF+5fySKNGlftZ7OdQeHp7xE0195cNZnrUrOX8iQY8U/pcg34EpXTZRBBw/45kypR/D cfiB1EJWPU52DBEKpHoLu8tu2CatTXbTuM/IcydWMPb5VIAZ7HzpaVy46hhLBmv4gpRW Eadg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pwwfXA+rEU6CNw5yarG1rr6JabYCIJTkmwnZuo96VHY=; b=tj9wsQ2MjYJWK+jLJ3NOILcxO9dxDKkavEmVOuHXLuRzgHCHBiynpxW0//GNgZZLpr koKROhPtZWmtWiJDYpxHNJyFes7xsQxWHEhL4C0y+pwoFE9s5z9MNlAn0kHQvAO4IBBe 4svw0h/Du9J7Zar8n9mCbnr1P+Fp+xmGP2J//mBdoW0RB2wAPTz0/kIxg5DFvC5QQj59 SX7H3sXzMI/p+Xo4di+dMFHjHU/6nVNC/RNGYDX8WHDV8xXTOCLSi89YdZRA3N0U+y9P w/VTIKbuERXGIGTs/XmEi/XWX9YTj+UHxXVW53l/yrhcNZOyHjQG2BWp9SiLW6/9mz1c fDuw== X-Gm-Message-State: ANoB5pmhQ4OpXzqs+YJuCN38oD402AnbbbvvwAI2QcG6jEvyJ5783/n8 qT+BVNxbToQsU2xn5Dq9SE0gwdVO7JadVyaDdBjFaOYWd0Y= X-Google-Smtp-Source: AA0mqf4Z/Fhw5XcmtRUvr46X6nHG4D5NZwoy5MTkpzUkWuIi3Kfjg9PGeidS+QHqkIcMgqXJsWpfp6pZf0iFFboL1jQ= X-Received: by 2002:a17:90a:bc95:b0:216:cdf6:54c0 with SMTP id x21-20020a17090abc9500b00216cdf654c0mr15673357pjr.34.1668468588474; Mon, 14 Nov 2022 15:29:48 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: "Pedro Falcato" Date: Mon, 14 Nov 2022 23:29:36 +0000 Message-ID: Subject: Re: [edk2-devel] 1024 VCPU limitation To: devel@edk2.groups.io, michael.d.kinney@intel.com Cc: Pawel Polawski , "Dong, Eric" , Laszlo Ersek , "Ni, Ray" , "Kumar, Rahul R" Content-Type: multipart/alternative; boundary="00000000000088f6c505ed769d2a" --00000000000088f6c505ed769d2a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Nov 7, 2022 at 5:28 PM Michael D Kinney wrote: > Hi Pawel, > > > > I see the following union involved in the size of this structure. > > > > typedef union { > > IA32_HANDOFF_STATUS IA32HealthFlags; > > X64_HANDOFF_STATUS x64HealthFlags; > > ITANIUM_HANDOFF_STATUS ItaniumHealthFlags; > > } EFI_SEC_PLATFORM_INFORMATION_RECORD; > > > > IA32 is 4 bytes per CPU > > X64 is 4 bytes per CPU > > Itanium is 56 bytes per CPU > > > > We have removed the Itanium content from edk2 repo and it look like we > missed this > > union. > Hi Mike, I just want to note that I don't think you can remove ITANIUM_HANDOFF_STATUS (in an upstreamable way at least) since it's specified in the EFI PI spec, and it would also break any sort of ABI. Maybe once you update the spec? Or maybe we could find a way to pass these handoff statuses in multiple HOBs, or have v2 HOBs with UINT32 lengths (with an appropriate spec update). If you comment out the following line from the union does it resolve the > issue? > > > > > https://github.com/tianocore/edk2/blob/7c0ad2c33810ead45b7919f8f8d0e282da= e52e71/MdePkg/Include/Ppi/SecPlatformInformation.h#L137 > > > > I know this only increases the total number of CPUs that can be handled b= y > a single 64kb HOB, so we would run into > > it again at a higher number of CPUs. However, I think this gets the > overhead per CPU down to 8 bytes, which should > > scale to about 8091 CPUs. > > > > Thanks, > > > > Mike > > > > > > *From:* Pawel Polawski > *Sent:* Monday, November 7, 2022 3:52 AM > *To:* devel@edk2.groups.io > *Cc:* Dong, Eric ; Laszlo Ersek ; > Ni, Ray ; Kumar, Rahul R ; > Kinney, Michael D > *Subject:* [edk2-devel] 1024 VCPU limitation > > > > Hi All, > > > > I am trying to run edk2 with more than 1024 VCPU. It looks like it is not > possible > > at the moment and results in an ASSERT trigger. > > > > In the past the topic has been analyzed by Laszlo Ersek [1]. It turns out > that the limit > > is result of HOB default allocation being limited to ~64KB, quoting > original email thread: > > > > """ > > If "NumberOfProcessors" is large enough, such as ~1024, then > "BistInformationSize" will exceed ~64KB, and PeiServicesAllocatePool() > will fail with EFI_OUT_OF_RESOURCES. The reason is that pool allocations > in PEI are implemented with memory alloaction HOBs, and HOBs can't be > larger than ~64KB. (See PeiAllocatePool() in > "MdeModulePkg/Core/Pei/Memory/MemoryServices.c".) > > """ > > > > Even with HOB allocation being changed, I am afraid it may break some > > compatibility on the DXE level. This is the reason I am looking for a mor= e > universal solution. > > I believe the same limitation exists for the physical x86 platforms with > more than 1024 CPU. > > > > If someone has encountered the same issue or has knowledge that workaroun= d > / solution for > > this already exists or is being developed? > > > > [1] > https://listman.redhat.com/archives/edk2-devel-archive/2021-June/msg01493 > > > > Best regards, > > Pawel > > > -- > > *Pawe=C5=82 Po=C5=82awski* > > Red Hat Virtualization > > ppolawsk@redhat.com > > @RedHat Red Hat > Red Hat > > > >=20 > > --=20 Pedro Falcato --00000000000088f6c505ed769d2a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, Nov 7, 2022 at 5:28 PM Michael D = Kinney <michael.d.kinney@i= ntel.com> wrote:

Hi Pawel,

=C2=A0

I see the following union involved in the size= of this structure.

=C2=A0

typedef union {

=C2=A0 IA32_HANDOFF_STATUS =C2=A0 =C2=A0 =C2=A0 IA32HealthFlags;

=C2=A0 X64_HANDOFF_STATUS =C2=A0 =C2=A0 =C2=A0 =C2=A0x64HealthFlags;

=C2=A0 ITANIUM_HANDOFF_STATUS =C2=A0 =C2=A0ItaniumHealthFlags;

} EFI_SEC_PLATFORM_INFORMATION_RECORD;

=C2=A0

IA32 is 4 bytes per CPU

X64 is 4 bytes per CPU

Itanium is 56 bytes per CPU

=C2=A0

We have removed the Itanium content from edk2 = repo and it look like we missed this

union.

Hi Mike,

I just want to note that I don'= ;t think you can remove ITANIUM_HANDOFF_STATUS (in an upstreamable way at l= east) since it's specified in the EFI PI spec, and it would also break = any sort of ABI.
Maybe once you update the spec? Or maybe we = could find a way to pass these handoff statuses in multiple HOBs, or have v= 2 HOBs with UINT32 lengths (with an appropriate spec update).

If you comment out the following line from the union does it resolve= the issue?

=C2=A0

https://github.com/tianocore/edk2/blo= b/7c0ad2c33810ead45b7919f8f8d0e282dae52e71/MdePkg/Include/Ppi/SecPlatformIn= formation.h#L137

=C2=A0

I know this only increases the total number of= CPUs that can be handled by a single 64kb HOB, so we would run into=

it again at a higher number of CPUs.=C2= =A0 However, I think this gets the overhead per CPU down to 8 bytes, whi= ch should

scale to about 8091 CPUs.=

=C2=A0

Thanks,

=C2=A0

Mike

=C2=A0

=C2=A0

From: Pawel Polawski <<= a href=3D"mailto:ppolawsk@redhat.com" target=3D"_blank">ppolawsk@redhat.com= >
Sent: Monday, November 7, 2022 3:52 AM
To: devel@= edk2.groups.io
Cc: Dong, Eric <eric.dong@intel.com>; Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>= ;; Kumar, Rahul R <rahul.r.kumar@intel.com>; Kinney, Michael D <michael.d.kinney@intel.= com>
Subject: [edk2-devel] 1024 VCPU limitation

=C2=A0

Hi All,

=C2=A0

I am trying to run edk2 with more than 1024 VCPU. It= looks like it is not possible

at the moment and results in an ASSERT trigger.

=C2=A0

In the past the topic has been analyzed by Laszlo Er= sek [1]. It turns out that the limit

is result of HOB default allocation being limited to= ~64KB, quoting original email thread:

=C2=A0

"""

If "NumberOfProcessors" is large enough, s= uch as ~1024, then
"BistInformationSize" will exceed ~64KB, and PeiServicesAllocateP= ool()
will fail with EFI_OUT_OF_RESOURCES. The reason is that pool allocations in PEI are implemented with memory alloaction HOBs, and HOBs can't be larger than ~64KB. (See PeiAllocatePool() in
"MdeModulePkg/Core/Pei/Memory/MemoryServices.c".)

"""

=C2=A0

Even with HOB allocation being changed, I am afraid = it may break some

compatibility on the DXE level. This is the reason I= am looking for a more universal solution.

I believe the same limitation exists for the physica= l x86 platforms with more than 1024 CPU.

=C2=A0

If someone has encountered the same issue or has kno= wledge that workaround / solution for

this already exists or is being developed?=

=C2=A0

=C2=A0

Best regards,

Pawel
=20



--
Pedro Falcato
--00000000000088f6c505ed769d2a--