From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web09.1980.1668466865301115335 for ; Mon, 14 Nov 2022 15:01:05 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Rl0nx0fr; spf=pass (domain: redhat.com, ip: 170.10.133.124, mailfrom: ppolawsk@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668466864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hb7ZPY+f0/i6Xsj4BmiQ5RvuoJM+pWuCbc0qUAMI570=; b=Rl0nx0frKR/3+JSIzenAmnILATAX2+3bgGs/Xzlr0VFmnBmlYsdJuWzvuejI0a5KNagqjT 3wV1rdbW4/WJDg7rLyg3v2XtLMM/XcrAGrXW8sSTpHjlSNqz/LDdGJqUrJiJUH4ZM8hqsd 2oUr7IfCUoPGAeoOFBTjewOlumWnvvs= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-68-h8xh9zOoOIe3yyJ4IFY2Bg-1; Mon, 14 Nov 2022 18:01:03 -0500 X-MC-Unique: h8xh9zOoOIe3yyJ4IFY2Bg-1 Received: by mail-wr1-f69.google.com with SMTP id d23-20020adfa417000000b002364a31b7c9so2302886wra.15 for ; Mon, 14 Nov 2022 15:01:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hb7ZPY+f0/i6Xsj4BmiQ5RvuoJM+pWuCbc0qUAMI570=; b=Bckg/QRUs63eMImC1HYqwXtiN+fiIDfxP8a8jDMrfV0eAYmpKLy6Qvp7EFhdx0u2Ot YrShJ8XzPqwhdbP4dC7dFoK0Tn7oJGDmOYkj9zpBCLz4amSTs1neEJAdy+apyT1N3g0N wLU3U+2BINRlEyVk8o7RAUIRNNchjNV0dGb6GKPrzfZR6ScvSrDYI1waCLANMiS/aHu8 3tnTzXmkuDUZcuJVboKLxUwraEwXKuErZiHFyuvNt6b5lDUcawZiASmzoRX1MRprAtsE 1PtPf0bGNXKsYyHXVwtAAZOHrVRDGE7DEh0YzgsdEcxD2BTVtoluec29ePxVx3sJGdJb vblA== X-Gm-Message-State: ANoB5pnrAY+5DreQQvr6VyF3gKXqU7z1GeD/b0ePFw1qAxsSXOo/ouha 5x98+egxygxi+X8TgkWh+Ex+71EzqYV5muVRw5xY74HorJMe+zR2pKgwrc0V8QpiCG/Ear275nV 4MAvTJWFT9UDGxLmI4qjT6TpTycAHpQ== X-Received: by 2002:a5d:6306:0:b0:22e:224c:3443 with SMTP id i6-20020a5d6306000000b0022e224c3443mr8985737wru.361.1668466861035; Mon, 14 Nov 2022 15:01:01 -0800 (PST) X-Google-Smtp-Source: AA0mqf7ms5IkqAoQPK8kADmY1q381XDRui5tCr8uBW2Hp8atbTAwjFANHRfMaKG4r34aX2q5cX52PxZGYC/25uFttMs= X-Received: by 2002:a5d:6306:0:b0:22e:224c:3443 with SMTP id i6-20020a5d6306000000b0022e224c3443mr8985723wru.361.1668466860733; Mon, 14 Nov 2022 15:01:00 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?UGF3ZcWCIFBvxYJhd3NraQ==?= Date: Tue, 15 Nov 2022 00:00:49 +0100 Message-ID: Subject: Re: [edk2-devel] 1024 VCPU limitation To: "Kinney, Michael D" Cc: "devel@edk2.groups.io" , "Dong, Eric" , Laszlo Ersek , "Ni, Ray" , "Kumar, Rahul R" X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/alternative; boundary="0000000000008dd49505ed763623" --0000000000008dd49505ed763623 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Michael, Thank you for the fast response and suggestion on how to resolve the issue. I am trying to build a custom qemu (as qemu has its own limitations) to verify if a fix will work. Best regards, Pawel On Mon, Nov 7, 2022 at 6:28 PM Kinney, Michael D wrote: > Hi Pawel, > > > > I see the following union involved in the size of this structure. > > > > typedef union { > > IA32_HANDOFF_STATUS IA32HealthFlags; > > X64_HANDOFF_STATUS x64HealthFlags; > > ITANIUM_HANDOFF_STATUS ItaniumHealthFlags; > > } EFI_SEC_PLATFORM_INFORMATION_RECORD; > > > > IA32 is 4 bytes per CPU > > X64 is 4 bytes per CPU > > Itanium is 56 bytes per CPU > > > > We have removed the Itanium content from edk2 repo and it look like we > missed this > > union. > > > > If you comment out the following line from the union does it resolve the > issue? > > > > > https://github.com/tianocore/edk2/blob/7c0ad2c33810ead45b7919f8f8d0e282da= e52e71/MdePkg/Include/Ppi/SecPlatformInformation.h#L137 > > > > I know this only increases the total number of CPUs that can be handled b= y > a single 64kb HOB, so we would run into > > it again at a higher number of CPUs. However, I think this gets the > overhead per CPU down to 8 bytes, which should > > scale to about 8091 CPUs. > > > > Thanks, > > > > Mike > > > > > > *From:* Pawel Polawski > *Sent:* Monday, November 7, 2022 3:52 AM > *To:* devel@edk2.groups.io > *Cc:* Dong, Eric ; Laszlo Ersek ; > Ni, Ray ; Kumar, Rahul R ; > Kinney, Michael D > *Subject:* [edk2-devel] 1024 VCPU limitation > > > > Hi All, > > > > I am trying to run edk2 with more than 1024 VCPU. It looks like it is not > possible > > at the moment and results in an ASSERT trigger. > > > > In the past the topic has been analyzed by Laszlo Ersek [1]. It turns out > that the limit > > is result of HOB default allocation being limited to ~64KB, quoting > original email thread: > > > > """ > > If "NumberOfProcessors" is large enough, such as ~1024, then > "BistInformationSize" will exceed ~64KB, and PeiServicesAllocatePool() > will fail with EFI_OUT_OF_RESOURCES. The reason is that pool allocations > in PEI are implemented with memory alloaction HOBs, and HOBs can't be > larger than ~64KB. (See PeiAllocatePool() in > "MdeModulePkg/Core/Pei/Memory/MemoryServices.c".) > > """ > > > > Even with HOB allocation being changed, I am afraid it may break some > > compatibility on the DXE level. This is the reason I am looking for a mor= e > universal solution. > > I believe the same limitation exists for the physical x86 platforms with > more than 1024 CPU. > > > > If someone has encountered the same issue or has knowledge that workaroun= d > / solution for > > this already exists or is being developed? > > > > [1] > https://listman.redhat.com/archives/edk2-devel-archive/2021-June/msg01493 > > > > Best regards, > > Pawel > > > -- > > *Pawe=C5=82 Po=C5=82awski* > > Red Hat Virtualization > > ppolawsk@redhat.com > > @RedHat Red Hat > Red Hat > > > > --=20 Pawe=C5=82 Po=C5=82awski Red Hat Virtualization ppolawsk@redhat.com @RedHat Red Hat Red Hat --0000000000008dd49505ed763623 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Michael,

Thank you for th= e fast response and suggestion on how to resolve the issue.
I am = trying to build a custom qemu (as qemu has its own limitations) to verify i= f a fix will work.

Best regards,
Pawel

On Mon, Nov 7, 2022 at 6:28 PM Kinney, Michael D <michael.d.kinney@intel.com> wrote:

Hi Pawel,

=C2=A0

I see the following union involved in the size= of this structure.

=C2=A0

typedef union {

=C2=A0 IA32_HANDOFF_STATUS =C2=A0 =C2=A0 =C2=A0 IA32HealthFlags;

=C2=A0 X64_HANDOFF_STATUS =C2=A0 =C2=A0 =C2=A0 =C2=A0x64HealthFlags;

=C2=A0 ITANIUM_HANDOFF_STATUS =C2=A0 =C2=A0ItaniumHealthFlags;

} EFI_SEC_PLATFORM_INFORMATION_RECORD;

=C2=A0

IA32 is 4 bytes per CPU

X64 is 4 bytes per CPU

Itanium is 56 bytes per CPU

=C2=A0

We have removed the Itanium content from edk2 = repo and it look like we missed this

union.

=C2=A0

If you comment out the following line from the= union does it resolve the issue?

=C2=A0

https://github.com/tianocore/edk2/blo= b/7c0ad2c33810ead45b7919f8f8d0e282dae52e71/MdePkg/Include/Ppi/SecPlatformIn= formation.h#L137

=C2=A0

I know this only increases the total number of= CPUs that can be handled by a single 64kb HOB, so we would run into=

it again at a higher number of CPUs.=C2= =A0 However, I think this gets the overhead per CPU down to 8 bytes, whi= ch should

scale to about 8091 CPUs.=

=C2=A0

Thanks,

=C2=A0

Mike

=C2=A0

=C2=A0

From: Pawel Polawski <<= a href=3D"mailto:ppolawsk@redhat.com" target=3D"_blank">ppolawsk@redhat.com= >
Sent: Monday, November 7, 2022 3:52 AM
To: devel@= edk2.groups.io
Cc: Dong, Eric <eric.dong@intel.com>; Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>= ;; Kumar, Rahul R <rahul.r.kumar@intel.com>; Kinney, Michael D <michael.d.kinney@intel.= com>
Subject: [edk2-devel] 1024 VCPU limitation

=C2=A0

Hi All,

=C2=A0

I am trying to run edk2 with more than 1024 VCPU. It= looks like it is not possible

at the moment and results in an ASSERT trigger.

=C2=A0

In the past the topic has been analyzed by Laszlo Er= sek [1]. It turns out that the limit

is result of HOB default allocation being limited to= ~64KB, quoting original email thread:

=C2=A0

"""

If "NumberOfProcessors" is large enough, s= uch as ~1024, then
"BistInformationSize" will exceed ~64KB, and PeiServicesAllocateP= ool()
will fail with EFI_OUT_OF_RESOURCES. The reason is that pool allocations in PEI are implemented with memory alloaction HOBs, and HOBs can't be larger than ~64KB. (See PeiAllocatePool() in
"MdeModulePkg/Core/Pei/Memory/MemoryServices.c".)

"""

=C2=A0

Even with HOB allocation being changed, I am afraid = it may break some

compatibility on the DXE level. This is the reason I= am looking for a more universal solution.

I believe the same limitation exists for the physica= l x86 platforms with more than 1024 CPU.

=C2=A0

If someone has encountered the same issue or has kno= wledge that workaround / solution for

this already exists or is being developed?=

=C2=A0

=C2=A0

Best regards,

Pawel


--

Pawe=C5=82 Po=C5=82awski

Red Hat Virtualization

ppolawsk@redhat.com=C2=A0 =C2=A0

@RedHat=C2=A0=C2=A0=C2= =A0Red Hat=C2= =A0=C2=A0Red Hat

--0000000000008dd49505ed763623--