From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web11.750.1669766424150132342 for ; Tue, 29 Nov 2022 16:00:24 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PJ1S0HRF; spf=pass (domain: redhat.com, ip: 170.10.133.124, mailfrom: ppolawsk@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1669766423; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jdbmFWa6/GpavRfX5sXVlyO6VGJ2bsyltucw2tFiuoU=; b=PJ1S0HRFRyDuzIQsHTVSdNaItGwxUIgsChkY4zWMfecdAKZBhtKbylX3AEukdSrF5jFz0I 2ZsmACHjZXriQ4VxQCgaUh+JdU0pqVjrVsnOxcImo+323jVSP7Y1Bw7KQVLPVjdgzmbU5U wWbeuF0oIf6mN6iY0+JNjwaydWDxKRA= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-80-EhkiazBiNC-c8HrNMJQ6Iw-1; Tue, 29 Nov 2022 19:00:22 -0500 X-MC-Unique: EhkiazBiNC-c8HrNMJQ6Iw-1 Received: by mail-lf1-f69.google.com with SMTP id i14-20020ac25b4e000000b004b139c2c677so5818433lfp.12 for ; Tue, 29 Nov 2022 16:00:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jdbmFWa6/GpavRfX5sXVlyO6VGJ2bsyltucw2tFiuoU=; b=j8do4flD5WBLYeHxmoGPjnbaLBW84SGhJOvPCPPQXIFJYofxSaTWSC1VFiey4O5xne k/ni0sWSt+E24YKPNXQcjK59Ry9SDD9my9iGIbqTkWmG0qHUTOauuzsNdnKju2W9tXtB vbQF6tq10ylAPQb/QtwL6UeD6WQPoDlI8CfirjNIZ6Dwl9HXO5YK36In6bwd48iY37XE sQCXQwOLnhENyFcKnR0T5AdBhnm/Dli4Xw8n/B0Ac41FisAE30gp3sWegvNVtYL9UW3+ IFf1rA1vE1Arz/z2hspnY/37F2wWl6/5Cbo5LFZtypFHrYXNylxK9VBtAzwGdwO2bn7/ pQOA== X-Gm-Message-State: ANoB5pmQ85xYEaVsg2Fa1dpEqgTvziH4hau+9ksflfrZItahn5uI8zch po6MSCpXNy0/fk8CVRd9lfk4zReC283+NJpuadfsC5Tk/suRWreJ20EK9k43OkQX8xjQ+ilNt11 BgAJtb58RqNWX0w== X-Received: by 2002:ac2:5ddb:0:b0:4a2:500d:f031 with SMTP id x27-20020ac25ddb000000b004a2500df031mr13528657lfq.266.1669766420560; Tue, 29 Nov 2022 16:00:20 -0800 (PST) X-Google-Smtp-Source: AA0mqf7Q2f8EykOvsJbq60h8nj3iL28cyrD3yELzpw29WMnb1nb3hQ08BP/jmsmMYmjTVITmx0bPOA== X-Received: by 2002:ac2:5ddb:0:b0:4a2:500d:f031 with SMTP id x27-20020ac25ddb000000b004a2500df031mr13528642lfq.266.1669766420194; Tue, 29 Nov 2022 16:00:20 -0800 (PST) Return-Path: Received: from [192.168.26.202] ([93.177.91.185]) by smtp.gmail.com with ESMTPSA id b41-20020a0565120ba900b004947984b385sm2383831lfv.87.2022.11.29.16.00.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 29 Nov 2022 16:00:19 -0800 (PST) Message-ID: Date: Wed, 30 Nov 2022 01:00:18 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.5.0 Subject: Re: [edk2-devel] 1024 VCPU limitation To: "Kinney, Michael D" , "devel@edk2.groups.io" , "pedro.falcato@gmail.com" Cc: "Dong, Eric" , Laszlo Ersek , "Ni, Ray" , "Kumar, Rahul R" References: From: =?UTF-8?B?UGF3ZcWCIFBvxYJhd3NraQ==?= In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi all I did fast experiment already and it looks like this Itanium data structure removal solves my issue. Later this week I should have access to machine with 200 physical CPU so I will be able to run test against 1600 vCPU using Qemu. To reproduce test 3 things are needed: 1) Change in the edk2: Removal of Itanium data structure in MdePkg/Include/Ppi/SecPlatformInformation. 2) Change in Qemu: Default limit is 288 / 255 in hw/i386/pc_q35.c and hw/i386/pc.c 3) Change in KVM: Default limit is 1024 in arch/x86/include/asm/kvm_host.h At the end I am using Qemu to run modified firmware: $QEMU \ -accel kvm \ -m 4G -M q35,kernel-irqchip=on,smm=on \ -smp cpus=1024,maxcpus=1024 \ -global mch.extended-tseg-mbytes=128 \ \ -drive if=pflash,format=raw,file=${CODE},readonly=on \ -drive if=pflash,format=raw,file=${VARS} \ \ -chardev stdio,id=fwlog \ -device isa-debugcon,iobase=0x402,chardev=fwlog \ \ "$@" W dniu 15.11.2022 o 01:29, Kinney, Michael D pisze: > Hi Pedro, > > After Pawel runs an experiment, we can think about how to address. > > 1. Code First process for spec change to remove ItaniumHealthFlags field > 2. Code first process for spec change for a new GUID HIB value that > does not have ItaniumHealthFlags field > 3. Code First process to allow multiple instances of the GUIDed HOB. After checking the code it looks like this ITANIUM_HANDOFF_STATUS is only referenced in this header file (MdePkg/Include/Ppi/SecPlatformInformation.h) and nowhere else. I would like to prepare pull request for the mailing list with this data structure removed. From what I saw the whole Itanium support has been already removed back in 2019. You mentioned code first approach, so can I just remove unused part and submit a patch? Or should I introduce intermediate data structure with new GUID to not break any compatibility, and then at some point in the future remove old one? Please advise Best, Pawel > > I prefer option (1) or (2) because is reduces the temp RAM usage in HOBs > as the number of IA32/X64 CPUs increase. > > (3) may be required to scale above 64KB HOB size limit even with the > reduced per CPU size. > > Mike > > *From:*devel@edk2.groups.io *On Behalf Of *Pedro > Falcato > *Sent:* Monday, November 14, 2022 3:30 PM > *To:* devel@edk2.groups.io; Kinney, Michael D > *Cc:* Pawel Polawski ; Dong, Eric > ; Laszlo Ersek ; Ni, Ray > ; Kumar, Rahul R > *Subject:* Re: [edk2-devel] 1024 VCPU limitation > > On Mon, Nov 7, 2022 at 5:28 PM Michael D Kinney > > wrote: > > Hi Pawel, > > I see the following union involved in the size of this structure. > > typedefunion{ > >   IA32_HANDOFF_STATUS       IA32HealthFlags; > >   X64_HANDOFF_STATUS        x64HealthFlags; > >   ITANIUM_HANDOFF_STATUS ItaniumHealthFlags; > > } EFI_SEC_PLATFORM_INFORMATION_RECORD; > > IA32 is 4 bytes per CPU > > X64 is 4 bytes per CPU > > Itanium is 56 bytes per CPU > > We have removed the Itanium content from edk2 repo and it look like > we missed this > > union. > > Hi Mike, > > I just want to note that I don't think you can remove > ITANIUM_HANDOFF_STATUS (in an upstreamable way at least) since it's > specified in the EFI PI spec, and it would also break any sort of ABI. > > Maybe once you update the spec? Or maybe we could find a way to pass > these handoff statuses in multiple HOBs, or have v2 HOBs with UINT32 > lengths (with an appropriate spec update). > > If you comment out the following line from the union does it resolve > the issue? > > https://github.com/tianocore/edk2/blob/7c0ad2c33810ead45b7919f8f8d0e282dae52e71/MdePkg/Include/Ppi/SecPlatformInformation.h#L137 > > I know this only increases the total number of CPUs that can be > handled by a single 64kb HOB, so we would run into > > it again at a higher number of CPUs.  However, I think this gets the > overhead per CPU down to 8 bytes, which should > > scale to about 8091 CPUs. > > Thanks, > > Mike > > *From:* Pawel Polawski > > *Sent:* Monday, November 7, 2022 3:52 AM > *To:* devel@edk2.groups.io > *Cc:* Dong, Eric >; > Laszlo Ersek >; Ni, Ray > >; Kumar, Rahul R > >; Kinney, > Michael D > > *Subject:* [edk2-devel] 1024 VCPU limitation > > Hi All, > > I am trying to run edk2 with more than 1024 VCPU. It looks like it > is not possible > > at the moment and results in an ASSERT trigger. > > In the past the topic has been analyzed by Laszlo Ersek [1]. It > turns out that the limit > > is result of HOB default allocation being limited to ~64KB, quoting > original email thread: > > """ > > If "NumberOfProcessors" is large enough, such as ~1024, then > "BistInformationSize" will exceed ~64KB, and PeiServicesAllocatePool() > will fail with EFI_OUT_OF_RESOURCES. The reason is that pool allocations > in PEI are implemented with memory alloaction HOBs, and HOBs can't be > larger than ~64KB. (See PeiAllocatePool() in > "MdeModulePkg/Core/Pei/Memory/MemoryServices.c".) > > """ > > Even with HOB allocation being changed, I am afraid it may break some > > compatibility on the DXE level. This is the reason I am looking for > a more universal solution. > > I believe the same limitation exists for the physical x86 platforms > with more than 1024 CPU. > > If someone has encountered the same issue or has knowledge that > workaround / solution for > > this already exists or is being developed? > > [1] > https://listman.redhat.com/archives/edk2-devel-archive/2021-June/msg01493 > > Best regards, > > Pawel > > > -- > > *Paweł Poławski* > > Red Hat Virtualization > > ppolawsk@redhat.com > > @RedHat Red Hat > Red Hat > > > > > > > -- > > Pedro Falcato > > >