From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lucky1.263xmail.com (lucky1.263xmail.com [211.157.147.130]) by mx.groups.io with SMTP id smtpd.web11.4840.1653019263235908635 for ; Thu, 19 May 2022 21:01:04 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: byosoft.net, ip: 211.157.147.130, mailfrom: byomail@byosoft.net) Received: from localhost (unknown [192.168.167.32]) by lucky1.263xmail.com (Postfix) with ESMTP id B11B2E4D08 for ; Fri, 20 May 2022 12:00:54 +0800 (CST) X-MAIL-GRAY: 1 X-MAIL-DELIVERY: 0 X-ADDR-CHECKED4: 1 bpcheck: 1 X-ANTISPAM-LEVEL: 2 X-ABS-CHECKED: 1 Received: from mail.byosoft.com.cn (unknown [58.240.74.242]) by smtp.263.net (postfix) whith ESMTP id P29418T140422194099968S1653019254251702_; Fri, 20 May 2022 12:00:54 +0800 (CST) X-IP-DOMAINF: 1 X-RL-SENDER: byomail@byosoft.net X-SENDER: byomail@byosoft.net X-LOGIN-NAME: byomail@byosoft.net X-FST-TO: devel@edk2.groups.io X-RCPT-COUNT: 1 X-LOCAL-RCPT-COUNT: 0 X-MUTI-DOMAIN-COUNT: 0 X-SENDER-IP: 58.240.74.242 X-ATTACHMENT-NUM: 0 X-UNIQUE-TAG: X-System-Flag: 0 Received: from DESKTOP-M5NI163 ([58.246.60.130]) (envelope-sender ) by 192.168.6.13 with ESMTP for ; Fri, 20 May 2022 11:50:54 +0800 X-WM-Sender: fanjianfeng@byosoft.com.cn X-Originating-IP: 58.246.60.130 X-WM-AuthFlag: YES X-WM-AuthUser: fanjianfeng@byosoft.com.cn Date: Fri, 20 May 2022 11:50:56 +0800 From: "Jeff Fan" To: "devel@edk2.groups.io" , patrick.henz , =?UTF-8?B?TGVuZGFja3ksIFRob21hcw==?= , "min.m.xu@intel.com" , =?UTF-8?B?TmksIFJheQ==?= Cc: "Jiewen Yao" , "Gerd Hoffmann" , "Anthony Perard" , "Julien Grall" , =?UTF-8?B?RG9uZywgRXJpYw==?= Subject: Re: [edk2-devel] [PATCH V7 36/37] UefiCpuPkg: Setting initial-count register as the last step References: , , <2d6f5751-e7de-3c1d-3985-849a9ae16462@amd.com>, , <0389cf2a-e9ba-0ba2-ac90-73ae402f7a08@amd.com>, , , X-Priority: 3 X-Has-Attach: no X-Mailer: Foxmail 7.2.20.269[cn] Mime-Version: 1.0 Message-ID: <2022052011505643485725@byosoft.com.cn> Sender: "fanjianfeng@byosoft.com.cn" Content-Type: multipart/alternative; boundary="----=_001_NextPart762782240678_=----" ------=_001_NextPart762782240678_=---- Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 UmF5LA0KDQpDb3VsZCB3ZSBhZGQgb25lIEFQSSAobGlrZSBJbml0aWFsaXplQXBpY1RpbWVyRXgp IHRvIGludGlhbGl6ZSBDUFUgbG9jYWwgQVBDSSB3aXRoIGFkZGl0aW9uYWwgcGFyYW1ldGVyIHRv IGluZGljYXRlIGlmIGludGVycnVwdCBpcyB0byBiZSBlbmFibGVkIG9yIG5vdD8gSnVzdCBsaWtl IEhQRSdzIHNvbHV0aW9uLiANCg0KSmVmZg0KDQoNCmZhbmppYW5mZW5nQGJ5b3NvZnQuY29tLmNu DQogDQpGcm9tOiBIZW56LCBQYXRyaWNrDQpEYXRlOiAyMDIyLTA1LTIwIDA1OjU0DQpUbzogZGV2 ZWxAZWRrMi5ncm91cHMuaW87IExlbmRhY2t5LCBUaG9tYXM7IG1pbi5tLnh1QGludGVsLmNvbTsg TmksIFJheQ0KQ0M6IFlhbywgSmlld2VuOyBHZXJkIEhvZmZtYW5uOyBBbnRob255IFBlcmFyZDsg SnVsaWVuIEdyYWxsOyBEb25nLCBFcmljDQpTdWJqZWN0OiBSZTogW2VkazItZGV2ZWxdIFtQQVRD SCBWNyAzNi8zN10gVWVmaUNwdVBrZzogU2V0dGluZyBpbml0aWFsLWNvdW50IHJlZ2lzdGVyIGFz IHRoZSBsYXN0IHN0ZXANCkhpIGFsbCwNCiANCldlIChIZXdsZXR0IFBhY2thcmQgRW50ZXJwcmlz ZSkgYXJlIGFsc28gcnVubmluZyBpbnRvIGEgcmFjZSBjb25kaXRpb24gZHVlIHRvIGhvdyBJbml0 aWFsaXplQXBpY1RpbWVyIGluaXRpYWxpemVzIHRoZSBBUElDIHRpbWVycywgd2UgZmlndXJlZCB0 aGlzIG1pZ2h0IGJlIGEgZ29vZCBwbGFjZSB0byByZXBvcnQgb3VyIGZpbmRpbmdzLiBPbiB0aGUg b2NjYXNpb24gd2Ugbm90aWNlIHRoYXQgQVBzIGdldCBzdHVjayBpbiB0aGUgdGltZXIgaW50ZXJy dXB0IGhhbmRsaW5nIGNvZGUgYWZ0ZXIgZ2V0dGluZyB3b2tlbiB1cCBieSB0aGUgQlNQLiBJdCBh cHBlYXJzIHRoYXQgaWYgdGhlIEN1cnJlbnRDb3VudCB0aW1lciB2YWx1ZSBwcm92aWRlZCBieSB0 aGUgQlNQIGlzIHN1ZmZpY2llbnRseSBzbWFsbCwgdGhlIGJyaWVmIGFtb3VudCBvZiB0aW1lIGJl dHdlZW4gYW4gQVAgY2FsbGluZyBJbml0aWFsaXplQXBpY1RpbWVyIGFuZCBjYWxsaW5nIERpc2Fi bGVBcGljVGltZXJJbnRlcnJ1cHQgKHNlZSBTeW5jTG9jYWxBcGljVGltZXJTZXR0aW5nIGFzIGFu IGV4YW1wbGUpIGxlYXZlcyBlbm91Z2ggcm9vbSBmb3IgYW4gQVBJQyB0aW1lciBpbnRlcnJ1cHQg dG8gb2NjdXIuIFRoaXMgc2VlbXMgdG8gYmVjb21lIG1vcmUgZnJlcXVlbnQgb24gbGFyZ2VyIHN5 c3RlbXMgd2l0aCBoaWdoZXIgcHJvY2Vzc29yIGNvdW50cywgZnJvbSB3aGF0IHdlJ3ZlIGdhdGhl cmVkIHRoZSBpbmNyZWFzZSBpbiB0aGUgbnVtYmVyIG9mIGxvY2tpbmcgc2VxdWVuY2UgaW52b2Nh dGlvbnMgYXBwZWFycyB0byBiZSBtYWtpbmcgdGhpcyBjb25kaXRpb24gZmFyIG1vcmUgbGlrZWx5 IHRvIG9jY3VyLiBXZSB3b3JrIG9uIHNjYWxlZCBzeXN0ZW1zIHdpdGggbm9kZSBjb250cm9sbGVy cyBhbmQgd2UndmUgcmVhbGx5IG9ubHkgc2VlbiB0aGlzIG9uIGxhcmdlciBzeXN0ZW1zLCBidXQg aXQgc2VlbXMgdG8gdXMgdGhpcyBjb3VsZCBmZWFzaWJseSBoYXBwZW4gb24gc21hbGxlciBzeXN0 ZW1zIHRvby4gT3VyIGN1cnJlbnQgc29sdXRpb24gaXMgdG8gYWRkIGFuIGFkZGl0aW9uYWwgYXJn dW1lbnQgdG8gSW5pdGlhbGl6ZUFwaWNUaW1lciwgYWxsb3dpbmcgdGhlIGNhbGxlciB0byBzcGVj aWZ5IHdoZXRoZXIgb3Igbm90IEFQSUMgdGltZXIgaW50ZXJydXB0cyBhcmUgdG8gYmUgZW5hYmxl ZCBmb3IgdGhlIGN1cnJlbnQgdGhyZWFkLg0KIA0KVGhhbmtzLA0KUGF0cmljayBIZW56DQogDQpF bnRlcnByaXNlIFg4NiBMYWJzDQpIZXdsZXR0IFBhY2thcmQgRW50ZXJwcmlzZQ0KcGF0cmljay5o ZW56QGhwZS5jb20NCiANCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBkZXZlbEBl ZGsyLmdyb3Vwcy5pbyBbbWFpbHRvOmRldmVsQGVkazIuZ3JvdXBzLmlvXSBPbiBCZWhhbGYgT2Yg TGVuZGFja3ksIFRob21hcyB2aWEgZ3JvdXBzLmlvDQpTZW50OiBGcmlkYXksIE1heSAxMywgMjAy MiA1OjEzIFBNDQpUbzogZGV2ZWxAZWRrMi5ncm91cHMuaW87IG1pbi5tLnh1QGludGVsLmNvbTsg TmksIFJheSA8cmF5Lm5pQGludGVsLmNvbT4NCkNjOiBZYW8sIEppZXdlbiA8amlld2VuLnlhb0Bp bnRlbC5jb20+OyBHZXJkIEhvZmZtYW5uIDxrcmF4ZWxAcmVkaGF0LmNvbT47IEFudGhvbnkgUGVy YXJkIDxhbnRob255LnBlcmFyZEBjaXRyaXguY29tPjsgSnVsaWVuIEdyYWxsIDxqdWxpZW5AeGVu Lm9yZz47IERvbmcsIEVyaWMgPGVyaWMuZG9uZ0BpbnRlbC5jb20+DQpTdWJqZWN0OiBSZTogW2Vk azItZGV2ZWxdIFtQQVRDSCBWNyAzNi8zN10gVWVmaUNwdVBrZzogU2V0dGluZyBpbml0aWFsLWNv dW50IHJlZ2lzdGVyIGFzIHRoZSBsYXN0IHN0ZXANCiANCk9uIDUvMTEvMjIgMTk6NTIsIE1pbiBY dSB2aWEgZ3JvdXBzLmlvIHdyb3RlOg0KPiBPbiBNYXkgMTEsIDIwMjIgMTA6MDYgUE0sIExlbmRh Y2t5LCBUaG9tYXMgd3JvdGU6DQo+PiBPbiA1LzEwLzIyIDIxOjAwLCBYdSwgTWluIE0gd3JvdGU6 DQo+Pj4gT24gTWF5IDExLCAyMDIyIDQ6MzAgQU0sIFRvbSBMZW5kYWNreSB3cm90ZToNCj4+Pj4g SSdtIHJlcGx5aW5nIHRvIHRoaXMgcGF0Y2ggc2luY2UgSSBjYW4ndCBmaW5kIHBhdGNoIFYxMiA0 Ni80NyANCj4+Pj4gYW55d2hlcmUgaW4gbXkgZW1haWwuDQo+Pj4+DQo+Pj4+IEkndmUgYmlzZWN0 ZWQgYSByZWdyZXNzaW9uIGluIHRoZSBMaW51eCBrZXJuZWwgdG8gdGhpcyBwYXRjaCB3aGVuIA0K Pj4+PiBhbiBTRVYtU05QIGd1ZXN0IGlzIGJvb3RlZC4gVGhlIGZvbGxvd2luZyBtZXNzYWdlIGlz IGlzc3VlZCBpbiB0aGUgDQo+Pj4+IGtlcm5lbCBmb3IgZXZlcnkgQVAgYmVpbmcgYnJvdWdodCBv bmxpbmU6DQo+Pj4+DQo+Pj4+IEFQSUM6IFN0YWxlIElSUjoNCj4+Pj4NCj4+IDAwMDAwMDAwLDAw MDAwMDAwLDAwMDAwMDAwLDAwMDAwMDAwLDAwMDAwMDAwLDAwMDAwMDAwLDAwMDAwMDAwLDAwMA0K Pj4+PiAwMDAyMCBJU1I6DQo+Pj4+DQo+PiAwMDAwMDAwMCwwMDAwMDAwMCwwMDAwMDAwMCwwMDAw MDAwMCwwMDAwMDAwMCwwMDAwMDAwMCwwMDAwMDAwMCwwMDANCj4+Pj4gMDAwMDANCj4+Pj4NCj4+ Pj4gUG9zc2libHkgYSB0aW1pbmcgaXNzdWUgaW52b2x2aW5nIHRoZSBtb2RlIHN3aXRjaCB3aXRo IHRoZSANCj4+Pj4gaW50ZXJydXB0IHVubWFza2VkLiBJZiBJIGxlYXZlIHRoZSBpbnRlcnJ1cHQg bWFza2VkIGFuZCBvbmx5IA0KPj4+PiB1bi1tYXNrIGl0IGFmdGVyIHRoZSBwcm9ncmFtbWluZyBv ZiB0aGUgaW5pdC1jb3VudCwgdGhlbiB0aGUgbWVzc2FnZSBnb2VzIGF3YXkuDQo+Pj4NCj4+PiBE byB5b3UgbWVhbiBpbiBJbml0aWFsaXplQXBpY1RpbWVyLCBpdCBzaG91bGQgZm9sbG93IGJlbG93 IHN0ZXBzOg0KPj4+IDEuIG1hc2sgTHZ0VGltZXIuIChzZXQgTHZ0VGltZXIuQml0cy5NYXNrID0g MSkgMi4gRG8gb3RoZXIgc3R1ZmYsIA0KPj4+IGluY2x1ZGluZyBwcm9ncmFtaW5nIHRoZSBpbml0 LWNvdW50IHJlZ2lzdGVyLg0KPj4+IDMuIHVuLW1hc2sgTHZ0VGltZXIgKHNldCBMdnRUaW1lci5C aXQuTWFzayA9IDApDQo+Pg0KPj4gWWVzLCBJIGJlbGlldmUgc28uIEknbSBub3QgYW4gZXhwZXJ0 IG9uIHRoZSBBUElDIHRpbWVyLCBidXQgdGhhdCANCj4+IHNlZW1zIHJlYXNvbmFibGUgdG8gbWUu DQo+IEkgdGVzdGVkIHRoaXMgZml4IGluIFRkIGd1ZXN0IGFuZCBpdCBoYXMgbm8gc2lkZSBlZmZl Y3QuDQo+IEkgY2hlY2sgdGhlIEludGVsIFNETSAoVm9sLjNBIENoYXAgMTAuNSBIYW5kbGluZyBM b2NhbCBJbnRlcnJ1cHRzKSBidXQgaXQgZG9lc24ndCBkZXNjcmliZSB0aGUgYWN0dWFsIHNlcXVl bmNlIG9mIEx2dFRpbWVyLkJpdHMuTWFzayBhbmQgIHByb2dyYW1taW5nIG9mIGluaXQtY291bnQg cmVnaXN0ZXIuDQo+IEAgTmksIFJheSwgIFdoYXQncyB5b3VyIHRob3VnaHQgYWJvdXQgaXQ/DQog DQpJIGd1ZXNzIHlvdSBjYW4gdGhlb3JldGljYWxseSBtaXNzIGFuIGludGVycnVwdCBpZiB5b3Vy IGluaXRpYWwgY291bnQgaXMgZXhwaXJlcyBiZWZvcmUgeW91IHVubWFzayB0aGUgaW50ZXJydXB0 LCBzbyBJIHRoaW5rIHlvdXIgZml4IGlzIGNvcnJlY3QgYW5kIG5vIGNoYW5nZXMgYXJlIG5lZWRl ZC4NCiANCkkgbmVlZCB0byBkb3VibGUgY2hlY2sgd2hldGhlciBJJ20gcHJvcGVybHkgcmVzZXR0 aW5nIHRoZSBBUElDIHdoZW4gQVBzIGFyZSBib290ZWQgbXVsdGlwbGUgdGltZXMuIFNpbmNlIHRo aXMgb25seSBvY2N1cnMgd2l0aCBTTlAsIEkgdGhpbmsgdGhpcyBpcyBvbiBteSBlbmQuDQogDQpU aGFua3MsDQpUb20NCiANCj4gDQo+IFRoYW5rcw0KPiBNaW4NCj4gDQo+IA0KPiANCj4gDQo+IA0K IA0KIA0KIA0KIA0KIA0KIA0KIA0KDQogDQogDQo= ------=_001_NextPart762782240678_=---- Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ray,

Could we add one API (like&= nbsp;InitializeApicTimerEx)&= nbsp;to intialize CPU= local APCI with additional parameter to indicate if interrupt is to be ena= bled or not? Just like HPE's solution. 

Jeff
fanjianfeng@byosoft.com.cn
 
Fr= om: Henz, Patrick
Date: 2022-05-20 05:54
Subject: Re: [edk2-devel] [PA= TCH V7 36/37] UefiCpuPkg: Setting initial-count register as the last step
Hi all,
 
We (Hewlett Packard Enterprise) are also running into a race condition= due to how InitializeApicTimer initializes the APIC timers, we figured thi= s might be a good place to report our findings. On the occasion we notice t= hat APs get stuck in the timer interrupt handling code after getting woken = up by the BSP. It appears that if the CurrentCount timer value provided by = the BSP is sufficiently small, the brief amount of time between an AP calli= ng InitializeApicTimer and calling DisableApicTimerInterrupt (see SyncLocal= ApicTimerSetting as an example) leaves enough room for an APIC timer interr= upt to occur. This seems to become more frequent on larger systems with hig= her processor counts, from what we've gathered the increase in the number o= f locking sequence invocations appears to be making this condition far more= likely to occur. We work on scaled systems with node controllers and we've= really only seen this on larger systems, but it seems to us this could fea= sibly happen on smaller systems too. Our current solution is to add an addi= tional argument to InitializeApicTimer, allowing the caller to specify whet= her or not APIC timer interrupts are to be enabled for the current thread.<= /div>
 
Thanks,
Patrick Henz
 
Enterprise X86 Labs
Hewlett Packard Enterprise
patrick.henz@hpe.com
 
-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of = Lendacky, Thomas via groups.io
Sent: Friday, May 13, 2022 5:13 PM
To: devel@edk2.groups.io; min.m.xu@intel.com; Ni, Ray <ray.ni@intel= .com>
Cc: Yao, Jiewen <jiewen.yao@intel.com>; Gerd Hoffmann <kraxel= @redhat.com>; Anthony Perard <anthony.perard@citrix.com>; Julien G= rall <julien@xen.org>; Dong, Eric <eric.dong@intel.com>
Subject: Re: [edk2-devel] [PATCH V7 36/37] UefiCpuPkg: Setting initial= -count register as the last step
 
On 5/11/22 19:52, Min Xu via groups.io wrote:
> On May 11, 2022 10:06 PM, Lendacky, Thomas wrote:
>> On 5/10/22 21:00, Xu, Min M wrote:
>>> On May 11, 2022 4:30 AM, Tom Lendacky wrote:
>>>> I'm replying to this patch since I can't find patch V= 12 46/47
>>>> anywhere in my email.
>>>>
>>>> I've bisected a regression in the Linux kernel to thi= s patch when
>>>> an SEV-SNP guest is booted. The following message is = issued in the
>>>> kernel for every AP being brought online:
>>>>
>>>> APIC: Stale IRR:
>>>>
>> 00000000,00000000,00000000,00000000,00000000,00000000,0000000= 0,000
>>>> 00020 ISR:
>>>>
>> 00000000,00000000,00000000,00000000,00000000,00000000,0000000= 0,000
>>>> 00000
>>>>
>>>> Possibly a timing issue involving the mode switch wit= h the
>>>> interrupt unmasked. If I leave the interrupt masked a= nd only
>>>> un-mask it after the programming of the init-count, t= hen the message goes away.
>>>
>>> Do you mean in InitializeApicTimer, it should follow belo= w steps:
>>> 1. mask LvtTimer. (set LvtTimer.Bits.Mask =3D 1) 2. Do ot= her stuff,
>>> including programing the init-count register.
>>> 3. un-mask LvtTimer (set LvtTimer.Bit.Mask =3D 0)
>>
>> Yes, I believe so. I'm not an expert on the APIC timer, but t= hat
>> seems reasonable to me.
> I tested this fix in Td guest and it has no side effect.
> I check the Intel SDM (Vol.3A Chap 10.5 Handling Local Interrupts= ) but it doesn't describe the actual sequence of LvtTimer.Bits.Mask and&nbs= p; programming of init-count register.
> @ Ni, Ray,  What's your thought about it?
 
I guess you can theoretically miss an interrupt if your initial count = is expires before you unmask the interrupt, so I think your fix is correct = and no changes are needed.
 
I need to double check whether I'm properly resetting the APIC when AP= s are booted multiple times. Since this only occurs with SNP, I think this = is on my end.
 
Thanks,
Tom
 
>
> Thanks
> Min
>
>
>
>
>
 
 
 
 
 
 
 
 
 
------=_001_NextPart762782240678_=------