Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

On Wed, 2019-06-26 at 12:37 +0100, David Woodhouse wrote:
> For v3, leaving out the cosmetic parts that touch code outside OvmfPkg. 
> This series is now purely the correctness fixes within OvmfPkg which are 
> required to make CSM boots work properly again.
> 
> The first two patches allow NVMe and VirtIO disks to be used as Legacy
> boot targets, since nobody really uses IDE any more.
> 
> The third avoids using QemuVideoDxe when we have CSM, as the INT 10h 
> shim installed by QemuVideoDxe conflicts with a real legacy video BIOS
> being installed.
> 
> Finally, avoid placing PCI BARs above 4GiB. Strictly speaking we only 
> need this for PCI devices which might be natively supported by the CSM 
> BIOS, like NVMe. Devices with an OpRom already get special-cased to stay 
> below 4GiB. But an IncompatiblePciDeviceSupportProtocol implementation 
> doesn't get to see the PCI device class; only the vendor/device IDs so 
> we can't use it for that purpose to downgrade more selectively. Instead, 
> just default to putting everything below 4GiB.

Btw, these four patches alone are pushed to
http://git.infradead.org/users/dwmw2/edk2.git/shortlog/refs/heads/csm-ovmfpkg

But... this cover letter has been detached from the thread with the patches.
When git-send-email created them, the cover letter had
Message-Id: <20190626113742.819933-1-dwmw2@infradead.org>

All the subsequent patches had:
References: <20190626113742.819933-1-dwmw2@infradead.org>

But they all get their message-id corrupted on the way through the
list, it seems. So the References: headers in the patches are
referencing a Message-ID: that no longer exists. 

Why does the list do this? Can it be turned off, please?

Or at the very least, can it be set to mangle References: and In-Reply-
To: headers to match the corruption it introduces into the Message-Id:
headers and at least keep threading intact? That still doesn't keep
things in sync with messages that *don't* go through the list though,
so fixing it not to do any corruption at all would be best.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]

Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

[Laszlo Ersek]

[-- Attachment #1: Type: text/plain, Size: 2862 bytes --]

On 06/26/19 13:43, David Woodhouse wrote:
> On Wed, 2019-06-26 at 12:37 +0100, David Woodhouse wrote:
>> For v3, leaving out the cosmetic parts that touch code outside OvmfPkg. 
>> This series is now purely the correctness fixes within OvmfPkg which are 
>> required to make CSM boots work properly again.
>>
>> The first two patches allow NVMe and VirtIO disks to be used as Legacy
>> boot targets, since nobody really uses IDE any more.
>>
>> The third avoids using QemuVideoDxe when we have CSM, as the INT 10h 
>> shim installed by QemuVideoDxe conflicts with a real legacy video BIOS
>> being installed.
>>
>> Finally, avoid placing PCI BARs above 4GiB. Strictly speaking we only 
>> need this for PCI devices which might be natively supported by the CSM 
>> BIOS, like NVMe. Devices with an OpRom already get special-cased to stay 
>> below 4GiB. But an IncompatiblePciDeviceSupportProtocol implementation 
>> doesn't get to see the PCI device class; only the vendor/device IDs so 
>> we can't use it for that purpose to downgrade more selectively. Instead, 
>> just default to putting everything below 4GiB.
> 
> Btw, these four patches alone are pushed to
> http://git.infradead.org/users/dwmw2/edk2.git/shortlog/refs/heads/csm-ovmfpkg
> 
> But... this cover letter has been detached from the thread with the patches.
> When git-send-email created them, the cover letter had
> Message-Id: <20190626113742.819933-1-dwmw2@infradead.org>
> 
> All the subsequent patches had:
> References: <20190626113742.819933-1-dwmw2@infradead.org>
> 
> But they all get their message-id corrupted on the way through the
> list, it seems. So the References: headers in the patches are
> referencing a Message-ID: that no longer exists. 
> 
> Why does the list do this? Can it be turned off, please?

Yes, it can be turned off. It is a common hiccup for new subscribers
(the groups.io default is broken). I think we meant to document it
somewhere (outside of the mailing list archive), and I guess we may have
even done so, but currently a non-list reference escapes me.

Anyway, please see the attached messages -- and then please log in to
your groups.io account, locate

  Account
    Preferences
      Email Preferences
        My Posts
          I always want copies of my own emails

and *uncheck* it.

Because, in reality, that checkbox stands for "munge my Message-IDs so
that GMail doesn't de-duplicate my own emails when the list reflects
them to me". But, two wrongs don't make a right :/

Thanks,
Laszlo

> Or at the very least, can it be set to mangle References: and In-Reply-
> To: headers to match the corruption it introduces into the Message-Id:
> headers and at least keep threading intact? That still doesn't keep
> things in sync with messages that *don't* go through the list though,
> so fixing it not to do any corruption at all would be best.
> 


[-- Attachment #2: Attached Message --]
[-- Type: message/rfc822, Size: 6900 bytes --]

From: Laszlo Ersek <lersek@redhat.com>
To: Stephano Cetola <stephanoc@gmail.com>
Cc: devel@edk2.groups.io
Subject: Re: [edk2-devel] Groups.io Eval - Step 9
Date: Wed, 6 Mar 2019 10:52:48 +0100
Message-ID: <578e211a-e5f2-6195-46b2-df25ebea0fce@redhat.com>

Hi Stephano,

On 03/05/19 22:29, Laszlo Ersek wrote:

> Thanks. I'll check step 12 soon.
>

>> (12) I check both locally, and in the web archive, whether the
>>      threading is nested (that is, not flattened), over steps (09)
>>      through (11).

Unfortunately, <groups.io> currently fails the threading / nesting
requirement.

This is how the thread looks like, considering those copies of the
messages that we sent to each other directly:

  "I'm emailing the list and CC'ing Laszlo."
  Message-ID: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> <-----+
                                                                                         |
    "This is indeed suitable for filtering."                                             |
    In-Reply-To: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> ---+
    Message-ID: <7969c322-a5ae-fa7d-73ee-600aae34c5de@redhat.com> <------------------------+
                                                                                           |
      "This email should cover us for step 11"                                             |
      In-Reply-To: <7969c322-a5ae-fa7d-73ee-600aae34c5de@redhat.com> ----------------------+
      Message-ID: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> <------+
                                                                                              |
        "Thanks. I'll check step 12 soon"                                                     |
        In-Reply-To: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> ----+
        Message-ID: <c3fe7e05-220a-87e4-f9d2-1421f5742268@redhat.com>

As you can see, the following happened (correctly):
- both your MUA (or SMTP server) and my MUA (or SMTP server) generated a
  fresh Message-ID header whenever a message was sent,
- both your MUA (or SMTP server) and my MUA (or SMTP server) copied the
  Message-ID of the email being responded to into the "In-Reply-To"
  field of the response.

These ensured a complete linkage for the thread, and indeed the thread
appears with correct nesting in my personal mailbox.


In turn, this is how the thread looks like, when we investigate those
copies of the same messages that were reflected by the list (in other
words, those that have the "List-Id: <devel.edk2.groups.io>" header
set):

  "I'm emailing the list and CC'ing Laszlo."
  Message-ID: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> <-----+
                                                                                         |
    "This is indeed suitable for filtering."                                             |
    In-Reply-To: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> ---+
    Message-ID: <15891DA65810CEBF.717@groups.io> !!!

      "This email should cover us for step 11"
      In-Reply-To: <7969c322-a5ae-fa7d-73ee-600aae34c5de@redhat.com>
      Message-ID: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> <------+
                                                                                              |
        "Thanks. I'll check step 12 soon"                                                     |
        In-Reply-To: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> ----+
        Message-ID: <15892D4A5DAD2B11.29479@groups.io> !!!

Please note the two Message-IDs that I marked with "!!!". <groups.io>
did not honor the Message-IDs that my MUA (or SMTP server) generated: in
the list-reflected copies, <groups.io> replaced those Message-IDs with
new ones that it generated itself. (For some reason, <groups.io> did
this only for messages that were sent by me.)

As a result, in my list folder, the threading is broken. The message
that you sent in response to mine (i.e. where you wrote "This email
should cover us for step 11") does not chain to where I wrote "This is
indeed suitable for filtering", in my list folder.

This must be fixed -- <groups.io> must absolutely honor the original
Message-IDs in *all* messages that it reflects --, otherwise threading
will be broken in everyone's list folder.

Correctly nested threading is also a requirement for posting patches to
the list. (I realize that we might move the patch workflow to another
medium, but that is a separate effort. We should not switch both
services at the same time, but in stages.)

... Ahh, the following discussion entry indicates that <groups.io>
falsifies the Message-ID field *only for the sender* (i.e., only for me
in this case):

  https://groups.io/g/GroupManagersForum/message/8420

And they mean it as a workaround for a GMail bug:

> What that option does is cause Groups.io to replace the message's
> Message-ID field when sending it back to you. That's a recent change:
> it used to replace the field in the copy sent to everyone else as
> well.
>
> This only affects users whose email service (such as Gmail) hides
> messages from you if you already have them, in your inbox or your Sent
> folder. Changing the ID makes it a different message.

This is obviously totally bonkers. GMail screws up threading (by
de-duplicating messages reflected by the list back to the sender), so
let's try to hack around that, via changing the message-id... and in the
process, screw up threading in *conformant* MUAs?!

This insanity must stop. I definitely need my own messages relfected to
me, with the proper List-Id header, and with the original Message-ID
intact. It's pure lunacy that bug compatibility with GMail should break
correctly behaving MUAs.

Do you know if <groups.io> has an account setting that says "send me
copies with original Message-IDs"?

... Well, the text that I quoted above seems to refer to the following
option: "Account | Preferences | Email Preferences | My Posts | I always
want copies of my own emails". This is a good option, but it needs a
sub-option where I can disable the Message-ID regeneration.

Can we report a bug to <groups.io>? (Because, yes, this is a bug, which
they introduced intentionally for working around GMail's madness.)

Thanks!
Laszlo

[-- Attachment #3: Attached Message --]
[-- Type: message/rfc822, Size: 6272 bytes --]

From: Stephano Cetola <stephanoc@gmail.com>
To: Laszlo Ersek <lersek@redhat.com>
Cc: devel@edk2.groups.io
Subject: Re: [edk2-devel] Groups.io Eval - Step 9
Date: Thu, 7 Mar 2019 16:54:28 -0800
Message-ID: <CAB25Ju5ZMbnBAfLshEqcmEt5WfkoGHxfuHzTFfyV=TuYdmzy=Q@mail.gmail.com>

On Wed, Mar 6, 2019 at 1:52 AM Laszlo Ersek <lersek@redhat.com> wrote:

> Do you know if <groups.io> has an account setting that says "send me
> copies with original Message-IDs"?
>
> ... Well, the text that I quoted above seems to refer to the following
> option: "Account | Preferences | Email Preferences | My Posts | I always
> want copies of my own emails". This is a good option, but it needs a
> sub-option where I can disable the Message-ID regeneration.
>
> Can we report a bug to <groups.io>? (Because, yes, this is a bug, which
> they introduced intentionally for working around GMail's madness.)

As it turns out, I believe that check box is simply a poorly worded
checkbox. From their GroupManagersForum:

https://groups.io/g/GroupManagersForum/message/15410

> Groups.io never fails to send your own emailed posts back to your email. Assuming of course that your subscription is set up to receive all messages by email.

I take this to mean that the default setting in their system is to
*always* send all the emails to you, even those emails where you are
the sender.

>
> The issue is that Gmail customarily "hides" them (puts them into your Sent folder, not your Inbox). So most Gmail users' impression (including those using a POP or IMAP client) is that Groups.io isn't returning their messages - unless the Message-ID substitution option (I always want copies of my own emails) in their Account Preferences is checked.

So, in essence, that checkbox should have a parenthetical addendum:
(munge my Message-ID). :)

To fix this issue, we should simply need to uncheck that box. We
should still see our emails, but the Message-IDs will no longer be
changed.

I'm going to subscribe with my linux.intel.com email and test this
theory, but assuming it does work, I believe this will fix our
Message-ID issue.

Cheers,
Stephano

[-- Attachment #4: Attached Message --]
[-- Type: message/rfc822, Size: 4992 bytes --]

From: Stephano Cetola <stephanoc@gmail.com>
To: Laszlo Ersek <lersek@redhat.com>
Cc: devel@edk2.groups.io
Subject: Re: [edk2-devel] Groups.io Eval - Step 9
Date: Thu, 7 Mar 2019 17:15:10 -0800
Message-ID: <CAB25Ju7M28ZQq4a1ectdNYH2HA0TC2PNWv5aONhhG971o=NAew@mail.gmail.com>

On Thu, Mar 7, 2019 at 4:54 PM Stephano Cetola <stephanoc@gmail.com> wrote:
> I'm going to subscribe with my linux.intel.com email and test this
> theory, but assuming it does work, I believe this will fix our
> Message-ID issue.

Verified. If I uncheck that box, my linux.intel.com email will still
receive *all* emails (even those sent from me), however it will *not*
munge the Message-ID.

Let me know if you do not see similar results. I'm glad I used gmail
(webmail) to test this. It is a terrible mailing-list-email
experience, but is a more thorough test. :)

Cheers,
Stephano

Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

On Wed, 2019-06-26 at 14:18 +0200, Laszlo Ersek wrote:
> 
> Yes, it can be turned off. It is a common hiccup for new subscribers
> (the groups.io default is broken). I think we meant to document it
> somewhere (outside of the mailing list archive), and I guess we may have
> even done so, but currently a non-list reference escapes me.
> 
> Anyway, please see the attached messages -- and then please log in to
> your groups.io account, locate
> 
>   Account
>     Preferences
>       Email Preferences
>         My Posts
>           I always want copies of my own emails
> 
> and *uncheck* it.

Thanks.

> Because, in reality, that checkbox stands for "munge my Message-IDs so
> that GMail doesn't de-duplicate my own emails when the list reflects
> them to me". But, two wrongs don't make a right :/

Wow, this is just completely brain-damaged on so many levels. So people
who do, and who don't, have that box checked will receive the same
message with *different* Message-Id: headers?

If one of them replies, the In-Reply-To: threading header in their
reply will refer to a message that *doesn't* exist in the other
person's mailbox. This kind of explains why threading was so broken for
messages on the list, with some response getting 'lost' because they
were detached from the thread. This does not facilitate effective
communication.

Did nobody at groups.io ever stop and think this through? Can it really
not be turned off for the whole list? Why on earth did we move the list
to somewhere that can't even get the *basics* right?

Should I offer to set one up @lists.infradead.org?

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]

Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

On Wed, 2019-06-26 at 13:33 +0100, David Woodhouse wrote:
> 
> Wow, this is just completely brain-damaged on so many levels. So people
> who do, and who don't, have that box checked will receive the same
> message with *different* Message-Id: headers?
> 
> If one of them replies, the In-Reply-To: threading header in their
> reply will refer to a message that *doesn't* exist in the other
> person's mailbox. This kind of explains why threading was so broken for
> messages on the list, with some response getting 'lost' because they
> were detached from the thread. This does not facilitate effective
> communication.
> 
> Did nobody at groups.io ever stop and think this through? Can it really
> not be turned off for the whole list? Why on earth did we move the list
> to somewhere that can't even get the *basics* right?

Ah, so it's not *quite* as broken as that, because it does only mangle
a user's *own* messages back to them; not all messages to them.

So threading works a bit.... except for threads you *participate* in.
Yay!

But at least opting out can mostly isolate an individual from the
adverse effects of this strangeness, and it isn't inflicted upon us
(much) by others who haven't opted out.


> Should I offer to set one up @lists.infradead.org?

Offer still stands :)

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]

Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

[Laszlo Ersek]

On 06/26/19 14:33, David Woodhouse wrote:
> On Wed, 2019-06-26 at 14:18 +0200, Laszlo Ersek wrote:
>>
>> Yes, it can be turned off. It is a common hiccup for new subscribers
>> (the groups.io default is broken). I think we meant to document it
>> somewhere (outside of the mailing list archive), and I guess we may have
>> even done so, but currently a non-list reference escapes me.
>>
>> Anyway, please see the attached messages -- and then please log in to
>> your groups.io account, locate
>>
>>   Account
>>     Preferences
>>       Email Preferences
>>         My Posts
>>           I always want copies of my own emails
>>
>> and *uncheck* it.
> 
> Thanks.
> 
>> Because, in reality, that checkbox stands for "munge my Message-IDs so
>> that GMail doesn't de-duplicate my own emails when the list reflects
>> them to me". But, two wrongs don't make a right :/
> 
> Wow, this is just completely brain-damaged on so many levels. So people
> who do, and who don't, have that box checked will receive the same
> message with *different* Message-Id: headers?
> 
> If one of them replies, the In-Reply-To: threading header in their
> reply will refer to a message that *doesn't* exist in the other
> person's mailbox. This kind of explains why threading was so broken for
> messages on the list, with some response getting 'lost' because they
> were detached from the thread. This does not facilitate effective
> communication.
> 
> Did nobody at groups.io ever stop and think this through? Can it really
> not be turned off for the whole list? Why on earth did we move the list
> to somewhere that can't even get the *basics* right?

We did evaluate groups.io quite carefully, before moving to it. I
proposed a ~15 step plan for the evaluation, and after some tweaking,
groups.io passed it. The eval plan targeted basic mailing list
functionality, and in my judgement, groups.io has worked as a suitable
replacement for the prior lists, ever since we moved.

There were two (sets of) motives for migrating away from the previous
(01.org-based) list, as I recall.

One, list administration for the 01.org owners had been too much of a
chore -- our traffic had been too high for 01.org proportions, spam was
a constant problem, and moderation / whitelisting for non-subscribers
could never be handled effectively. <groups.io> is a lot more flexible
in that regard -- the list / subscriber / messages management that I get
from groups.io is far better than I got from 01.org *in practice*, for
example.

Two, the community wanted a "groupware" solution, with calendars, a
space for uploading/storing design documents (presentations, PDFs),
actually working email attachments, and such. The community also
requested WebUI-based thread filtering / message tagging, IIRC. <01.org>
offered nothing of the sort; <groups.io> looks viable thus far.

My personal requirement was that, with all the above features in place,
groups.io should primarily continue working as a (drop-in replacement)
mailing list. With some account tweaking, I think it functions well in
that regard.

Its web archive for the mailing list has a disastrous UI, admittedly,
but that has been solved by feeding the traffic to other (independent)
archives. <mail-archive.com> is one, and I happen to run another at
<https://www.redhat.com/archives/edk2-devel-archive/> (this latter is
plain mailman2).

Mail-archive.com in particular offers message-id-based search, which is
a hugely useful feature.  Whenever I need to capture a message reference
somewhere, I usually include two -- a msgid-based one, from
mail-archive.com, and another, native to groups.io.


> Should I offer to set one up @lists.infradead.org?

<lists.infradead.org> looks like standard Mailman2.

Mailman2 is good for development mailing lists, and its moderation
features are quite good in my experience -- as long as the list owner
makes those available to list moderators anyway (I'm looking at you,
01.org). However, mailman2 does not offer the "groupware" aspect.

I think we should stick with groups.io for now -- it's not ideal as a
*development* mailing list, but it looks like a suitable compromise,
between multiple goals. There is some pain associated with it when
someone subscribes and tweaks stuff initially, but then it's relatively
painless.

Thanks
Laszlo