Re: [edk2-devel] [PATCH v3 0/4] OvmfPkg: CSM boot fixes

["Laszlo Ersek" <lersek@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2862 bytes --]

On 06/26/19 13:43, David Woodhouse wrote:
> On Wed, 2019-06-26 at 12:37 +0100, David Woodhouse wrote:
>> For v3, leaving out the cosmetic parts that touch code outside OvmfPkg. 
>> This series is now purely the correctness fixes within OvmfPkg which are 
>> required to make CSM boots work properly again.
>>
>> The first two patches allow NVMe and VirtIO disks to be used as Legacy
>> boot targets, since nobody really uses IDE any more.
>>
>> The third avoids using QemuVideoDxe when we have CSM, as the INT 10h 
>> shim installed by QemuVideoDxe conflicts with a real legacy video BIOS
>> being installed.
>>
>> Finally, avoid placing PCI BARs above 4GiB. Strictly speaking we only 
>> need this for PCI devices which might be natively supported by the CSM 
>> BIOS, like NVMe. Devices with an OpRom already get special-cased to stay 
>> below 4GiB. But an IncompatiblePciDeviceSupportProtocol implementation 
>> doesn't get to see the PCI device class; only the vendor/device IDs so 
>> we can't use it for that purpose to downgrade more selectively. Instead, 
>> just default to putting everything below 4GiB.
> 
> Btw, these four patches alone are pushed to
> http://git.infradead.org/users/dwmw2/edk2.git/shortlog/refs/heads/csm-ovmfpkg
> 
> But... this cover letter has been detached from the thread with the patches.
> When git-send-email created them, the cover letter had
> Message-Id: <20190626113742.819933-1-dwmw2@infradead.org>
> 
> All the subsequent patches had:
> References: <20190626113742.819933-1-dwmw2@infradead.org>
> 
> But they all get their message-id corrupted on the way through the
> list, it seems. So the References: headers in the patches are
> referencing a Message-ID: that no longer exists. 
> 
> Why does the list do this? Can it be turned off, please?

Yes, it can be turned off. It is a common hiccup for new subscribers
(the groups.io default is broken). I think we meant to document it
somewhere (outside of the mailing list archive), and I guess we may have
even done so, but currently a non-list reference escapes me.

Anyway, please see the attached messages -- and then please log in to
your groups.io account, locate

  Account
    Preferences
      Email Preferences
        My Posts
          I always want copies of my own emails

and *uncheck* it.

Because, in reality, that checkbox stands for "munge my Message-IDs so
that GMail doesn't de-duplicate my own emails when the list reflects
them to me". But, two wrongs don't make a right :/

Thanks,
Laszlo

> Or at the very least, can it be set to mangle References: and In-Reply-
> To: headers to match the corruption it introduces into the Message-Id:
> headers and at least keep threading intact? That still doesn't keep
> things in sync with messages that *don't* go through the list though,
> so fixing it not to do any corruption at all would be best.
> 


[-- Attachment #2: Attached Message --]
[-- Type: message/rfc822, Size: 6900 bytes --]

From: Laszlo Ersek <lersek@redhat.com>
To: Stephano Cetola <stephanoc@gmail.com>
Cc: devel@edk2.groups.io
Subject: Re: [edk2-devel] Groups.io Eval - Step 9
Date: Wed, 6 Mar 2019 10:52:48 +0100
Message-ID: <578e211a-e5f2-6195-46b2-df25ebea0fce@redhat.com>

Hi Stephano,

On 03/05/19 22:29, Laszlo Ersek wrote:

> Thanks. I'll check step 12 soon.
>

>> (12) I check both locally, and in the web archive, whether the
>>      threading is nested (that is, not flattened), over steps (09)
>>      through (11).

Unfortunately, <groups.io> currently fails the threading / nesting
requirement.

This is how the thread looks like, considering those copies of the
messages that we sent to each other directly:

  "I'm emailing the list and CC'ing Laszlo."
  Message-ID: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> <-----+
                                                                                         |
    "This is indeed suitable for filtering."                                             |
    In-Reply-To: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> ---+
    Message-ID: <7969c322-a5ae-fa7d-73ee-600aae34c5de@redhat.com> <------------------------+
                                                                                           |
      "This email should cover us for step 11"                                             |
      In-Reply-To: <7969c322-a5ae-fa7d-73ee-600aae34c5de@redhat.com> ----------------------+
      Message-ID: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> <------+
                                                                                              |
        "Thanks. I'll check step 12 soon"                                                     |
        In-Reply-To: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> ----+
        Message-ID: <c3fe7e05-220a-87e4-f9d2-1421f5742268@redhat.com>

As you can see, the following happened (correctly):
- both your MUA (or SMTP server) and my MUA (or SMTP server) generated a
  fresh Message-ID header whenever a message was sent,
- both your MUA (or SMTP server) and my MUA (or SMTP server) copied the
  Message-ID of the email being responded to into the "In-Reply-To"
  field of the response.

These ensured a complete linkage for the thread, and indeed the thread
appears with correct nesting in my personal mailbox.


In turn, this is how the thread looks like, when we investigate those
copies of the same messages that were reflected by the list (in other
words, those that have the "List-Id: <devel.edk2.groups.io>" header
set):

  "I'm emailing the list and CC'ing Laszlo."
  Message-ID: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> <-----+
                                                                                         |
    "This is indeed suitable for filtering."                                             |
    In-Reply-To: <CAB25Ju54Ad80khPGtMzmVo4ucOysS_wmD9MpGnfRT7m97nZA5g@mail.gmail.com> ---+
    Message-ID: <15891DA65810CEBF.717@groups.io> !!!

      "This email should cover us for step 11"
      In-Reply-To: <7969c322-a5ae-fa7d-73ee-600aae34c5de@redhat.com>
      Message-ID: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> <------+
                                                                                              |
        "Thanks. I'll check step 12 soon"                                                     |
        In-Reply-To: <CAB25Ju62J292FLzEE9xjRfpJKDQ_7S+ZVJHfwBzE69Eiwf9jrA@mail.gmail.com> ----+
        Message-ID: <15892D4A5DAD2B11.29479@groups.io> !!!

Please note the two Message-IDs that I marked with "!!!". <groups.io>
did not honor the Message-IDs that my MUA (or SMTP server) generated: in
the list-reflected copies, <groups.io> replaced those Message-IDs with
new ones that it generated itself. (For some reason, <groups.io> did
this only for messages that were sent by me.)

As a result, in my list folder, the threading is broken. The message
that you sent in response to mine (i.e. where you wrote "This email
should cover us for step 11") does not chain to where I wrote "This is
indeed suitable for filtering", in my list folder.

This must be fixed -- <groups.io> must absolutely honor the original
Message-IDs in *all* messages that it reflects --, otherwise threading
will be broken in everyone's list folder.

Correctly nested threading is also a requirement for posting patches to
the list. (I realize that we might move the patch workflow to another
medium, but that is a separate effort. We should not switch both
services at the same time, but in stages.)

... Ahh, the following discussion entry indicates that <groups.io>
falsifies the Message-ID field *only for the sender* (i.e., only for me
in this case):

  https://groups.io/g/GroupManagersForum/message/8420

And they mean it as a workaround for a GMail bug:

> What that option does is cause Groups.io to replace the message's
> Message-ID field when sending it back to you. That's a recent change:
> it used to replace the field in the copy sent to everyone else as
> well.
>
> This only affects users whose email service (such as Gmail) hides
> messages from you if you already have them, in your inbox or your Sent
> folder. Changing the ID makes it a different message.

This is obviously totally bonkers. GMail screws up threading (by
de-duplicating messages reflected by the list back to the sender), so
let's try to hack around that, via changing the message-id... and in the
process, screw up threading in *conformant* MUAs?!

This insanity must stop. I definitely need my own messages relfected to
me, with the proper List-Id header, and with the original Message-ID
intact. It's pure lunacy that bug compatibility with GMail should break
correctly behaving MUAs.

Do you know if <groups.io> has an account setting that says "send me
copies with original Message-IDs"?

... Well, the text that I quoted above seems to refer to the following
option: "Account | Preferences | Email Preferences | My Posts | I always
want copies of my own emails". This is a good option, but it needs a
sub-option where I can disable the Message-ID regeneration.

Can we report a bug to <groups.io>? (Because, yes, this is a bug, which
they introduced intentionally for working around GMail's madness.)

Thanks!
Laszlo

[-- Attachment #3: Attached Message --]
[-- Type: message/rfc822, Size: 6272 bytes --]

From: Stephano Cetola <stephanoc@gmail.com>
To: Laszlo Ersek <lersek@redhat.com>
Cc: devel@edk2.groups.io
Subject: Re: [edk2-devel] Groups.io Eval - Step 9
Date: Thu, 7 Mar 2019 16:54:28 -0800
Message-ID: <CAB25Ju5ZMbnBAfLshEqcmEt5WfkoGHxfuHzTFfyV=TuYdmzy=Q@mail.gmail.com>

On Wed, Mar 6, 2019 at 1:52 AM Laszlo Ersek <lersek@redhat.com> wrote:

> Do you know if <groups.io> has an account setting that says "send me
> copies with original Message-IDs"?
>
> ... Well, the text that I quoted above seems to refer to the following
> option: "Account | Preferences | Email Preferences | My Posts | I always
> want copies of my own emails". This is a good option, but it needs a
> sub-option where I can disable the Message-ID regeneration.
>
> Can we report a bug to <groups.io>? (Because, yes, this is a bug, which
> they introduced intentionally for working around GMail's madness.)

As it turns out, I believe that check box is simply a poorly worded
checkbox. From their GroupManagersForum:

https://groups.io/g/GroupManagersForum/message/15410

> Groups.io never fails to send your own emailed posts back to your email. Assuming of course that your subscription is set up to receive all messages by email.

I take this to mean that the default setting in their system is to
*always* send all the emails to you, even those emails where you are
the sender.

>
> The issue is that Gmail customarily "hides" them (puts them into your Sent folder, not your Inbox). So most Gmail users' impression (including those using a POP or IMAP client) is that Groups.io isn't returning their messages - unless the Message-ID substitution option (I always want copies of my own emails) in their Account Preferences is checked.

So, in essence, that checkbox should have a parenthetical addendum:
(munge my Message-ID). :)

To fix this issue, we should simply need to uncheck that box. We
should still see our emails, but the Message-IDs will no longer be
changed.

I'm going to subscribe with my linux.intel.com email and test this
theory, but assuming it does work, I believe this will fix our
Message-ID issue.

Cheers,
Stephano

[-- Attachment #4: Attached Message --]
[-- Type: message/rfc822, Size: 4992 bytes --]

From: Stephano Cetola <stephanoc@gmail.com>
To: Laszlo Ersek <lersek@redhat.com>
Cc: devel@edk2.groups.io
Subject: Re: [edk2-devel] Groups.io Eval - Step 9
Date: Thu, 7 Mar 2019 17:15:10 -0800
Message-ID: <CAB25Ju7M28ZQq4a1ectdNYH2HA0TC2PNWv5aONhhG971o=NAew@mail.gmail.com>

On Thu, Mar 7, 2019 at 4:54 PM Stephano Cetola <stephanoc@gmail.com> wrote:
> I'm going to subscribe with my linux.intel.com email and test this
> theory, but assuming it does work, I believe this will fix our
> Message-ID issue.

Verified. If I uncheck that box, my linux.intel.com email will still
receive *all* emails (even those sent from me), however it will *not*
munge the Message-ID.

Let me know if you do not see similar results. I'm glad I used gmail
(webmail) to test this. It is a terrible mailing-list-email
experience, but is a more thorough test. :)

Cheers,
Stephano