Re: bootloader only see first disk

[Aurelien Minet <amlabs@free.fr>]

Hi Thomas,

Le 17/09/2018 à 10:34, Thomas Lamprecht a écrit :
> Hi Laszlo,
>
> On 9/16/18 12:04 PM, Laszlo Ersek wrote:
>> Hello Aurélien,
>>
>> adding Thomas to the CC list, and commenting at the bottom:
> thanks for CC'ing and thanks to you Aurélien for writing your report.
>
>> On 09/15/18 18:36, Aurélien wrote:
>>> Hi,
>>>
>>> I don't know if it is the right place to report a bug, sorry if not.
>>>
>>> I'm a Promox user and the last version (5.2) has a package versionned
>>> 20180612-1 actually  (
>>> https://git.proxmox.com/?p=pve-edk2-firmware.git;a=summary ) which is
>>> based on 5a56c0493955cf55e7eef96dbba815cfbb067d7d  commit of edk2. A
>>> build from the current master a few days ago has the same problem. The
>>> previous Promox version (5.1 whith this pve-qemu-kvm package) use a
>>> edk2 version dating from 2017 or before (it could be from before
>>> September 2016 but I'm not sure), which has no problem. I tested on my
>>> workstation running gentoo which has a version based on UDK2017 branch
>>> with source code from February 11, 2018 and it's working too. So I
>>> think the problem is located in  the OVMF_CODE.fd code from 2018 (but
>>> without any certainty).
>>>
>>> Actual behaviour (with all the various Linux distribution: Debian 9,
>>> Ubuntu 18.04, Centos 7.5 ...) with a VM which has EFI enabled:
>>> - grub is loaded and only see the first disk (where it is installed),
>>>   the "ls" command in the grub shell return: "(hd0) (hd0,gpt1)
>>>   (cd0)"   and  there is also hd1 and hd2 and in my case this is
>>>   problematic as /boot is on hd1.
>>> - when pressing "escape" on the boot to go the the setup screen and
>>>   just chose "continue" it's ok (grub sees all the disks), my current
>>>   workaround to boot the installed OS.
>>> - when installing an OS like Ubuntu, CentOS ... on the first reboot
>>>   it's ok (I thinks something in the installation, like seting a new
>>>   boot entry with grub-install, efibootmgr)
>>>
>>> Expected behaviour:
>>> - grub see all the disk (the ls command return hd0, hd1 hd2 ....) and
>>>   not the first one (where the EFI boot partition is located).
>>>   (sorry, I haven't tried any other boot loader)
>>>
>>> - Step to reproduce:
>>>   start a VM with 3 disks and EFI enabled (OVMF_CODE), install a Linux
>>>   on the 2nd disk (/ & swap) and the EFI boot partition on the first
>>>   disk (/boot/efi on "sda1"), cold start the VM.
>>>
>>> I can help by testing with specific version (or another boot loader),
>>> tell me.
>>>
>>> Regards
>>>
>>> Aurélien
>>>
>>> ps: typical command line to start the VM:
>>> /usr/bin/kvm \
>>>   -id 20 \
>>>   -name testvm.mydomain \
>>>   -chardev 'socket,id=qmp,path=/var/run/qemu-server/20.qmp,server,nowait' \
>>>   -mon 'chardev=qmp,mode=control' \
>>>   -pidfile /var/run/qemu-server/20.pid \
>>>   -daemonize \
>>>   -smbios 'type=1,uuid=88dc4e6b-34b8-46a7-a573-26596c34855d' \
>>>   -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
>>>   -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-4.raw' \
>>>   -smp '8,sockets=1,cores=8,maxcpus=8' \
>>>   -nodefaults \
>>>   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
>>>   -vga qxl \
>>>   -vnc unix:/var/run/qemu-server/20.vnc,x509,password \
>>>   -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi \
>>>   -m 512 \
>>>   -object 'memory-backend-ram,id=ram-node0,size=512M' \
>>>   -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' \
>>>   -k fr \
>>>   -object 'iothread,id=iothread-virtioscsi1' \
>>>   -object 'iothread,id=iothread-virtioscsi2' \
>>>   -machine 'type=pc-i440fx-2.9' \
>>>   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
>>>   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
>>>   -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \
>>>   -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
>>>   -chardev 'socket,path=/var/run/qemu-server/20.qga,server,nowait,id=qga0' \
>>>   -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
>>>   -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
>>>   -spice 'tls-port=61004,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' \
>>>   -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' \
>>>   -chardev 'spicevmc,id=vdagent,name=vdagent' \
>>>   -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' \
>>>   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
>>>   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f9e4631950da' \
>>>   -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' \
>>>   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100' \
>>>   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
>>>   -drive 'file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' \
>>>   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200' \
>>>   -device 'virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2,iothread=iothread-virtioscsi1' \
>>>   -drive 'file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-2.raw,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on' \
>>>   -device 'scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' \
>>>   -device 'virtio-scsi-pci,id=virtioscsi2,bus=pci.3,addr=0x3,iothread=iothread-virtioscsi2' \
>>>   -drive 'file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-3.raw,if=none,id=drive-scsi2,format=raw,cache=none,aio=native,detect-zeroes=on' \
>>>   -device 'scsi-hd,bus=virtioscsi2.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2' \
>>>   -netdev 'type=tap,id=net0,ifname=tap20i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
>>>   -device 'virtio-net-pci,mac=4A:63:6D:11:11:11,netdev=net0,bus=pci.0,addr=0x12,id=net0'
>> This is a very good issue report, thank you for it. The amount of detail
>> you provided allows me to explain the status & behavior to you.
>>
>> (1) Originally, OVMF used to connect all drivers to all devices in the
>> system. In a sense this was convenient, however it resulted in a lot of
>> wasted work too -- if you had tens or hundreds of devices, the boot
>> could take extremely long. And again, most of that work was wasted,
>> since you almost never want to actually attempt booting off of tens or
>> hundreds of devices. You only really want to connect those devices that
>> you reasonably want to attempt booting off of as well.
>>
>> (In the above, "connect" is used in the sense "bind" or "probe". Esp.
>> with a Linux background, "probe" is the most fitting word, probably.)
>>
>>
>> (2) Because the set of devices connected at boot is platform policy
>> (i.e., UEFI platforms are at liberty to decide what devices they
>> connect), we changed the above behavior to a minimalistic approach, in
>> the following patch series:
>>
>>   [edk2] [PATCH 0/6] OvmfPkg, ArmVirtQemu: leaner platform BDS policy for connecting devices
>>   http://mid.mail-archive.com/20180313212233.19215-1-lersek@redhat.com
>>
>> This was then pushed as commit range 12957e56d26d..ff1d0fbfbaec, in
>> March 2018.
>>
>> With this set in place, if you provided *some* devices with the
>> 'bootindex=N' property, then OVMF would no longer connect all devices in
>> the system, it would connect only those that you marked with the
>> bootindex property. (And then OVMF would auto-generate UEFI boot options
>> for those only, as well.)
>>
>> In other words, the bootindex property was promoted from its previous
>> "boot option filtering/reordering only" role to a "steering of device
>> probing" role as well.
>>
> Ah yes, I should have figured that out, I even mentioned the faster boot
> on multiple devices in the package changelog but never wondered about what
> this implies for setups like Aurélien has, argh!
>
>> (3) The EFI build of grub most likely doesn't find your disks -- after
>> (2) -- because it doesn't connect devices itself, it relies on the
>> platform firmware to connect disks.
>>
>>
>> (4) If you interrupt the boot process and enter the setup utility, the
>> latter will connect all devices, regardless of (2). This is your current
>> workaround, IIUC.
>>
>>
>> (5) There are many classes of devices, and some of those should be
>> connected even if they are not "bootable" at all. The series in (2)
>> caused a regression for them, and we had to fix them up gradually. See
>> the following commits:
>>
>> - OvmfPkg/PlatformBootManagerLib: add USB keyboard to ConIn
>> - OvmfPkg/PlatformBootManagerLib: connect consoles unconditionally
>> - OvmfPkg/PlatformBootManagerLib: connect Virtio RNG devices again
>>
>> However, the disks that you mention are *not* like this; they should
>> indeed *not* be connected unless you mark them for booting.
>>
>>
>> (6) The solution is either to instruct grub (I don't know how) to
>> connect the disks in question itself, *or* to add the "bootindex=N"
>> properties (with some suitable N values) to the disk devices that you
>> want OVMF to connect for you (even if they contain no EFI system
>> partition and/or a UEFI boot loader application). Given your current
>> command line, the latter option translates to the "scsi-hd" devices with
>> identifiers "scsi1" and "scsi2". For example:
>>
>>   -device scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=201 \
>>   -device scsi-hd,bus=virtioscsi2.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=202 \
> So we probably want to add a per-disk 'bootable' flag in our frontend,
> which adds a higher bootindex (= lower priority) to all marked disks.
> This would allow to still boot faster with lots of disks but allows
> setups like yours, Aurélien.

As actually it is only possible to chose one drive (which add "d"  to
boot property and drive id to bootdisk property).
I was thinking of just being able to chose another drive in the GUI (for
"boot device 3") which would result by adding this drive after the first
one (and a coma) to the bootdisk property.
So your proposition sounds good to me :)  and would be wonderful as it
will support  more possibilities.

Best regards

Aurélien



>
>> Hope this helps,
> Yes, it helps a lot! Much thanks for your explanation, appreciated!
>
> cheers,
> Thomas
>
>