Re: bootloader only see first disk

[Laszlo Ersek <lersek@redhat.com>]

Hello Aurélien,

adding Thomas to the CC list, and commenting at the bottom:

On 09/15/18 18:36, Aurélien wrote:
> Hi,
>
> I don't know if it is the right place to report a bug, sorry if not.
>
> I'm a Promox user and the last version (5.2) has a package versionned
> 20180612-1 actually  (
> https://git.proxmox.com/?p=pve-edk2-firmware.git;a=summary ) which is
> based on 5a56c0493955cf55e7eef96dbba815cfbb067d7d  commit of edk2. A
> build from the current master a few days ago has the same problem. The
> previous Promox version (5.1 whith this pve-qemu-kvm package) use a
> edk2 version dating from 2017 or before (it could be from before
> September 2016 but I'm not sure), which has no problem. I tested on my
> workstation running gentoo which has a version based on UDK2017 branch
> with source code from February 11, 2018 and it's working too. So I
> think the problem is located in  the OVMF_CODE.fd code from 2018 (but
> without any certainty).
>
> Actual behaviour (with all the various Linux distribution: Debian 9,
> Ubuntu 18.04, Centos 7.5 ...) with a VM which has EFI enabled:
> - grub is loaded and only see the first disk (where it is installed),
>   the "ls" command in the grub shell return: "(hd0) (hd0,gpt1)
>   (cd0)"   and  there is also hd1 and hd2 and in my case this is
>   problematic as /boot is on hd1.
> - when pressing "escape" on the boot to go the the setup screen and
>   just chose "continue" it's ok (grub sees all the disks), my current
>   workaround to boot the installed OS.
> - when installing an OS like Ubuntu, CentOS ... on the first reboot
>   it's ok (I thinks something in the installation, like seting a new
>   boot entry with grub-install, efibootmgr)
>
> Expected behaviour:
> - grub see all the disk (the ls command return hd0, hd1 hd2 ....) and
>   not the first one (where the EFI boot partition is located).
>   (sorry, I haven't tried any other boot loader)
>
> - Step to reproduce:
>   start a VM with 3 disks and EFI enabled (OVMF_CODE), install a Linux
>   on the 2nd disk (/ & swap) and the EFI boot partition on the first
>   disk (/boot/efi on "sda1"), cold start the VM.
>
> I can help by testing with specific version (or another boot loader),
> tell me.
>
> Regards
>
> Aurélien
>
> ps: typical command line to start the VM:
> /usr/bin/kvm \
>   -id 20 \
>   -name testvm.mydomain \
>   -chardev 'socket,id=qmp,path=/var/run/qemu-server/20.qmp,server,nowait' \
>   -mon 'chardev=qmp,mode=control' \
>   -pidfile /var/run/qemu-server/20.pid \
>   -daemonize \
>   -smbios 'type=1,uuid=88dc4e6b-34b8-46a7-a573-26596c34855d' \
>   -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
>   -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-4.raw' \
>   -smp '8,sockets=1,cores=8,maxcpus=8' \
>   -nodefaults \
>   -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
>   -vga qxl \
>   -vnc unix:/var/run/qemu-server/20.vnc,x509,password \
>   -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi \
>   -m 512 \
>   -object 'memory-backend-ram,id=ram-node0,size=512M' \
>   -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' \
>   -k fr \
>   -object 'iothread,id=iothread-virtioscsi1' \
>   -object 'iothread,id=iothread-virtioscsi2' \
>   -machine 'type=pc-i440fx-2.9' \
>   -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' \
>   -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' \
>   -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' \
>   -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
>   -chardev 'socket,path=/var/run/qemu-server/20.qga,server,nowait,id=qga0' \
>   -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
>   -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
>   -spice 'tls-port=61004,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' \
>   -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' \
>   -chardev 'spicevmc,id=vdagent,name=vdagent' \
>   -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' \
>   -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
>   -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f9e4631950da' \
>   -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' \
>   -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100' \
>   -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1' \
>   -drive 'file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' \
>   -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200' \
>   -device 'virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2,iothread=iothread-virtioscsi1' \
>   -drive 'file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-2.raw,if=none,id=drive-scsi1,format=raw,cache=none,aio=native,detect-zeroes=on' \
>   -device 'scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' \
>   -device 'virtio-scsi-pci,id=virtioscsi2,bus=pci.3,addr=0x3,iothread=iothread-virtioscsi2' \
>   -drive 'file=/mnt/pve/pve-nfs-sas/images/20/vm-20-disk-3.raw,if=none,id=drive-scsi2,format=raw,cache=none,aio=native,detect-zeroes=on' \
>   -device 'scsi-hd,bus=virtioscsi2.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2' \
>   -netdev 'type=tap,id=net0,ifname=tap20i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
>   -device 'virtio-net-pci,mac=4A:63:6D:11:11:11,netdev=net0,bus=pci.0,addr=0x12,id=net0'

This is a very good issue report, thank you for it. The amount of detail
you provided allows me to explain the status & behavior to you.

(1) Originally, OVMF used to connect all drivers to all devices in the
system. In a sense this was convenient, however it resulted in a lot of
wasted work too -- if you had tens or hundreds of devices, the boot
could take extremely long. And again, most of that work was wasted,
since you almost never want to actually attempt booting off of tens or
hundreds of devices. You only really want to connect those devices that
you reasonably want to attempt booting off of as well.

(In the above, "connect" is used in the sense "bind" or "probe". Esp.
with a Linux background, "probe" is the most fitting word, probably.)


(2) Because the set of devices connected at boot is platform policy
(i.e., UEFI platforms are at liberty to decide what devices they
connect), we changed the above behavior to a minimalistic approach, in
the following patch series:

  [edk2] [PATCH 0/6] OvmfPkg, ArmVirtQemu: leaner platform BDS policy for connecting devices
  http://mid.mail-archive.com/20180313212233.19215-1-lersek@redhat.com

This was then pushed as commit range 12957e56d26d..ff1d0fbfbaec, in
March 2018.

With this set in place, if you provided *some* devices with the
'bootindex=N' property, then OVMF would no longer connect all devices in
the system, it would connect only those that you marked with the
bootindex property. (And then OVMF would auto-generate UEFI boot options
for those only, as well.)

In other words, the bootindex property was promoted from its previous
"boot option filtering/reordering only" role to a "steering of device
probing" role as well.


(3) The EFI build of grub most likely doesn't find your disks -- after
(2) -- because it doesn't connect devices itself, it relies on the
platform firmware to connect disks.


(4) If you interrupt the boot process and enter the setup utility, the
latter will connect all devices, regardless of (2). This is your current
workaround, IIUC.


(5) There are many classes of devices, and some of those should be
connected even if they are not "bootable" at all. The series in (2)
caused a regression for them, and we had to fix them up gradually. See
the following commits:

- OvmfPkg/PlatformBootManagerLib: add USB keyboard to ConIn
- OvmfPkg/PlatformBootManagerLib: connect consoles unconditionally
- OvmfPkg/PlatformBootManagerLib: connect Virtio RNG devices again

However, the disks that you mention are *not* like this; they should
indeed *not* be connected unless you mark them for booting.


(6) The solution is either to instruct grub (I don't know how) to
connect the disks in question itself, *or* to add the "bootindex=N"
properties (with some suitable N values) to the disk devices that you
want OVMF to connect for you (even if they contain no EFI system
partition and/or a UEFI boot loader application). Given your current
command line, the latter option translates to the "scsi-hd" devices with
identifiers "scsi1" and "scsi2". For example:

  -device scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=201 \
  -device scsi-hd,bus=virtioscsi2.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=202 \

Hope this helps,
Laszlo