Getting Synchronous Exception while run avocado-vt tests

Getting Synchronous Exception while run avocado-vt tests

[Zhanghailiang]

[-- Attachment #1: Type: text/plain, Size: 11391 bytes --]

Hi All,

We caught an 'Synchronous Exception' error while booting VM with uefi firmware in the avocado-vt tests.
The Edk2 version we used is edk2-stable201905. The qemu version is qemu-4.0.0 and kernel version is 4.19.0.
Parts of the log we got from serial is bellow, you can get the full log from attachment.
We can easily reproduce this issue with running avocado-vt tests. Actually, we tried the new edk2 from upstream,
It is still can be reproduced.

Reproduce command:
# avocado run type_specific.io-github-autotest-qemu.qmp_event_notification --vt-type qemu --vt-guest-os Guest.Linux.Fedora.29

Qemu command is :
/usr/libexec/qemu-kvm -name avocado-vt-vm1 -chardev file,path=/root/vm1_edk2.log,id=charseria_edk -serial chardev:charseria_edk -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/avocado/data/avocado-vt/images/f28-aarch64_AAVMF_VARS.fd,if=pflash,format=raw,unit=1 -machine virt-4.0,gic-version=3 -nodefaults -device virtio-gpu-pci,bus=pcie.0,addr=0x1 -chardev socket,id=hmp_id_hmp1,path=/var/tmp/avocado_pwoki1gy/monitor-hmp1-20190822-144456-BNpsdyCz,server,nowait -mon chardev=hmp_id_hmp1,mode=readline -chardev socket,id=qmp_id_qmp1,path=/var/tmp/avocado_pwoki1gy/monitor-qmp1-20190822-144456-BNpsdyCz,server,nowait -mon chardev=qmp_id_qmp1,mode=control -chardev socket,id=qmp_id_qmp2,path=/var/tmp/avocado_pwoki1gy/monitor-qmp2-20190822-144456-BNpsdyCz,server,nowait -mon chardev=qmp_id_qmp2,mode=control -chardev socket,id=hmp_id_catch_monitor,path=/var/tmp/avocado_pwoki1gy/monitor-catch_monitor-20190822-144456-BNpsdyCz,server,nowait -mon chardev=hmp_id_catch_monitor,mode=readline -serial unix:/var/tmp/avocado_pwoki1gy/serial-serial0-20190822-144456-BNpsdyCz,server,nowait -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 -device virtio-serial-pci,id=virtio_serial_pci0,bus=pcie.0-root-port-2,addr=0x0 -chardev socket,path=/var/tmp/avocado_pwoki1gy/virtio_port-vc1-20190822-144456-BNpsdyCz,id=id8egefs,server,nowait -device virtserialport,id=idmRzd3r,name=org.qemu.guest_agent.0,bus=virtio_serial_pci0.0,chardev=id8egefs -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 -device qemu-xhci,id=usb1,bus=pcie.0-root-port-3,addr=0x0 -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-4,addr=0x0 -drive id=drive_image1,if=none,format=qcow2,file=/var/lib/avocado/data/avocado-vt/images/f28-aarch64.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 -device virtio-net-pci,mac=9a:44:45:46:47:48,rombar=0,id=idqgzz84,vectors=4,netdev=id7rFJf9,bus=pcie.0-root-port-5,addr=0x0 -netdev tap,id=id7rFJf9,fd=22 -m 1024 -smp 2,maxcpus=2,cores=1,threads=1,sockets=2 -cpu host -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -device usb-kbd,id=usb-kbd1,bus=usb1.0,port=2 -vnc :0 -rtc base=utc,clock=host -enable-kvm -device pcie-root-port,id=pcie_extra_root_port_0,slot=6,chassis=6,addr=0x6,bus=pcie.0 -device pcie-root-port,id=pcie_extra_root_port_1,slot=7,chassis=7,addr=0x7,bus=pcie.0

It reports that this is a alignment fault from log, We analyzed the callstack from log:
VirtioScsiPassThru-> VirtioFlush->virtio10SetQueueNotify->Virtio10Transfer->PciIoMemWrite-> CpuMemoryServiceWrite-> MmioWrite32 <- here, the address is not align.
I tried to catch this case with the follow patch, but we never got the debug info while it happens, this is quite confused. :(

     } else if (OperationWidth == EfiCpuIoWidthUint16) {
+      if ((Address & 1) != 0) {^M
+            DEBUG((DEBUG_INFO, "%a: MmioWrite16 address 0x%lx Width %d!!!! not aligh \n", __FUNCTION__, Address, Width));^M
+      }^M
       MmioWrite16 ((UINTN)Address, *((UINT16 *)Uint8Buffer));
     } else if (OperationWidth == EfiCpuIoWidthUint32) {
+      if ((Address & 3) != 0) {^M
+          DEBUG((DEBUG_INFO, "%a: MmioWrite32 address 0x%lx Width %d!!!! not aligh\n", __FUNCTION__, Address, Width));^M
+      }^M
       MmioWrite32 ((UINTN)Address, *((UINT32 *)Uint8Buffer));
     } else if (OperationWidth == EfiCpuIoWidthUint64) {
+       if ((Address & 7) != 0) {^M
+               DEBUG((DEBUG_INFO, "%a: MmioWrite64 address 0x%lx Width %d!!!! not aligh\n", __FUNCTION__, Address, Width));^M
+       }^M
       Mmi
Any idea ?

Synchronous Exception at 0x000000007BD1D318^M
^[[0m^[[37m^[[40m^M
^M
Synchronous Exception at 0x000000007BD1D318^M
PC 0x00007BD1D318 (0x00007BD1B000+0x00002318) [ 0] ArmPciCpuIo2Dxe.dll^M
PC 0x00007BD1D164 (0x00007BD1B000+0x00002164) [ 0] ArmPciCpuIo2Dxe.dll^M
PC 0x00007BD1C868 (0x00007BD1B000+0x00001868) [ 0] ArmPciCpuIo2Dxe.dll^M
PC 0x00007BC7AECC (0x00007BC74000+0x00006ECC) [ 1] PciHostBridgeDxe.dll^M
PC 0x0000785C301C (0x0000785B4000+0x0000F01C) [ 2] PciBusDxe.dll^M
PC 0x00007859B6FC (0x00007859A000+0x000016FC) [ 3] Virtio10.dll^M
PC 0x00007859C0FC (0x00007859A000+0x000020FC) [ 3] Virtio10.dll^M
PC 0x00007BB76F34 (0x00007BB72000+0x00004F34) [ 4] VirtioScsiDxe.dll^M
PC 0x00007BB73F1C (0x00007BB72000+0x00001F1C) [ 4] VirtioScsiDxe.dll^M
PC 0x00007862DA38 (0x00007862B000+0x00002A38) [ 5] ScsiBus.dll^M
PC 0x000078620BB4 (0x000078614000+0x0000CBB4) [ 6] ScsiDisk.dll^M
PC 0x00007861AAC0 (0x000078614000+0x00006AC0) [ 6] ScsiDisk.dll^M
PC 0x000078619E34 (0x000078614000+0x00005E34) [ 6] ScsiDisk.dll^M
PC 0x0000786164D8 (0x000078614000+0x000024D8) [ 6] ScsiDisk.dll^M
PC 0x00007BBDF34C (0x00007BBDC000+0x0000334C) [ 7] DiskIoDxe.dll^M
PC 0x00007BBDFA74 (0x00007BBDC000+0x00003A74) [ 7] DiskIoDxe.dll^M
PC 0x00007BBCD3DC (0x00007BBCB000+0x000023DC) [ 8] PartitionDxe.dll^M
PC 0x00007BBDF34C (0x00007BBDC000+0x0000334C) [ 9] DiskIoDxe.dll^M
PC 0x00007BBDFA74 (0x00007BBDC000+0x00003A74) [ 9] DiskIoDxe.dll^M
PC 0x00007BBB7878 (0x00007BBB4000+0x00003878) [ 10] Fat.dll^M
PC 0x00007BBB7628 (0x00007BBB4000+0x00003628) [ 10] Fat.dll^M
PC 0x00007BBB5BC0 (0x00007BBB4000+0x00001BC0) [ 10] Fat.dll^M
PC 0x00007F4C8F9C (0x00007F4BA000+0x0000EF9C) [ 11] DxeCore.dll^M
PC 0x00007F4C8368 (0x00007F4BA000+0x0000E368) [ 11] DxeCore.dll^M
PC 0x00007F4C86B8 (0x00007F4BA000+0x0000E6B8) [ 11] DxeCore.dll^M
PC 0x00007F4C86B8 (0x00007F4BA000+0x0000E6B8) [ 11] DxeCore.dll^M
PC 0x00007BC508BC (0x00007BC44000+0x0000C8BC) [ 12] BdsDxe.dll^M
PC 0x00007BC5092C (0x00007BC44000+0x0000C92C) [ 12] BdsDxe.dll^M
PC 0x00007BC5F3E8 (0x00007BC44000+0x0001B3E8) [ 12] BdsDxe.dll^M
PC 0x00007BC46EBC (0x00007BC44000+0x00002EBC) [ 12] BdsDxe.dll^M
PC 0x00007F4BC88C (0x00007F4BA000+0x0000288C) [ 13] DxeCore.dll^M
PC 0x00007F4BB8C0 (0x00007F4BA000+0x000018C0) [ 13] DxeCore.dll^M
PC 0x00007F4BB024 (0x00007F4BA000+0x00001024) [ 13] DxeCore.dll^M
^M
[ 0] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/ArmPkg/Drivers/ArmPciCpuIo2Dxe/ArmPciCpuIo2Dxe/DEBUG/ArmPciCpuIo2Dxe.dll^M
[ 1] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridgeDxe/DEBUG/PciHostBridgeDxe.dll^M
[ 2] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe/DEBUG/PciBusDxe.dll^M
[ 3] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/OvmfPkg/Virtio10Dxe/Virtio10/DEBUG/Virtio10.dll^M
[ 4] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/OvmfPkg/VirtioScsiDxe/VirtioScsi/DEBUG/VirtioScsiDxe.dll^M
[ 5] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Bus/Scsi/ScsiBusDxe/ScsiBusDxe/DEBUG/ScsiBus.dll^M
[ 6] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Bus/Scsi/ScsiDiskDxe/ScsiDiskDxe/DEBUG/ScsiDisk.dll^M
[ 7] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Universal/Disk/DiskIoDxe/DiskIoDxe/DEBUG/DiskIoDxe.dll^M
[ 8] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.dll^M
[ 9] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Universal/Disk/DiskIoDxe/DiskIoDxe/DEBUG/DiskIoDxe.dll^M
[ 10] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/FatPkg/EnhancedFatDxe/Fat/DEBUG/Fat.dll^M
[ 11] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll^M
[ 12] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll^M
[ 13] /root/rpmbuild/BUILD/edk2-edk2-stable201905/Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll^M
^M
  X0 0x0000008000100016   X1 0x0000000000000002   X2 0x0000008000100016   X3 0x0000000000000001^M
  X4 0x000000007F4B96E6   X5 0x000000007BD1C798   X6 0x000000007BD23008   X7 0x0000000000000000^M
  X8 0x000000007C01F508   X9 0x0000000400000000  X10 0x00000000792DF000  X11 0x000000007BA57FFF^M
X12 0x0000000000000000  X13 0x0000000000000008  X14 0x0000000000000000  X15 0x0000000000000000^M
X16 0x000000007F4B9E00  X17 0x0000000000000000  X18 0x0000000000000000  X19 0x00000000792F3C28^M
X20 0x0000000000000000  X21 0x0000000000000000  X22 0x0000000000000000  X23 0x0000000000000000^M
X24 0x0000000000000000  X25 0x0000000000000000  X26 0x0000000000000000  X27 0x0000000000000000^M
X28 0x0000000000000000   FP 0x000000007F4B9540   LR 0x000000007BD1D164  ^M
^M
  V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000 0000000000000000^M
  V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 0000000000000000^M
  V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 0000000000000000^M
  V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 0000000000000000^M
  V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 0000000000000000^M
V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 0000000000000000^M
V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 0000000000000000^M
V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 0000000000000000^M
V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 0000000000000000^M
V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 0000000000000000^M
V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 0000000000000000^M
V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 0000000000000000^M
V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 0000000000000000^M
V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 0000000000000000^M
V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 0000000000000000^M
V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 0000000000000000^M
^M
  SP 0x000000007F4B9540  ELR 0x000000007BD1D318  SPSR 0x60000205  FPSR 0x00000000^M
ESR 0x96000021          FAR 0x0000008000100016^M
^M
ESR : EC 0x25  IL 0x1  ISS 0x00000021^M
^M
Data abort: Alignment fault^M

Thanks.

[-- Attachment #2: Type: text/html, Size: 25421 bytes --]

Re: Getting Synchronous Exception while run avocado-vt tests

[Ard Biesheuvel]

On Thu, 22 Aug 2019 at 10:40, Zhanghailiang
<zhang.zhanghailiang@huawei.com> wrote:
>
> Hi All,
>
>
>
> We caught an ‘Synchronous Exception’ error while booting VM with uefi firmware in the avocado-vt tests.
>
> The Edk2 version we used is edk2-stable201905. The qemu version is qemu-4.0.0 and kernel version is 4.19.0.
>
> Parts of the log we got from serial is bellow, you can get the full log from attachment.
>
> We can easily reproduce this issue with running avocado-vt tests. Actually, we tried the new edk2 from upstream,
>
> It is still can be reproduced.
>
>
>
> Reproduce command:
>
> # avocado run type_specific.io-github-autotest-qemu.qmp_event_notification --vt-type qemu --vt-guest-os Guest.Linux.Fedora.29
>
>
>
> Qemu command is :
>
..
>
> It reports that this is a alignment fault from log, We analyzed the callstack from log:
>
> VirtioScsiPassThru-> VirtioFlush->virtio10SetQueueNotify->Virtio10Transfer->PciIoMemWrite-> CpuMemoryServiceWrite-> MmioWrite32 <- here, the address is not align.
>

The faulting address ends in 0x16, so the access is to the QueueSelect
field in VIRTIO_PCI_COMMON_CFG. This is a UINT16 field, so the access
should be 16-bit not 32-bits wide.

Could you dump the instructions leading up to the first
Virtio10Transfer() call in Virtio10SetQueueNotify()? (from
Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/OvmfPkg/Virtio10Dxe/Virtio10/DEBUG/Virtio10.dll)

    2280:       aa0103e5        mov     x5, x1
    2284:       d2800044        mov     x4, #0x2                        // #2
    2288:       d28002c3        mov     x3, #0x16                       // #22
    228c:       52800002        mov     w2, #0x0                        // #0
    2290:       aa0003e1        mov     x1, x0
    2294:       aa0603e0        mov     x0, x6
    2298:       97fffcf3        bl      1664 <Virtio10Transfer>

If the size is passed correctly here, we'll have to track down how the
call gets routed to Mmio32Write instead of Mmio16Write(). Do you have
any patches on top of edk2-stable-201905 ?

Re: [edk2-devel] Getting Synchronous Exception while run avocado-vt tests

[Laszlo Ersek]

On 08/22/19 11:24, Ard Biesheuvel wrote:
> On Thu, 22 Aug 2019 at 10:40, Zhanghailiang
> <zhang.zhanghailiang@huawei.com> wrote:
>>
>> Hi All,
>>
>>
>>
>> We caught an ‘Synchronous Exception’ error while booting VM with uefi firmware in the avocado-vt tests.
>>
>> The Edk2 version we used is edk2-stable201905. The qemu version is qemu-4.0.0 and kernel version is 4.19.0.
>>
>> Parts of the log we got from serial is bellow, you can get the full log from attachment.
>>
>> We can easily reproduce this issue with running avocado-vt tests. Actually, we tried the new edk2 from upstream,
>>
>> It is still can be reproduced.
>>
>>
>>
>> Reproduce command:
>>
>> # avocado run type_specific.io-github-autotest-qemu.qmp_event_notification --vt-type qemu --vt-guest-os Guest.Linux.Fedora.29
>>
>>
>>
>> Qemu command is :
>>
> ..
>>
>> It reports that this is a alignment fault from log, We analyzed the callstack from log:
>>
>> VirtioScsiPassThru-> VirtioFlush->virtio10SetQueueNotify->Virtio10Transfer->PciIoMemWrite-> CpuMemoryServiceWrite-> MmioWrite32 <- here, the address is not align.
>>
> 
> The faulting address ends in 0x16, so the access is to the QueueSelect
> field in VIRTIO_PCI_COMMON_CFG. This is a UINT16 field, so the access
> should be 16-bit not 32-bits wide.
> 
> Could you dump the instructions leading up to the first
> Virtio10Transfer() call in Virtio10SetQueueNotify()? (from
> Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/OvmfPkg/Virtio10Dxe/Virtio10/DEBUG/Virtio10.dll)
> 
>     2280:       aa0103e5        mov     x5, x1
>     2284:       d2800044        mov     x4, #0x2                        // #2
>     2288:       d28002c3        mov     x3, #0x16                       // #22
>     228c:       52800002        mov     w2, #0x0                        // #0
>     2290:       aa0003e1        mov     x1, x0
>     2294:       aa0603e0        mov     x0, x6
>     2298:       97fffcf3        bl      1664 <Virtio10Transfer>
> 
> If the size is passed correctly here, we'll have to track down how the
> call gets routed to Mmio32Write instead of Mmio16Write(). Do you have
> any patches on top of edk2-stable-201905 ?

Right -- checking the "QueueSelect" (whole word) references in
Virtio10SetQueueNotify(), the "FieldSize" arguments passed to
Virtio10Transfer() are:

- sizeof SavedQueueSelect
- sizeof Index
- sizeof SavedQueueSelect

and both "SavedQueueSelect" and "Index" are of type UINT16.

Virtio10Transfer() maps (FieldSize==2) to "EfiPciIoWidthUint16".

PciIoMemWrite() can only decrease "Width" (provided
"PcdUnalignedPciIoEnable" is set to TRUE -- which is not the case in
ArmVirtPkg). So "Width" is passed to RootBridgeIoMemWrite() unchanged,
in "MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c".

The latter passes "Width" unchanged to CpuMemoryServiceWrite(), in
"ArmPkg/Drivers/ArmPciCpuIo2Dxe/ArmPciCpuIo2Dxe.c".

That function seems to set "OperationWidth" to "EfiCpuIoWidthUint16"
(value 1, unchanged), which should result in a call to MmioWrite16()...


I have a different question. We recently saw a bunch of Synchronous
Exceptions, but those were not deterministic. Whenever they fired (which
was not always), they popped up in different spots. It turned out to be
a KVM regression, apparently a problem with the vtimer. I believe it was
fixed by a backport of upstream commit 6bc210003dff ("KVM: arm/arm64:
Don't emulate virtual timers on userspace ioctls", 2019-04-25). I could
be totally off-target, of course.

(The RHBZ is <https://bugzilla.redhat.com/show_bug.cgi?id=1720125>, but
*of course* it has to be a private bug; it was reported for the kernel
after all! /s)

Thanks
Laszlo

Re: [edk2-devel] Getting Synchronous Exception while run avocado-vt tests

[guoheyi]

Hi Ard, Laszlo,

Greetings and thanks for your time to help investigate the issue. 
Finally we found it is caused by KVM and fixed by this patch:

https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git/commit/?id=2113c5f62b7423e4a72b890bd479704aa85c81ba


KVM: arm/arm64: Only skip MMIO insn once
If after an MMIO exit to userspace a VCPU is immediately run with an 
immediate_exit request, such as when a signal is delivered or an MMIO 
emulation completion is needed, then the VCPU completes the MMIO 
emulation and immediately returns to userspace. As the exit_reason does 
not get changed from KVM_EXIT_MMIO in these cases we have to be careful 
not to complete the MMIO emulation again, when the VCPU is eventually 
run again, because the emulation does an instruction skip (and doing too 
many skips would be a waste of guest code :-) We need to use additional 
VCPU state to track if the emulation is complete. As luck would have it, 
we already have 'mmio_needed', which even appears to be used in this way 
by other architectures already. Fixes: 0d640732dbeb ("arm64: KVM: Skip 
MMIO insn after emulation") Acked-by: Mark Rutland 
<mark.rutland@arm.com> Signed-off-by: Andrew Jones <drjones@redhat.com> 
Signed-off-by: Marc Zyngier <maz@kernel.org>

Before this patch, MMIO instructions may be skipped more than once when 
VCPU is requested to exit immediately, and mmio32 assembly function is 
not far from mmio16...

Thanks,

Heyi

On 2019/8/23 2:56, Laszlo Ersek wrote:
> On 08/22/19 11:24, Ard Biesheuvel wrote:
>> On Thu, 22 Aug 2019 at 10:40, Zhanghailiang
>> <zhang.zhanghailiang@huawei.com> wrote:
>>> Hi All,
>>>
>>>
>>>
>>> We caught an ‘Synchronous Exception’ error while booting VM with uefi firmware in the avocado-vt tests.
>>>
>>> The Edk2 version we used is edk2-stable201905. The qemu version is qemu-4.0.0 and kernel version is 4.19.0.
>>>
>>> Parts of the log we got from serial is bellow, you can get the full log from attachment.
>>>
>>> We can easily reproduce this issue with running avocado-vt tests. Actually, we tried the new edk2 from upstream,
>>>
>>> It is still can be reproduced.
>>>
>>>
>>>
>>> Reproduce command:
>>>
>>> # avocado run type_specific.io-github-autotest-qemu.qmp_event_notification --vt-type qemu --vt-guest-os Guest.Linux.Fedora.29
>>>
>>>
>>>
>>> Qemu command is :
>>>
>> ..
>>> It reports that this is a alignment fault from log, We analyzed the callstack from log:
>>>
>>> VirtioScsiPassThru-> VirtioFlush->virtio10SetQueueNotify->Virtio10Transfer->PciIoMemWrite-> CpuMemoryServiceWrite-> MmioWrite32 <- here, the address is not align.
>>>
>> The faulting address ends in 0x16, so the access is to the QueueSelect
>> field in VIRTIO_PCI_COMMON_CFG. This is a UINT16 field, so the access
>> should be 16-bit not 32-bits wide.
>>
>> Could you dump the instructions leading up to the first
>> Virtio10Transfer() call in Virtio10SetQueueNotify()? (from
>> Build/ArmVirtQemu-AARCH64/DEBUG_GCC49/AARCH64/OvmfPkg/Virtio10Dxe/Virtio10/DEBUG/Virtio10.dll)
>>
>>      2280:       aa0103e5        mov     x5, x1
>>      2284:       d2800044        mov     x4, #0x2                        // #2
>>      2288:       d28002c3        mov     x3, #0x16                       // #22
>>      228c:       52800002        mov     w2, #0x0                        // #0
>>      2290:       aa0003e1        mov     x1, x0
>>      2294:       aa0603e0        mov     x0, x6
>>      2298:       97fffcf3        bl      1664 <Virtio10Transfer>
>>
>> If the size is passed correctly here, we'll have to track down how the
>> call gets routed to Mmio32Write instead of Mmio16Write(). Do you have
>> any patches on top of edk2-stable-201905 ?
> Right -- checking the "QueueSelect" (whole word) references in
> Virtio10SetQueueNotify(), the "FieldSize" arguments passed to
> Virtio10Transfer() are:
>
> - sizeof SavedQueueSelect
> - sizeof Index
> - sizeof SavedQueueSelect
>
> and both "SavedQueueSelect" and "Index" are of type UINT16.
>
> Virtio10Transfer() maps (FieldSize==2) to "EfiPciIoWidthUint16".
>
> PciIoMemWrite() can only decrease "Width" (provided
> "PcdUnalignedPciIoEnable" is set to TRUE -- which is not the case in
> ArmVirtPkg). So "Width" is passed to RootBridgeIoMemWrite() unchanged,
> in "MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c".
>
> The latter passes "Width" unchanged to CpuMemoryServiceWrite(), in
> "ArmPkg/Drivers/ArmPciCpuIo2Dxe/ArmPciCpuIo2Dxe.c".
>
> That function seems to set "OperationWidth" to "EfiCpuIoWidthUint16"
> (value 1, unchanged), which should result in a call to MmioWrite16()...
>
>
> I have a different question. We recently saw a bunch of Synchronous
> Exceptions, but those were not deterministic. Whenever they fired (which
> was not always), they popped up in different spots. It turned out to be
> a KVM regression, apparently a problem with the vtimer. I believe it was
> fixed by a backport of upstream commit 6bc210003dff ("KVM: arm/arm64:
> Don't emulate virtual timers on userspace ioctls", 2019-04-25). I could
> be totally off-target, of course.
>
> (The RHBZ is <https://bugzilla.redhat.com/show_bug.cgi?id=1720125>, but
> *of course* it has to be a private bug; it was reported for the kernel
> after all! /s)
>
> Thanks
> Laszlo
>
> 
>
>
> .
>

Re: [edk2-devel] Getting Synchronous Exception while run avocado-vt tests

[Laszlo Ersek]

(+Drew)

On 08/28/19 08:12, Guoheyi wrote:
> Hi Ard, Laszlo,
> 
> Greetings and thanks for your time to help investigate the issue.
> Finally we found it is caused by KVM and fixed by this patch:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git/commit/?id=2113c5f62b7423e4a72b890bd479704aa85c81ba
> 
> 
> 
> KVM: arm/arm64: Only skip MMIO insn once

Heyi, thanks for reporting back!

Funnily enough, Drew's original

  [PATCH] arm64: KVM: Only skip MMIO insn once
  http://mid.mail-archive.com/20190821195030.2569-1-drjones@redhat.com

has been one of the few kvmarm postings that I've even attempted to
read. I congratulated Drew off-list on the debugging and the fix. But, I
never connected the patch to your report on edk2-devel!

  http://mid.mail-archive.com/4e8a0c5f50b642538b310a8edd9ce248@huawei.com

Makes me wonder what other connections I must be missing all the time.
:)

Thank you, Drew, for the fix!
Laszlo