Re: [Help] UEFI boot KVM4T vm hang On TianoCore

[Laszlo Ersek <lersek@redhat.com>]

On 11/07/17 03:39, Hangaohuai wrote:
> Hi, Laszlo Ersek;
> Thanks for your reply.
> 
> I have trying to shoot this trouble:
> 
> 1. edk get the MtrrValidBitsMask by "AsmCpuid (CPUID_VIR_PHY_ADDRESS_SIZE, &VirPhyAddressSize.Uint32, NULL, NULL, NULL); "(0x80000008)
> 2. Cpuid(0x80000008) is held in kvm struct (&vcpu->arch.cpuid_entries[i])
> 3. (0x80000008) is set by qemu;
> 
> Part1:
> We set the <maxMemory slots='256' unit='KiB'>4294967296</maxMemory>, the "GetFirstNonAddress: Pci64Base=0x44800000000"
> But the MtrrValidBitsMask get 0x3FF FFFF FFFF;
> Does not pass the check.
>   MtrrLibInitializeMtrrMask (&MtrrValidBitsMask, &MtrrValidAddressMask);
>   if (((BaseAddress & ~MtrrValidAddressMask) != 0) || (Length & ~MtrrValidAddressMask) != 0) {
>     return RETURN_UNSUPPORTED;
>   }
> 
> Part2:
> Xhci_hcd will use the address above maxMemory, so we visit the wrong address.
> Checking by starting a <maxMemory slots='256' unit='KiB'>3294967296</maxMemory> memory with USB3.0.
> 
> So , I think edk2 is ok; maxMemory depend on CPU support more the 42bit.

That's right; if you look at /proc/cpuinfo on your KVM host, it will say
something like:

address sizes   : 39 bits physical, 48 bits virtual
                  ^^^^^^^^^^^^^^^^

(the actual number will vary).

With EPT enabled (see "/sys/module/kvm_intel/parameters/ept"), the guest
cannot have a larger guest-phys address space than what's supported by
the host CPU's physical address width.

Thanks,
Laszlo


> 
> 
> ***********************
> Hi,
> 
> sorry about the delayed response.
> 
> CC'ing Igor, and commenting below:
> 
> On 10/31/17 15:29, Hangaohuai wrote:
>> Hi, Laszlo Ersek;
>>
>> I have tested the uefi booting KVM vm with the configuration(xml); but 
>> start hang.
>> Enable the memoryhotplug, with usb3.0 config. The config as Config1 
>> below.
>>
>> Tested branches:
>> UDK2017     eea98eea4ccbb1d640657770bccb5497fddc6064
>> Master       76fd5a660d704538a1b14a58d03a4eef9682b01c
>>
>> Both hang on the snapshot TianoCore in VNC
>>
>> Try to shoot the problem;
>> I find the early version can boot success with the same config.
>> Maybe the patch1 below cause the problem;
>>
>> Try to ignore the patch does, the master/ UDK2017 both can boot 
>> success.
>> But I don't know why. Hope for your help,thanks.
>>
>> Patch1
>> ***********************************************
>> commit 4f5eff8193096eb847639f090a7dfae3cff95fde
>> Author: Laszlo Ersek <lersek@redhat.com>
>> Date:   Fri Mar 4 20:06:26 2016 +0100
>>
>> OvmfPkg: PciHostBridgeLib: install 64-bit PCI host aperture
>>
>> diff --git a/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c 
>> b/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib
>> index 3e02778..1d3d10a 100644
>> --- a/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c
>> +++ b/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c
>> @@ -132,6 +132,13 @@ InitRootBridge (
>>    RootBus->MemAbove4G.Base      = 0;
>>    RootBus->MemAbove4G.Limit     = 0;
>> +  if (PcdGet64 (PcdPciMmio64Size) > 0) {
>> +    RootBus->AllocationAttributes |= EFI_PCI_HOST_BRIDGE_MEM64_DECODE;
>> +    RootBus->MemAbove4G.Base       = PcdGet64 (PcdPciMmio64Base);
>> +    RootBus->MemAbove4G.Limit      = PcdGet64 (PcdPciMmio64Base) +
>> +                                     (PcdGet64 (PcdPciMmio64Size) - 
>> + 1);  }
>> +
>>    RootBus->Bus.Base  = RootBusNumber;
>>    RootBus->Bus.Limit = MaxSubBusNumber;
>>    RootBus->Io.Base   = PcdGet64 (PcdPciIoBase);
>> diff --git a/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.inf 
>> b/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeL
>> index bbec746..7a964c7 100644
>> --- a/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.inf
>> +++ b/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.inf
>> @@ -51,4 +51,6 @@
>>    gUefiOvmfPkgTokenSpaceGuid.PcdPciIoSize
>>    gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Base
>>    gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Size
>> +  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Base
>> +  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size
>>    gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId
>> ***********************************************
>>
>>
>> Config1
>> ***********************************************
>> <maxMemory slots='256' unit='KiB'>4294967296</maxMemory>
>>   <memory unit='KiB'>4294967</memory>
>>   <currentMemory unit='KiB'>4294967</currentMemory>
>> Xxxx
>> <controller type='usb' index='2' model='nec-xhci'>
>>     <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' 
>> function='0x0'/>  </controller>
>> ***********************************************
> 
> The patch that you identified is responsible for exposing the 64-bit MMIO aperture for PCI MMIO BAR allocation purposes.
> 
> Reverting this patch might mask the issue, but I don't expect the actual problem to be in this patch.
> 
> Instead, the base address and the size of the aperture in question is determined in the file
> 
>   OvmfPkg/PlatformPei/MemDetect.c
> 
> in the function
> 
>   GetFirstNonAddress()
> 
> This function takes the memory (DIMM) hotplug range in question, the end of which is presented by QEMU in the fw_cfg file
> 
>   etc/reserved-memory-end
> 
> To my knowledge, this fw_cfg file is an 8-byte integer in LE encoding.
> 
> Quoting the code:
> 
>>   //
>>   // If QEMU presents an E820 map, then get the highest exclusive >=4GB RAM
>>   // address from it. This can express an address >= 4GB+1TB.
>>   //
>>   // Otherwise, get the flat size of the memory above 4GB from the CMOS (which
>>   // can only express a size smaller than 1TB), and add it to 4GB.
>>   //
>>   Status = ScanOrAdd64BitE820Ram (&FirstNonAddress);
>>   if (EFI_ERROR (Status)) {
>>     FirstNonAddress = BASE_4GB + GetSystemMemorySizeAbove4gb ();
>>   }
>>
>> ...
>>
>>   //
>>   // The "etc/reserved-memory-end" fw_cfg file, when present, contains an
>>   // absolute, exclusive end address for the memory hotplug area. This area
>>   // starts right at the end of the memory above 4GB. The 64-bit PCI host
>>   // aperture must be placed above it.
>>   //
>>   Status = QemuFwCfgFindFile ("etc/reserved-memory-end", &FwCfgItem,
>>              &FwCfgSize);
>>   if (!EFI_ERROR (Status) && FwCfgSize == sizeof HotPlugMemoryEnd) {
>>     QemuFwCfgSelectItem (FwCfgItem);
>>     QemuFwCfgReadBytes (FwCfgSize, &HotPlugMemoryEnd);
>>
>>     ASSERT (HotPlugMemoryEnd >= FirstNonAddress);
>>     FirstNonAddress = HotPlugMemoryEnd;
>>   }
> 
> Can you check if the ASSERT fires for you?
> 
> For that, please capture the OVMF debug log like this:
> 
> (1) build OVMF with the following flag:
> 
>   --pcd=gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel=0x8040004F
> 
> (2) modify your domain XML as follows:
> 
>   <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   <--- don't forget this attribute!
>     <qemu:commandline>
>       <qemu:arg value='-global'/>
>       <qemu:arg value='isa-debugcon.iobase=0x402'/>
>       <qemu:arg value='-debugcon'/>
>       <qemu:arg value='file:/tmp/ovmf.log'/>
>     </qemu:commandline>
>   </domain>
> 
> Then the file "/tmp/ovmf.log" should contain a verbose OVMF log -- please attach it.
> 
> Thanks
> Laszlo
>