Project

General

Profile

Actions

Bug #3310

open

NVMM+QEMU fail to boot with UEFI: Mem Assist Failed [gpa=0xfffffff0]

Added by liweitianux about 2 years ago. Updated over 1 year ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
nvmm
Target version:
Start date:
01/09/2022
Due date:
% Done:

0%

Estimated time:

Description

NVMM+QEMU fail to boot with UEFI, for example:

% qemu-system-x86_64 \
      -boot menu=on -display sdl -accel nvmm \
      -drive file=OVMF_CODE.fd,if=pflash,format=raw,readonly=on \
      -drive file=OVMF_VARS.fd,if=pflash,format=raw
NetBSD Virtual Machine Monitor accelerator is operational
qemu-system-x86_64: NVMM: Mem Assist Failed [gpa=0xfffff000]
qemu-system-x86_64: NVMM: Failed to execute a VCPU.

The UEFI firmware can be obtained by installing the uefi-edk2-qemu-x86_64 package
or by downloading from: https://leaf.dragonflybsd.org/~aly/uefi/

First reported by Mario Marietto and confirmed by me, see:
https://lists.dragonflybsd.org/pipermail/users/2022-January/404898.html


Files

qemu-system-x86.7z (3.5 MB) qemu-system-x86.7z marietto, 07/15/2022 07:34 AM
Actions #1

Updated by liweitianux about 2 years ago

  • Category set to nvmm
  • Status changed from New to In Progress

A temporary workaround to boot with UEFI in NVMM+QEMU is:
specify the UEFI code with the -bios option instead of the -device if=pflash (or -pflash) option.

However, this is not recommended because the UEFI variables are partially emulated and aren't persistent.
See: https://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00045.html

Actions #2

Updated by liweitianux about 2 years ago

After investigation, the issue is caused by the missing memory maps of the UEFI firmware for the guest.

The UEFI firmware are mapped by QEMU as ROM devices in the ROMD mode. And this is excluded for guest memory mappings in the QEMU NVMM code.

I figured out the following patch that makes NVMM+QEMU to boot with UEFI:

diff --git target/i386/nvmm/nvmm-all.c target/i386/nvmm/nvmm-all.c
index 290077f62..e3c948b31 100644
--- target/i386/nvmm/nvmm-all.c
+++ target/i386/nvmm/nvmm-all.c
@@ -1082,7 +1082,11 @@ nvmm_process_section(MemoryRegionSection *section, int add)
     unsigned int delta;
     uintptr_t hva;

-    if (!memory_region_is_ram(mr)) {
+    /*
+     * Don't exclude ROMD memory; for example, it's used to map UEFI firmware
+     * (if=pflash) and should be mapped for guest.
+     */
+    if (!memory_region_is_ram(mr) && !memory_region_is_romd(mr)) {
         return;
     }

However, NVMM+QEMU is extremely slow and uses 100% CPU. It's even much slower than TCG (i.e., without -accel nvmm).

For example: for QEMU to boot into the UEFI shell on my desktop, it takes ~12 seconds with TCG, but it takes ~170 seconds with NVMM !

Test command:

qemu-system-x86_64 -boot menu=on \
  -drive file=OVMF_CODE.fd,if=pflash,format=raw,readonly=on \
  -drive file=OVMF_VARS.fd,if=pflash,format=raw \
  -display sdl [-accel nvmm]
Actions #3

Updated by marietto about 2 years ago

Hello. I tried using the -bios parameter to add the efi code,like this :

qemu-system-x86_64 \
-machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/OVMF.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022-:22 \
-device virtio-net-pci,netdev=net0 \  
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \ 
-display curses \    
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

but it didn't work :

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

and this :

qemu-system-x86_64 \
-machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022-:22 \
-device virtio-net-pci,netdev=net0 \  
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \ 
-display curses \    
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

qemu: could not load PC BIOS '/home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd'

with this : OVMF_CODE.fd =

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored        

with this :

Actions #4

Updated by marietto about 2 years ago

with this : -bios /usr/pkg/share/qemu/edk2-x86_64-code.fd \

qemu: could not load PC BIOS '/usr/pkg/share/qemu/edk2-x86_64-code.fd'
Actions #5

Updated by tuxillo over 1 year ago

marietto wrote in #note-3:

Hello. I tried using the -bios parameter to add the efi code,like this :

qemu-system-x86_64 \
machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/OVMF.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022
:22 \
-device virtio-net-pci,netdev=net0 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \
-display curses \
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

but it didn't work :

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

and this :

qemu-system-x86_64 \
machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022
:22 \
-device virtio-net-pci,netdev=net0 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \
-display curses \
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

qemu: could not load PC BIOS '/home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd'

with this : OVMF_CODE.fd =

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

with this :

Is there any specific reason why you need UEFI?

Also, what's that "impish cuda" image? Some custom linux? Please remember we do not have hardware devices passthrough.

Actions #6

Updated by marietto over 1 year ago

1) Is there any specific reason why you need UEFI?

Yes,because today every modern hypervisor uses UEFI and not BIOS anymore. In addition,I'm trying to start a collaboration to implement the passthrough. I'm not interested in using a hypervisor that uses the old BIOS bootloader.

2) Also, what's that "impish cuda" image? Some custom linux?

it is only a linux VM (ubuntu impish + cuda 470) that I've previously created for bhyve. I've also tried with windows 11 (also in this case it is a vm created for bhyve) and it gave the same error message.

4) I tried another experiment. I've created a fresh new img file with the command :

qemu-img create -f raw jammy.img 200G

and then I've launched the vm with these parameters :

qemu-system-x86_64 \
-machine type=q35,accel=nvmm \
-smp cpus=4 -m 8G \
-drive if=pflash,format=raw,readonly=on,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_CODE-x86_64.fd \
-drive if=pflash,format=raw,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_VARS-x86_64.fd \
-drive id=cdrom,if=none,media=cdrom,file="ubuntu-22.04-desktop-amd64.iso" \
#-drive file=ubuntu-22.04-desktop-amd64.iso,media=cdrom,id=cdrom \
-drive file=/mnt/da16s1d/home/marietto/Desktop/VMS/jammy.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022-:22 \
-device virtio-net-pci,netdev=net0 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \
-display curses \
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

the error messages are :

root@marietto:/mnt/da16s1d/home/marietto/Desktop/VMS # ./vm2.sh

NetBSD Virtual Machine Monitor accelerator is operational
libGL error: MESA-LOADER: failed to open iris: Cannot open "/usr/local/lib/dri/iris_dri.so" (search paths /usr/local/lib/dri, suffix _dri)
libGL error: failed to load driver: iris
qemu-system-x86_64: NVMM: Mem Assist Failed [gpa=0xfffffff0]
qemu-system-x86_64: NVMM: Failed to execute a VCPU.

Someone wants to debug the core file produced by the bug ? I've attached it.

Actions

Also available in: Atom PDF