Bug #3310: NVMM+QEMU fail to boot with UEFI: Mem Assist Failed [gpa=0xfffffff0] - DragonFlyBSD - DragonFlyBSD bugtracker

Actions

Copy link

Bug #3310

open

NVMM+QEMU fail to boot with UEFI: Mem Assist Failed [gpa=0xfffffff0]

Added by liweitianux over 3 years ago. Updated almost 3 years ago.

Status:

In Progress

Priority:

Normal

Assignee:

Category:

nvmm

Target version:

6.4

Start date:

01/09/2022

Due date:

% Done:

Estimated time:

Description

NVMM+QEMU fail to boot with UEFI, for example:

% qemu-system-x86_64 \
      -boot menu=on -display sdl -accel nvmm \
      -drive file=OVMF_CODE.fd,if=pflash,format=raw,readonly=on \
      -drive file=OVMF_VARS.fd,if=pflash,format=raw
NetBSD Virtual Machine Monitor accelerator is operational
qemu-system-x86_64: NVMM: Mem Assist Failed [gpa=0xfffff000]
qemu-system-x86_64: NVMM: Failed to execute a VCPU.

The UEFI firmware can be obtained by installing the uefi-edk2-qemu-x86_64 package
or by downloading from: https://leaf.dragonflybsd.org/~aly/uefi/

First reported by Mario Marietto and confirmed by me, see:
https://lists.dragonflybsd.org/pipermail/users/2022-January/404898.html

Files

qemu-system-x86.7z (3.5 MB) qemu-system-x86.7z

marietto, 07/15/2022 07:34 AM

Actions

Copy link

Updated by liweitianux over 3 years ago

Category set to nvmm
Status changed from New to In Progress

A temporary workaround to boot with UEFI in NVMM+QEMU is:
specify the UEFI code with the -bios option instead of the -device if=pflash (or -pflash) option.

However, this is not recommended because the UEFI variables are partially emulated and aren't persistent.
See: https://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00045.html

Actions

Copy link

Updated by liweitianux over 3 years ago

After investigation, the issue is caused by the missing memory maps of the UEFI firmware for the guest.

The UEFI firmware are mapped by QEMU as ROM devices in the ROMD mode. And this is excluded for guest memory mappings in the QEMU NVMM code.

I figured out the following patch that makes NVMM+QEMU to boot with UEFI:

diff --git target/i386/nvmm/nvmm-all.c target/i386/nvmm/nvmm-all.c
index 290077f62..e3c948b31 100644
--- target/i386/nvmm/nvmm-all.c
+++ target/i386/nvmm/nvmm-all.c
@@ -1082,7 +1082,11 @@ nvmm_process_section(MemoryRegionSection *section, int add)
     unsigned int delta;
     uintptr_t hva;

-    if (!memory_region_is_ram(mr)) {
+    /*
+     * Don't exclude ROMD memory; for example, it's used to map UEFI firmware
+     * (if=pflash) and should be mapped for guest.
+     */
+    if (!memory_region_is_ram(mr) && !memory_region_is_romd(mr)) {
         return;
     }

However, NVMM+QEMU is extremely slow and uses 100% CPU. It's even much slower than TCG (i.e., without -accel nvmm).

For example: for QEMU to boot into the UEFI shell on my desktop, it takes ~12 seconds with TCG, but it takes ~170 seconds with NVMM !

Test command:

qemu-system-x86_64 -boot menu=on \
  -drive file=OVMF_CODE.fd,if=pflash,format=raw,readonly=on \
  -drive file=OVMF_VARS.fd,if=pflash,format=raw \
  -display sdl [-accel nvmm]

Actions

Copy link

Updated by marietto over 3 years ago

Hello. I tried using the -bios parameter to add the efi code,like this :

qemu-system-x86_64 \
-machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/OVMF.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022-:22 \
-device virtio-net-pci,netdev=net0 \  
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \ 
-display curses \    
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

but it didn't work :

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

and this :

qemu-system-x86_64 \
-machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022-:22 \
-device virtio-net-pci,netdev=net0 \  
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \ 
-display curses \    
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

qemu: could not load PC BIOS '/home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd'

with this : OVMF_CODE.fd =

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

with this :

Actions

Copy link

Updated by marietto over 3 years ago

with this : -bios /usr/pkg/share/qemu/edk2-x86_64-code.fd \

qemu: could not load PC BIOS '/usr/pkg/share/qemu/edk2-x86_64-code.fd'

Actions

Copy link

Updated by tuxillo almost 3 years ago

marietto wrote in #note-3:

Hello. I tried using the -bios parameter to add the efi code,like this :

qemu-system-x86_64 \
machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/OVMF.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022:22 \
-device virtio-net-pci,netdev=net0 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \
-display curses \
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

but it didn't work :

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

and this :

qemu-system-x86_64 \
machine type=q35,accel=nvmm \
-smp cpus=1 -m 8G \
-bios /home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd \
-drive file=/mnt/dk26/bhyve/impish-cuda-11-4-nvidia-470.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022:22 \
-device virtio-net-pci,netdev=net0 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \
-display curses \
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

qemu: could not load PC BIOS '/home/marietto/Desktop/Files/Virt/OVMF/QEMU_UEFI_CODE-x86_64.fd'

with this : OVMF_CODE.fd =

qemu-system-x86_64: NVMM: Unexpected RDMSR 0x3a, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x3a [val=0x1], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x140, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0xce, ignored
qemu-system-x86_64: NVMM: Unexpected WRMSR 0x140 [val=0x0], ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x64e, ignored
qemu-system-x86_64: NVMM: Unexpected RDMSR 0x34, ignored

with this :

Is there any specific reason why you need UEFI?

Also, what's that "impish cuda" image? Some custom linux? Please remember we do not have hardware devices passthrough.

Actions

Copy link

Updated by marietto almost 3 years ago

File qemu-system-x86.7z qemu-system-x86.7z added

1) Is there any specific reason why you need UEFI?

Yes,because today every modern hypervisor uses UEFI and not BIOS anymore. In addition,I'm trying to start a collaboration to implement the passthrough. I'm not interested in using a hypervisor that uses the old BIOS bootloader.

2) Also, what's that "impish cuda" image? Some custom linux?

it is only a linux VM (ubuntu impish + cuda 470) that I've previously created for bhyve. I've also tried with windows 11 (also in this case it is a vm created for bhyve) and it gave the same error message.

4) I tried another experiment. I've created a fresh new img file with the command :

qemu-img create -f raw jammy.img 200G

and then I've launched the vm with these parameters :

qemu-system-x86_64 \
-machine type=q35,accel=nvmm \
-smp cpus=4 -m 8G \
-drive if=pflash,format=raw,readonly=on,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_CODE-x86_64.fd \
-drive if=pflash,format=raw,file=/usr/local/share/uefi-edk2-qemu/QEMU_UEFI_VARS-x86_64.fd \
-drive id=cdrom,if=none,media=cdrom,file="ubuntu-22.04-desktop-amd64.iso" \
#-drive file=ubuntu-22.04-desktop-amd64.iso,media=cdrom,id=cdrom \
-drive file=/mnt/da16s1d/home/marietto/Desktop/VMS/jammy.img,if=none,id=disk0 \
-device virtio-blk-pci,drive=disk0 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:6022-:22 \
-device virtio-net-pci,netdev=net0 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0 \
-display curses \
-vga qxl \
-spice addr=127.0.0.1,port=5900,ipv4=on,disable-ticketing=on,seamless-migration=on

the error messages are :

root@marietto:/mnt/da16s1d/home/marietto/Desktop/VMS # ./vm2.sh

NetBSD Virtual Machine Monitor accelerator is operational
libGL error: MESA-LOADER: failed to open iris: Cannot open "/usr/local/lib/dri/iris_dri.so" (search paths /usr/local/lib/dri, suffix _dri)
libGL error: failed to load driver: iris
qemu-system-x86_64: NVMM: Mem Assist Failed [gpa=0xfffffff0]
qemu-system-x86_64: NVMM: Failed to execute a VCPU.

Someone wants to debug the core file produced by the bug ? I've attached it.

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

DragonFlyBSD

Bug #3310

NVMM+QEMU fail to boot with UEFI: Mem Assist Failed [gpa=0xfffffff0]

Updated by liweitianux over 3 years ago

Updated by liweitianux over 3 years ago

Updated by marietto over 3 years ago

Updated by marietto over 3 years ago

Updated by tuxillo almost 3 years ago

Updated by marietto almost 3 years ago