Bug #2561

Fwd: Re: DragonFly x86_64 won't boot on qemu-kvm

Added by c.turner1 11 months ago. Updated 11 months ago.

Status:ResolvedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

Forwarding to bugs@ - relavent threads contain
further details.

-------- Original Message --------
Subject: Re: DragonFly x86_64 won't boot on qemu-kvm
Date: Wed, 15 May 2013 21:48:22 -0500
From: Chris Turner <>
To:

On 05/15/13 11:41, Michael Neumann wrote:
> Hi,
>
> I am having exactly the same problems as described in:
>
> http://leaf.dragonflybsd.org/mailarchive/users/2012-10/msg00027.html

Indeed I am also having this problem as well

(RHEL6-stock host/kernel/etc/virt-manager on a core2-duo)

Have not yet reported since I know I haven't had time to get a debug
env properly setup.

I've tested x86-64 from ~1-2wks back, and also had tested around
september last year - have been using 32bit as a workaround since
it suits my purposes in this situation.

> It seem like FreeBSD has a similar problem:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=881579
> http://forums.freebsd.org/showthread.php?t=36761

I'm not clear from your email - did any of the workarounds work?

setting cpu flags, the kvm modprobe option, etc?

I was able to boot up when using 'pure emulation' QEMU,
(as outlined in original thread IIRC) but this is hardly ideal :D

> For now, the only solution seems to be to use i386 on that particual host (or wait that this is fixed in kvm).
> I am wondering why it fails on DF but not for example on Linux. Are they doing things differently?

It does appear that the panic happens consistently for me after 'low level' init
and right before mounting the root filesystem - IIRC this is when the actual
buffer cache/VM etc is setup, and so presumably lots would be going on w/r/t
memory pages/interrupts/etc in a fairly OS-dependent manner.

My thought was to add some spurious printfs here and maybe throw in some ASM
noops in the right places to see if that allows any narrowing down of the
problem, but again, I haven't had a chance to get a proper dev env setup
in this environment.

Unfortunately my time will likely be short until at least mid summer.
GRR! need more unix time!

... and while I'm at it - this + Bug#2133, etc.
seem like excellent examples of the 'why keeping >=2 platforms going is a good idea'
argument.

</soapbox>

will forward to bugs@ to get a thread going there.

Cheers,

- Chris

History

#1 Updated by vsrinivas 11 months ago

I think we tracked this down this morning, a fix is being tested.

I think the problem boils down to:
http://nxr.netbsd.org/xref/src-dragonflybsd/sys/platform/pc64/x86_64/vm_machdep.c#143
pcb2->pcb_cr3 |= PG_RW | PG_U | PG_V;

On x86 and x86-64, the CR3 register points to the root of the page tables for an address space. The bottom three bits of the address in CR3 are supposed to be zero, (AMD64 documentation states 'should be zero'), but DFly has been setting them with the corresponding flags from page table entries. This has worked out okay on real hardware, as the AMD docs only say 'should be zero' (Intel says 'ignored').

When (qemu-)KVM is used on hardware without two-dimensional paging, it has to handle writes to CR3 itself. The code to do so is in Linux's arch/x86/kvm/emulate.c, check_cr_write(). If any of the reserved bits in CR3 are set on a store, we see a #GP. This is perhaps arguably a bug.

To fix this, just remove that line in vm_machdep.c and try it.

#2 Updated by mneumann 11 months ago

Am 16.05.2013 04:49, schrieb Chris Turner via Redmine:
> Issue #2561 has been reported by c.turner1.
>
> ----------------------------------------
> Bug #2561: Fwd: Re: DragonFly x86_64 won't boot on qemu-kvm
> http://bugs.dragonflybsd.org/issues/2561
>
> Author: c.turner1
> Status: New
> Priority: Normal
> Assignee:
> Category:
> Target version:
>
>
> Forwarding to bugs@ - relavent threads contain
> further details.
>
> -------- Original Message --------
> Subject: Re: DragonFly x86_64 won't boot on qemu-kvm
> Date: Wed, 15 May 2013 21:48:22 -0500
> From: Chris Turner <>
> To:
>
> On 05/15/13 11:41, Michael Neumann wrote:
>> Hi,
>>
>> I am having exactly the same problems as described in:
>>
>> http://leaf.dragonflybsd.org/mailarchive/users/2012-10/msg00027.html
> Indeed I am also having this problem as well
>
> (RHEL6-stock host/kernel/etc/virt-manager on a core2-duo)
>
> Have not yet reported since I know I haven't had time to get a debug
> env properly setup.
>
> I've tested x86-64 from ~1-2wks back, and also had tested around
> september last year - have been using 32bit as a workaround since
> it suits my purposes in this situation.
>
>
>> It seem like FreeBSD has a similar problem:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=881579
>> http://forums.freebsd.org/showthread.php?t=36761
> I'm not clear from your email - did any of the workarounds work?
>
> setting cpu flags, the kvm modprobe option, etc?

No, nothing worked. It took me half a day until I realized that :)

> I was able to boot up when using 'pure emulation' QEMU,
> (as outlined in original thread IIRC) but this is hardly ideal :D

I did not try pure emulation, but I am very sure that it would work for
me as well,
but as the system is a server I did not take this "solution" into account.

>> For now, the only solution seems to be to use i386 on that particual host (or wait that this is fixed in kvm).
>> I am wondering why it fails on DF but not for example on Linux. Are they doing things differently?
> It does appear that the panic happens consistently for me after 'low level' init
> and right before mounting the root filesystem - IIRC this is when the actual
> buffer cache/VM etc is setup, and so presumably lots would be going on w/r/t
> memory pages/interrupts/etc in a fairly OS-dependent manner.
>
> My thought was to add some spurious printfs here and maybe throw in some ASM
> noops in the right places to see if that allows any narrowing down of the
> problem, but again, I haven't had a chance to get a proper dev env setup
> in this environment.
>
> Unfortunately my time will likely be short until at least mid summer.
> GRR! need more unix time!

I think Venkatesh has found the bug, so there is no need for that. In
this moment I am compiling
an ISO image with his suggested change. It takes some time to compile
DragonFly inside a virtual
machine on this low-end laptop :). Stay tuned.

Regards,

Michael

#3 Updated by vsrinivas 11 months ago

aggelos@ found a second place where there was an offending %cr3 register, in the x86-64 pmap:

http://leaf.dragonflybsd.org/~aggelos/0002-kernel-x86_64-Do-not-set-reserved-bits-in-CR3.patch

If you apply that and the original:
http://acm.jhu.edu/~me/0001-kernel-x86_64-Do-not-set-reserved-bits-in-CR3.patch

it works on qemu-kvm 1.1 on a Linux 3.2 host.

If folks could test these two patches together on a few physical x86-64 systems too, to make sure that it doesn't break them, that would be really helpful.

#4 Updated by mneumann 11 months ago

Am 19.05.2013 08:31, schrieb Venkatesh Srinivas via Redmine:
> Issue #2561 has been updated by vsrinivas.
>
>
> aggelos@ found a second place where there was an offending %cr3 register, in the x86-64 pmap:
>
> http://leaf.dragonflybsd.org/~aggelos/0002-kernel-x86_64-Do-not-set-reserved-bits-in-CR3.patch
>
> If you apply that and the original:
> http://acm.jhu.edu/~me/0001-kernel-x86_64-Do-not-set-reserved-bits-in-CR3.patch
>
> it works on qemu-kvm 1.1 on a Linux 3.2 host.
>
> If folks could test these two patches together on a few physical x86-64 systems too, to make sure that it doesn't break them, that would be really helpful.

Hi Vekatesh,

I can confirm that these two patches work on qemu-kvm. Awesome work!
Thank you so much!

Regards,

Michael

#5 Updated by aoiko 11 months ago

  • Status changed from New to Resolved

Should be fixed in 5e73d5fc48f6c48d9a638404bfea3e7040601f8e (http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/5e73d5fc48f6c48d9a638404bfea3e7040601f8e).

Aggelos

#6 Updated by emil 11 months ago

I cherry-picked commit 5e73d and tested, and it definitely fixes the problem I reported in http://leaf.dragonflybsd.org/mailarchive/users/2012-10/msg00027.html

Excellent work! Thank you!

Also available in: Atom PDF