https://bugs.dragonflybsd.org/https://bugs.dragonflybsd.org/favicon.ico?16293952082008-06-07T05:43:03ZDragonFlyBSD bugtrackerDragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47462008-06-07T05:43:03Zdragonfly1
<ul></ul><p>I've managed to get gdb attached and get some information.</p>
<p>The kernel is getting caught in a while loop in lwkt_acquire. I can <br />reliably trigger this with with a "make -j8 buildworld" under a SMP <br />kernel (Otherwise identical to GENERIC, no optimisations.) The OS is <br />completely unresponsive and all four cpu cores are running at 100%.</p>
<p>I've included the debug information.</p>
<p>Program received signal SIGINT, Interrupt.<br />lwkt_acquire (td=0xc6a59e70) at /usr/src/sys/kern/lwkt_thread.c:1048<br />1048 while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK))<br />(gdb) l<br />1043 mygd = mycpu;<br />1044 if (gd != mycpu) {<br />1045 cpu_lfence();<br />1046 <abbr title="(td->td_flags & TDF_RUNQ">KKASSERT</abbr> == 0);<br />1047 crit_enter_gd(mygd);<br />1048 while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK))<br />1049 cpu_lfence();<br />1050 td->td_gd = mygd;<br />1051 TAILQ_INSERT_TAIL(&mygd->gd_tdallq, td, td_allq);<br />1052 td->td_flags &= ~TDF_MIGRATING;<br />(gdb) p td->td_flags<br />$1 = 8390177<br />(gdb) p td<br />$2 = (thread_t) 0xc6a59e70<br />(gdb) bt<br />#0 lwkt_acquire (td=0xc6a59e70) at /usr/src/sys/kern/lwkt_thread.c:1048<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c66af in bsd4_select_curproc (gd=0xff800000) at <br />/usr/src/sys/kern/usched_bsd4.c:358<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c6829 in bsd4_release_curproc (lp=0xea634c00) at <br />/usr/src/sys/kern/usched_bsd4.c:322<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc04b8239 in passive_release (td=0xdfe8aba0) at <br />/usr/src/sys/platform/pc32/i386/trap.c:212<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c870b in lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:491<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c8b3b in lwkt_mp_lock_contested () at <br />/usr/src/sys/kern/lwkt_thread.c:1374<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b0751 in get_mplock () at <br />/usr/src/sys/platform/pc32/i386/mplock.s:168<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xe9ef6d34 in ?? ()<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xc04b94a4 in syscall2 (frame=0xe9ef6d40) at <br />/usr/src/sys/platform/pc32/i386/trap.c:1371<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: panic with HEAD (Closed)" href="https://bugs.dragonflybsd.org/issues/9">#9</a> 0xc04a3396 in Xint0x80_syscall () at <br />/usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: make buildworld broken (Closed)" href="https://bugs.dragonflybsd.org/issues/10">#10</a> 0xe9ef6d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) jump 1050<br />Continuing at 0xc02c8bbb.</p>
<p>Continuing execution does not appear to cause any problems.<br />I can provide additional debugging info if required but I'm unsure of <br />how to proceed with this myself.</p>
<p>Regards</p>
<p>Gary</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47472008-06-07T05:52:00Zdillon
<ul></ul><p>:I've managed to get gdb attached and get some information.<br />:<br />:The kernel is getting caught in a while loop in lwkt_acquire. I can <br />:reliably trigger this with with a "make <del>j8 buildworld" under a SMP <br />:kernel (Otherwise identical to GENERIC, no optimisations.) The OS is <br />:completely unresponsive and all four cpu cores are running at 100%.<br />:<br />:I've included the debug information.<br />:<br />:Program received signal SIGINT, Interrupt.<br />:lwkt_acquire (td=0xc6a59e70) at /usr/src/sys/kern/lwkt_thread.c:1048<br />:1048 while (td</del>>td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK))<br />:(gdb) l<br />:1043 mygd = mycpu;<br />:1044 if (gd != mycpu) {<br />:1045 cpu_lfence();<br />:1046 <abbr title="(td->td_flags & TDF_RUNQ">KKASSERT</abbr> == 0);<br />:1047 crit_enter_gd(mygd);<br />:1048 while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK))<br />:1049 cpu_lfence();<br />:1050 td->td_gd = mygd;<br />:1051 TAILQ_INSERT_TAIL(&mygd->gd_tdallq, td, td_allq);<br />:1052 td->td_flags &= ~TDF_MIGRATING;<br />:(gdb) p td->td_flags<br />:$1 = 8390177<br />:(gdb) p td<br />:$2 = (thread_t) 0xc6a59e70<br />:(gdb) bt<br />:#0 lwkt_acquire (td=0xc6a59e70) at /usr/src/sys/kern/lwkt_thread.c:1048<br />:#1 0xc02c66af in bsd4_select_curproc (gd=0xff800000) at <br />:/usr/src/sys/kern/usched_bsd4.c:358<br />:#2 0xc02c6829 in bsd4_release_curproc (lp=0xea634c00) at <br />:/usr/src/sys/kern/usched_bsd4.c:322<br />:#3 0xc04b8239 in passive_release (td=0xdfe8aba0) at <br />:..<br />:<br />:Continuing execution does not appear to cause any problems.<br />:I can provide additional debugging info if required but I'm unsure of <br />:how to proceed with this myself.<br />:<br />:Regards<br />:<br />:Gary</p>
<pre><code>This is great info. One thing, what do you mean by 'KVM'? What<br /> is DragonFly running under, exactly?</code></pre>
<pre><code>I think once I understand the environment I may be able to figure out<br /> why the acquisition loop is blowing up.</code></pre>
<pre><code>-Matt<br /> Matthew Dillon <br /> &lt;<a class="email" href="mailto:dillon@backplane.com">dillon@backplane.com</a>&gt;</code></pre> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47482008-06-07T11:59:02Zfjwcash
<ul></ul><p><a class="external" href="http://kvm.qumranet.com">http://kvm.qumranet.com</a><br />Linux Kernel-based Virtual Machine.</p>
<p>Uses a modified QEmu and hardware virtualisation support in AMD and<br />Intel CPUs to effectively turn the Linux kernel into a hypervisor.<br />The VMs run as user processes on top of a (fairly) standard Linux<br />install.</p>
<p>Freddie</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47492008-06-07T19:03:01Zdragonfly1
<ul></ul><p>As Freddie says I'm running the QEmu KVM software on Ubuntu x64. <br />DragonFly is running 32bit with 3.5Gb RAM and 4 CPUs (AMD-Phenom).</p>
<p>I can also trigger SMP DragonFly to lock up during a buildworld on <br />VMWare Server 1.0.5 (Dual Core, 2Gb RAM) and VirtualBox. I haven't tried <br />the bare metal but could do so if anyone thinks it would be useful. I'm <br />running HEAD as of yesterday but 1.12.2 also locks up.</p>
<p>The kernel is GENERIC with "options SMP" and "options APIC_IO" enabled <br />and no modifications to /etc/make.conf. I have also tried enabling <br />"options ACPI_QUIRK_VMWARE".</p>
<p>For debugging I copied the kernel and source to my Ubuntu home directory <br />and ran.</p>
<p>kvm -smp 4 -m 3500 -hda "~/img/dfly.img" -net nic,model=e1000,vlan=1 <br />-net user,vlan=1 -redir tcp:2201::22 -s<br />gdb -s kernel.SMP -d ~/source/<br />(gdb) target remote localhost:1234<br />(gdb) c</p>
<p>Regards</p>
<p>Gary</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47502008-06-07T19:14:00Zsepherosa
<ul></ul><p>I have a box using Phenom 9550 and 2GB ram. Dfly HEAD runs directly<br />on the box (no virtual tech involved). Kernel is configured with SMP<br />and APIC_IO. I build world and kernel with -j 8 or -j 16 w/o any<br />issues.</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47512008-06-07T20:14:01Zcedric
<ul></ul><p>I've had issues like this with OpenBSD on VMWare a couple year ago.<br />Actually, VMs emulate "bare metal" systems very well, but they introduce<br />very unusual timings. So if you've a race that is "to short to be a<br />serious problem", than that race might well become a real problem on<br />VMs. To sum up, testing OSes on virtualized environment is a good way<br />to find hidden/rare bugs :)</p>
<p>Cedric</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47522008-06-07T22:38:00Zdragonfly1
<ul></ul><p>I've triggered another lock up and this time got a different trace. <br />Execution appears to be looping indefinitely inside LWKT code.</p>
<p>Debugging gives the output below. Again all four core are running at 100%.</p>
<p>Program received signal SIGINT, Interrupt.<br />lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67a7000, frame=0x0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:522<br />522 while (wi - (ri = ip->ip_rindex) > 0) {<br />(gdb) l<br />517 *<br />518 * Note: due to additional IPI operations that the callback function<br />519 * may make, it is possible for both rindex and windex to <br />advance and<br />520 * thus for rindex to advance passed our cached windex.<br />521 */<br />522 while (wi - (ri = ip->ip_rindex) > 0) {<br />523 ri &= MAXCPUFIFO_MASK;<br />524 copy_func = ip->ip_func[ri];<br />525 copy_arg1 = ip->ip_arg1[ri];<br />526 copy_arg2 = ip->ip_arg2[ri];<br />(gdb) p wi<br />$5 = 356278<br />(gdb) p ip->ip_rindex<br />$6 = 356278<br />(gdb) bt<br />#0 lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67a7000,<br /> frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:522<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c94ad in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p>
<p>Program received signal SIGINT, Interrupt.<br />lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67ac000, frame=0x0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:558<br />558 return(wi != ip->ip_windex);<br />(gdb) l<br />553 * ipiq. ip_npoll is left set as long as possible to reduce the<br />554 * number of IPIs queued by the originating cpu, but must be cleared<br />555 * <strong>BEFORE</strong> checking windex.<br />556 */<br />557 atomic_poll_release_int(&ip->ip_npoll);<br />558 return(wi != ip->ip_windex);<br />559 }<br />560 <br />561 static void<br />562 lwkt_sync_ipiq(void *arg)<br />(gdb) p wi<br />$7 = 357733<br />(gdb) p ip->ip_windex<br />$8 = 357733<br />(gdb) bt<br />#0 lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67ac000,<br /> frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:558<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c94ad in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p>
<p>Program received signal SIGINT, Interrupt.<br />lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67b1000, frame=0x0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:558<br />558 return(wi != ip->ip_windex);<br />(gdb) p wi<br />$9 = 372884<br />(gdb) p ip->ip_windex<br />$10 = 372884<br />(gdb) bt<br />#0 lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67b1000,<br /> frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:558<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c94ad in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p>
<p>Program received signal SIGINT, Interrupt.<br />0xc02c93a5 in lwkt_process_ipiq_core (sgd=<value optimized out>,<br /> ip=0xff8001e8, frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:522<br />522 while (wi - (ri = ip->ip_rindex) > 0) {<br />(gdb) p wi<br />$11 = 1343383299<br />(gdb) p ip->ip_rindex<br />$12 = 1343383298<br />(gdb) bt<br />#0 0xc02c93a5 in lwkt_process_ipiq_core (sgd=<value optimized out>,<br /> ip=0xff8001e8, frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:522<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c94df in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:458<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p>
<p>Program received signal SIGINT, Interrupt.<br />0xc02c9495 in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br />452 while (lwkt_process_ipiq_core(sgd, &ip[gd->gd_cpuid], NULL))<br />(gdb) l<br />447 for (n = 0; n < ncpus; ++n) {<br />448 if (n != gd->gd_cpuid) {<br />449 sgd = globaldata_find(n);<br />450 ip = sgd->gd_ipiq;<br />451 if (ip != NULL) {<br />452 while (lwkt_process_ipiq_core(sgd, &ip[gd->gd_cpuid], NULL))<br />453 ;<br />454 }<br />455 }<br />456 }<br />(gdb) bt<br />#0 0xc02c9495 in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000, Cannot access <br />memory at address 0x8<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p>
<p>Program received signal SIGINT, Interrupt.<br />0xc02c944c in lwkt_process_ipiq_core (sgd=<value optimized out>,<br /> ip=<value optimized out>, frame=0x0) at <br />/usr/src/sys/kern/lwkt_ipiq.c:559<br />559 }<br />(gdb) l<br />554 * number of IPIs queued by the originating cpu, but must be cleared<br />555 * <strong>BEFORE</strong> checking windex.<br />556 */<br />557 atomic_poll_release_int(&ip->ip_npoll);<br />558 return(wi != ip->ip_windex);<br />559 }<br />560 <br />561 static void<br />562 lwkt_sync_ipiq(void *arg)<br />563 {<br />(gdb) p wi<br />$13 = 357733<br />(gdb) p ip->ip_windex<br />Cannot access memory at address 0x8<br />(gdb) bt<br />#0 0xc02c944c in lwkt_process_ipiq_core (sgd=<value optimized out>,<br /> ip=<value optimized out>, frame=0x0) at <br />/usr/src/sys/kern/lwkt_ipiq.c:559<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c94ad in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p>
<p>Program received signal SIGINT, Interrupt.<br />lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67ac000, frame=0x0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:558<br />558 return(wi != ip->ip_windex);<br />(gdb) l<br />553 * ipiq. ip_npoll is left set as long as possible to reduce the<br />554 * number of IPIs queued by the originating cpu, but must be cleared<br />555 * <strong>BEFORE</strong> checking windex.<br />556 */<br />557 atomic_poll_release_int(&ip->ip_npoll);<br />558 return(wi != ip->ip_windex);<br />559 }<br />560 <br />561 static void<br />562 lwkt_sync_ipiq(void *arg)<br />(gdb) p wi<br />$24 = 357733<br />(gdb) p ip->ip_windex<br />$25 = 357733<br />(gdb) bt<br />#0 lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67ac000,<br /> frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:558<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: lib/libcr/sys/ cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/1">#1</a> 0xc02c94ad in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: K&R -> ANSI cleanup status (Closed)" href="https://bugs.dragonflybsd.org/issues/2">#2</a> 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br /> func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br /> at /usr/src/sys/kern/lwkt_ipiq.c:185<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: freebsds pipe-reverse test fails on dfly (Closed)" href="https://bugs.dragonflybsd.org/issues/3">#3</a> 0xc02c863c in lwkt_schedule (td=0xc0600170)<br /> at /usr/src/sys/sys/thread2.h:244<br /><a class="issue tracker-1 status-5 priority-5 priority-high3 closed" title="Bug: Rework of nrelease (Closed)" href="https://bugs.dragonflybsd.org/issues/4">#4</a> 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:551<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/dev cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/5">#5</a> 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br /> at /usr/src/sys/kern/usched_bsd4.c:271<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: sys/emulation cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/6">#6</a> 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br /> at /usr/src/sys/platform/pc32/i386/trap.c:349<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: /sys/boot cleanup (Closed)" href="https://bugs.dragonflybsd.org/issues/7">#7</a> 0xc04a3396 in Xint0x80_syscall ()<br /> at /usr/src/sys/platform/pc32/i386/exception.s:876<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: make upgrade broken (Closed)" href="https://bugs.dragonflybsd.org/issues/8">#8</a> 0xe5a02d40 in ?? ()<br />Backtrace stopped: previous frame inner to this frame (corrupt stack?)<br />(gdb) c<br />Continuing.</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47532008-06-07T23:13:01Zdillon
<ul></ul><p>:http://kvm.qumranet.com<br />:Linux Kernel-based Virtual Machine.<br />:<br />:Uses a modified QEmu and hardware virtualisation support in AMD and<br />:Intel CPUs to effectively turn the Linux kernel into a hypervisor.<br />:The VMs run as user processes on top of a (fairly) standard Linux<br />:install.<br />:<br />:Freddie</p>
<pre><code>Try putting a pause instruction in those loops.</code></pre>
<pre><code>I am hitting a lockup on one of my test boxes but I do not yet know<br /> if it is due to a bug in HAMMER (which is what I'm testing on that box),<br /> or due to recent SMP work, or whether it is related to your report.</code></pre>
<pre><code>Is there any chance you can get a backtrace on each one of the cpu's<br /> when it gets into that lockup?</code></pre>
<pre><code>-Matt</code></pre> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47872008-06-14T05:37:01Zmsylvan
<ul></ul><p>@Gary:</p>
<p>| I can also trigger SMP DragonFly to lock up during a buildworld on
| VMWare Server 1.0.5 (Dual Core, 2Gb RAM) and VirtualBox. I haven't tried
| the bare metal but could do so if anyone thinks it would be useful. I'm
| running HEAD as of yesterday but 1.12.2 also locks up.</p>
<p>Pardon me asking, but how did you get DragonFly booting on VirtualBox? I tried<br />it again (with 1.6.2 on Linux x64) after reading your post. With ACPI enabled it<br />freezes during the countdown; with ACPI disabled, just after countdown. Bug <a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: DFly doesnt work on Vmware and Virtualbox (Closed)" href="https://bugs.dragonflybsd.org/issues/995">#995</a><br />still appears to be active too.</p>
<p>Would love to get DragonFly working, and I don't feel like using KVM.</p>
<p>-- <br />Michel</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47882008-06-14T23:05:02Zdillon
<ul></ul><p>:..<br />:> :#0 lwkt_acquire (td=0xc6a59e70) at /usr/src/sys/kern/lwkt_thread.c:1048<br />:> :#1 0xc02c66af in bsd4_select_curproc (gd=0xff800000) at <br />:> :/usr/src/sys/kern/usched_bsd4.c:358<br />:> :#2 0xc02c6829 in bsd4_release_curproc (lp=0xea634c00) at <br />:> :/usr/src/sys/kern/usched_bsd4.c:322<br />:> :#3 0xc04b8239 in passive_release (td=0xdfe8aba0) at <br />:> :..<br />:Execution appears to be looping indefinitely inside LWKT code.<br />:<br />:Debugging gives the output below. Again all four core are running at 100%.<br />:<br />:<br />:Program received signal SIGINT, Interrupt.<br />:lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67a7000, frame=0x0)<br />: at /usr/src/sys/kern/lwkt_ipiq.c:522<br />:522 while (wi - (ri = ip->ip_rindex) > 0) {<br />:(gdb) l<br />:...<br />: at /usr/src/sys/sys/thread2.h:244<br />:#4 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br />: at /usr/src/sys/kern/usched_bsd4.c:551<br />:#5 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br />: at /usr/src/sys/kern/usched_bsd4.c:271<br />:#6 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br />: at /usr/src/sys/platform/pc32/i386/trap.c:349<br />:#7 0xc04a3396 in Xint0x80_syscall ()<br />:...<br />:(gdb) bt<br />:#0 lwkt_process_ipiq_core (sgd=<value optimized out>, ip=0xc67b1000,<br />: frame=0x0) at /usr/src/sys/kern/lwkt_ipiq.c:558<br />:#1 0xc02c94ad in lwkt_process_ipiq () at /usr/src/sys/kern/lwkt_ipiq.c:452<br />:#2 0xc02c9830 in lwkt_send_ipiq3 (target=0xff808000,<br />: func=0xc02c8519 <lwkt_schedule>, arg1=0xc0600170, arg2=0)<br />: at /usr/src/sys/kern/lwkt_ipiq.c:185<br />:#3 0xc02c863c in lwkt_schedule (td=0xc0600170)<br />: at /usr/src/sys/sys/thread2.h:244<br />:#4 0xc02c71e6 in bsd4_setrunqueue (lp=0xe5f8b400)<br />: at /usr/src/sys/kern/usched_bsd4.c:551<br />:#5 0xc02c72be in bsd4_acquire_curproc (lp=0xe5f8b400)<br />: at /usr/src/sys/kern/usched_bsd4.c:271<br />:#6 0xc04b9603 in syscall2 (frame=0xe5a02d40)<br />: at /usr/src/sys/platform/pc32/i386/trap.c:349<br />:#7 0xc04a3396 in Xint0x80_syscall ()<br />: at /usr/src/sys/platform/pc32/i386/exception.s:876<br />:#8 0xe5a02d40 in ?? ()
===================================================================<br />RCS file: /cvs/src/sys/kern/lwkt_thread.c,v<br />retrieving revision 1.115<br />diff <del>u -p -r1.115 lwkt_thread.c<br />--</del> lwkt_thread.c 2 Jun 2008 16:54:21 <del>0000 1.115<br />+<ins>+ lwkt_thread.c 14 Jun 2008 15:56:28 -0000<br /><code>@ -1045,8 +1045,12 </code>@ if (gd != mycpu) {<br /> cpu_lfence();<br /> KKASSERT((td</del>>td_flags & TDF_RUNQ) == 0);<br /> crit_enter_gd(mygd);<br />- while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK))<br /></ins> while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK)) {<br /><ins>#ifdef SMP<br /></ins> lwkt_process_ipiq();<br /><ins>#endif<br /> cpu_lfence();<br /></ins> }<br /> td->td_gd = mygd;<br /> TAILQ_INSERT_TAIL(&mygd->gd_tdallq, td, td_allq);<br /> td->td_flags &= ~TDF_MIGRATING;<br /><code>@ -1222,8 +1226,12 </code>@ {<br /> thread_t td = arg;<br /> globaldata_t gd = mycpu;</p>
<pre><code>I think I see what may be happening here, and I am starting to wonder<br /> if it is also the cause of the system lockups I am getting when testing<br /> HAMMER under extreme loads (with hundreds of user threads which are<br /> sometimes cpu-bound).</code></pre>
<pre><code>I think it may be deadlocking between lwkt_acquire() and lwkt_schedule().<br /> The thread trying to migrate between cpu's is getting stuck and the <br /> acquisition loop is not processing incoming IPIs while it is waiting for<br /> the thread to deschedule on the other cpu.</code></pre>
<pre><code>Please try this patch.</code></pre>
<pre><code>-Matt<br /> Matthew Dillon <br /> &lt;<a class="email" href="mailto:dillon@backplane.com">dillon@backplane.com</a>&gt;<br />Index: lwkt_thread.c</code></pre>
<p>- while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK))<br />+ while (td->td_flags & (TDF_RUNNING|TDF_PREEMPT_LOCK)) {<br /><ins>#ifdef SMP<br /></ins> lwkt_process_ipiq();<br /><ins>#endif<br /> cpu_lfence();<br /></ins> }<br /> td->td_gd = gd;<br /> cpu_sfence();<br /> td->td_flags &= ~TDF_MIGRATING;</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47892008-06-16T03:03:02Zdragonfly1
<ul></ul><p>Hello,</p>
<p>I've applied the patch and left the system running a "make -j8 <br />buildworld" in a loop, cleaning out /usr/obj between each run.</p>
<p>It has been running for over 10 hours with no lock-ups with each build <br />taking ~45mins. I have also tested -j32 while taxing the disk and CPU of <br />the Linux host OS.</p>
<p>All now appears to be Ok.</p>
<p>Thanks</p>
<p>Gary</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47902008-06-16T03:23:01Zdragonfly1
<ul></ul><p>Hello Michel,</p>
<p>I attached the .vmdk file I've used for ages to run DragonFly. However <br />I've tested it again and it hangs just after the boot menu, as does the <br />1.12.2 install CD.</p>
<p>I've done a fresh install of DFly since I last tried VirtualBox, the <br />previous install had a custom make.conf and kernel config to build for a <br />VIA C3 system and had been upgraded in-place for several versions.</p>
<p>VirtualBox like KVM is partly based on QEmu but I've looked on the <br />website and it doesn't seem to support debugging of guest OSes. Any <br />testing is going to be trial and error.</p>
<p>Sending a bug report a Sun is probably the quickest route to getting it <br />fixed.</p>
<p>Regards</p>
<p>Gary</p> DragonFlyBSD - Bug #1029: Dragonfly under KVMhttps://bugs.dragonflybsd.org/issues/1029?journal_id=47912008-06-16T09:04:03Zdillon
<ul></ul><p>:Hello,<br />:<br />:I've applied the patch and left the system running a "make -j8 <br />:buildworld" in a loop, cleaning out /usr/obj between each run.<br />:<br />:It has been running for over 10 hours with no lock-ups with each build <br />:taking ~45mins. I have also tested -j32 while taxing the disk and CPU of <br />:the Linux host OS.<br />:<br />:All now appears to be Ok.<br />:<br />:Thanks<br />:<br />:Gary</p>
<pre><code>The patch also appears to fix the system lockups I was experiencing<br /> testing HAMMER. My worst-case stress test ran over the weekend without<br /> any problems.</code></pre>
<pre><code>The two things in common here are that both KVM and HAMMER (under stress)<br /> could impose very long latencies on particular cpus, creating a<br /> deadlock between the IPI sending code and the thread cpu-migration<br /> code.</code></pre>
<pre><code>-Matt<br /> Matthew Dillon <br /> &lt;<a class="email" href="mailto:dillon@backplane.com">dillon@backplane.com</a>&gt;</code></pre>