DragonFlyBSD

Observed couple of times.

Here is some more thread related information.

Script started on Sun Nov 8 03:10:03 2009

1. kgdb ker nel.1 vmcore.1
   GNU gdb (GDB) 7.0
   This GDB was configured as "x86_64-dragonfly".

Reading symbols from /var/crash/kernel.1...done.

Unread portion of the kernel message buffer:
panic: assertion: parent != NULL in hammer_cursor_removed_node
mp_lock = 00000000; cpuid = 0
Trace beginning at frame 0xffffffffc53ed198
ðH‹]àL‹eèL‹mðL‹uøÉÃUH‰åATSI‰üè‚[() at ðH‹]àL‹eèL‹mðL‹uøÉÃUH‰åATSI‰üè‚[+0x1f7
ðH‹]àL‹eèL‹mðL‹uøÉÃUH‰åATSI‰üè‚[() at ðH‹]àL‹eèL‹mðL‹uøÉÃUH‰åATSI‰üè‚[+0x1f7
º() at º+0x38
H‰ÖI‰ËI‹È() at H‰ÖI‰ËI‹È+0x165
H‰ÖI‰ËI‹È() at H‰ÖI‰ËI‹È+0x14b
() at 0x203
·Àë Hз() at ·Àë Hз+0x3ff
‰þI‰õI‰üH¿H() at ‰þI‰õI‰üH¿H+0x3b6
ÉÃUH‰å‹‡¤() at ÉÃUH‰å‹‡¤+0x1fd
Ú;uë$;t H‹hH…ÀuóÇE€() at Ú;uë$;t H‹@hH…ÀuóÇE€+0x2d
‹>è¦ð() at ‹>è¦ð+0x3e
@L‰wH‹E‰‡() at L‰wH‹E‰‡+0x110
() at +0x486
f%() at f%+0x1f
Ʊ_€HÇÇóƒ€è®>=() at Ʊ+_€HÇÇóƒ€è®>=+0x372
¾”6() at ¾”6+0xb3
Debugger("panic")

CPU0 stopping CPUs: 0x00000002
stopped
panic: from debugger
mp_lock = 00000000; cpuid = 0
boot() called on cpu#0
Uptime: 1h44m18s

dumping to dev ad4s1b, blockno 4456960

Reading symbols from /boot/modules/acpi.ko...done.
Loaded symbols for /boot/modules/acpi.ko
Reading symbols from /boot/modules/null.ko...done.
Loaded symbols for /boot/modules/null.ko
get_mycpu () at ./machine/thread.h:73
73 __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(_mycpu__dummy));
(kgdb) thread
[Current thread is 27 (pid 94642/0, hammer)]
(kgdb) thread apply all bt

Thread 84 (kernel idle_0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff805bd1cd in cpu_idle () at
/usr/src/sys/platform/pc64/x86_64/machdep.c:932
#3 0x0000000000000000 in ?? ()

Thread 83 (pid 0/0, swapper):
#0 0x0000000000000000 in ?? ()
Cannot access memory at address 0x0

Thread 82 (kernel ithread emerg):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 81 (kernel ithread 0):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 80 (kernel softclock 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80383d90 in softclock_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_timeout.c:313
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 79 (kernel usched 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80372737 in sched_thread (dummy=<value optimized out>) at
/usr/src/sys/kern/usched_bsd4.c:1169
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 78 (kernel dsched 0):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 77 (kernel devfs_msg_core):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit--- at
/usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808d53a0,
flags=-2138221664)
at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff8050c985 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 devfs_msg_core (arg=<value optimized out>) at
/usr/src/sys/vfs/devfs/devfs_core.c:1038
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 76 (kernel netisr_cpu 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff80891dd0,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 75 (kernel disk_msg_core):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff8087b920,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff8038c8f6 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 disk_msg_core (arg=<value optimized out>) at /usr/src/sys/kern/subr_disk.c:363
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 74 (kernel ifnet 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff8088e090,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---
Thread 73 (kernel taskq_cpu 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9334bac8, flags=1024,
wmesg=0xffffffff80642947 "tqthr", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8037fcb9 in ssleep (ident=0xffffffff9334bac8,
spin=0xffffffff9334bb00, flags=0,
wmesg=0xffffffff80642947 "tqthr", timo=0) at /usr/src/sys/kern/kern_synch.c:707
#4 0xffffffff803960b1 in TQ_SLEEP (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:98
#5 taskqueue_thread_loop (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:374
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 72 (kernel crypto):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff808f1070, flags=1024,
wmesg=0xffffffff806823a0 "crypto_wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80380572 in lksleep (ident=0xffffffff808f1070,
lock=0xffffffff808f10a0, flags=0,
wmesg=0xffffffff806823a0 "crypto_wait", timo=0) at
/usr/src/sys/kern/kern_synch.c:722
#4 0xffffffff80591609 in crypto_proc () at /usr/src/sys/opencrypto/crypto.c:1367
#5 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 71 (kernel crypto returns):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff808f10d0, flags=1024,
wmesg=0xffffffff8068224e "crypto_ret_wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80380572 in lksleep (ident=0xffffffff808f10d0,
lock=0xffffffff808f1100, flags=0,
wmesg=0xffffffff8068224e "crypto_ret_wait", timo=0) at
/usr/src/sys/kern/kern_synch.c:722
#4 0xffffffff8059028d in crypto_ret_proc () at
/usr/src/sys/opencrypto/crypto.c:1430
#5 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 70 (kernel ithread 69):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---Backtrace stopped:
previous frame inner to this frame (corrupt stack?)

Thread 69 (kernel xpt_thrd):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff8083c4a0, flags=1024,
wmesg=0xffffffff805dea77 "ccb_scanq", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8016daeb in xpt_scanner_thread (dummy=<value optimized out>) at
/usr/src/sys/bus/cam/cam_xpt.c:1421
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 68 (kernel ithread 67):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 67 (kernel acpi_task):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff83651620,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff80ad8a6a in lwkt_waitport (arg=<value optimized out>) at
@/sys/msgport2.h:85
#5 acpi_task_thread (arg=<value optimized out>) at
/usr/src/sys/dev/acpica5/Osd/OsdSchedule.c:100
#6 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 66 (kernel ithread 9):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 65 (kernel ithread 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---Thread 64 (kernel ithread 12):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 63 (kernel ithread 64):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 62 (kernel ithread 4):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 61 (kernel ithread 7):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 60 (kernel usb0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9d052570, flags=-1661944816,
wmesg=0xffffffff8067584f "usbevt", timo=6000)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8055c407 in usb_event_thread (arg=<value optimized out>) at
/usr/src/sys/bus/usb/usb.c:459
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 59 (kernel usbtask-hc):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff808ec900, flags=0,
wmesg=0xffffffff80675848 "usbtsk", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8055c22c in usb_task_thread (arg=0xffffffff808ec900) at
/usr/src/sys/bus/usb/usb.c:488
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 58 (kernel usbtask-dr):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2 0xffffffff8037f5c0 in
tsleep (ident=0xffffffff808ec928, flags=0, wmesg=0xffffffff80675848 "usbtsk",
timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8055c22c in usb_task_thread (arg=0xffffffff808ec928) at
/usr/src/sys/bus/usb/usb.c:488
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 57 (kernel ithread 11):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 56 (kernel ithread 14):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 55 (kernel ithread 15):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80344cda in ithread_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_intr.c:867
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 54 (kernel random):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff83652070, flags=-1658381296,
wmesg=0xffffffff8063cf65 "rwait", timo=12)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80365c40 in rand_thread_loop (dummy=<value optimized out>) at
/usr/src/sys/kern/kern_nrandom.c:568
#4 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 53 (kernel udp_thread 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit--- at
/usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bdb10,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 52 (kernel tcp_thread 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bbf90,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff804375e3 in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 tcpmsg_service_loop (dummy=<value optimized out>) at
/usr/src/sys/netinet/tcp_subr.c:423
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 51 (kernel rtable_cpu 0):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff83652260,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803fcb3d in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 rtable_service_loop (dummy=<value optimized out>) at
/usr/src/sys/net/route.c:199
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 50 (kernel hammer-M):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9f5c3128, flags=0,
wmesg=0xffffffff8066d783 "hmrwwa", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8051a5dd in hammer_flusher_master_thread (arg=<value optimized out>)
at /usr/src/sys/vfs/hammer/hammer_flusher.c:252
#4 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 49 (kernel hammer-S0):
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#0 _crit_exit_quick () at
/usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc125f8, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 48 (kernel hammer-S1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc12a78, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 47 (kernel hammer-S2):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc12ef8, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 46 (kernel hammer-S3):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9cc13378, flags=0,
wmesg=0xffffffff8066d758 "hmrssw", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff80519ec2 in hammer_flusher_slave_thread (arg=<value optimized out>)
at /usr/src/sys/vfs/hammer/hammer_flusher.c:444
#4 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 45 (kernel pagedaemon):
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#0 _crit_exit_quick () at
/usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff808ec668, flags=-1621075152,
wmesg=0xffffffff80646682 "psleep", timo=500)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8054b2f8 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1530
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 44 (kernel vmdaemon):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff808ec6c4, flags=0,
wmesg=0xffffffff80646682 "psleep", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8054a4ae in vm_daemon () at /usr/src/sys/vm/vm_pageout.c:1630
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 43 (kernel bufdaemon):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff8087cbd0, flags=-1621050416,
wmesg=0xffffffff80646682 "psleep", timo=100)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8037fcb9 in ssleep (ident=0xffffffff8087cbd0,
spin=0xffffffff8088cbe4, flags=0,
wmesg=0xffffffff80646682 "psleep", timo=100) at
/usr/src/sys/kern/kern_synch.c:707
#4 0xffffffff803be63b in buf_daemon () at /usr/src/sys/kern/vfs_bio.c:2356
#5 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 42 (kernel bufdaemon_hw):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff8087cbd4, flags=-1621038128,
wmesg=0xffffffff80646682 "psleep", timo=100)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8037fcb9 in ssleep (ident=0xffffffff8087cbd4,
spin=0xffffffff8088cbe4, flags=0,
wmesg=0xffffffff80646682 "psleep", timo=100) at
/usr/src/sys/kern/kern_synch.c:707
#4 0xffffffff803be2e5 in buf_daemon_hw () at /usr/src/sys/kern/vfs_bio.c:2416
#5 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 41 (kernel syncer):
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#0 _crit_exit_quick () at
/usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff808747ec, flags=0,
wmesg=0xffffffff806475d4 "syncer", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff803d1b64 in sched_sync () at /usr/src/sys/kern/vfs_sync.c:278
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 40 (kernel vnlru):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff83653730, flags=-1621013504,
wmesg=0xffffffff80649244 "vlruwt", timo=100)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff803d13de in vnlru_proc () at /usr/src/sys/kern/vfs_mount.c:701
#4 0xffffffff8035b493 in suspend_kproc (td=0x0, timo=0) at
/usr/src/sys/kern/kern_kthread.c:158
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 39 (pid 672/0, cron):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff80879960, flags=-1074800184,
wmesg=0xffffffff8064059b "nanslp", timo=6000)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff8038292f in nanosleep1 (uap=0xffffffffbfefdb48) at
/usr/src/sys/kern/kern_time.c:315
#4 sys_nanosleep (uap=0xffffffffbfefdb48) at /usr/src/sys/kern/kern_time.c:349
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffbfefdbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00007ffffffffa70 in ?? ()
#8 0x00007ffffffffa60 in ?? ()
#9 0x0000000000000008 in ?? ()
#10 0x0000000000000004 in ?? ()
#11 0x0000000800a6d7e0 in ?? ()
#12 0x0000000800a6d858 in ?? ()
#13 0x00000000000000f0 in ?? ()
#14 0x000000000000003c in ?? ()
#15 0x000000000000003c in ?? ()
#16 0x0000000000000000 in ?? ()

Thread 38 (pid 724/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2 0xffffffff8037f595 in
tsleep (ident=0xffffffff8362bc60, flags=256, wmesg=0xffffffff80643b5b "ttyin",
timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362bc50, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362bc50, uio=0xffffffffc006bad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffc006b9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde730, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce69eb8,
uio=0xffffffffc006bad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc006bad8, flags=0,
res=0xffffffffc006bb48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc006bad8, flags=0, res=0xffffffffc006bb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc006bbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 37 (pid 726/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362c140, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362c130, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362c130, uio=0xffffffffc0073ad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffc00739a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde910, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce69f88,
uio=0xffffffffc0073ad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc0073ad8, flags=0,
res=0xffffffffc0073b48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc0073ad8, flags=0, res=0xffffffffc0073b48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc0073bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---
Thread 36 (pid 727/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362d4c0, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362d4b0, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362d4b0, uio=0xffffffffc0078ad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffc00789a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cddea00, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce695c8,
uio=0xffffffffc0078ad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc0078ad8, flags=0,
res=0xffffffffc0078b48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc0078ad8, flags=0, res=0xffffffffc0078b48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc0078bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 35 (pid 729/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362d9a0, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362d990, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362d990, uio=0xffffffffc00a2ad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffc00a29a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cddebe0, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffff9ce69b10,
uio=0xffffffffc00a2ad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc00a2ad8, flags=0,
res=0xffffffffc00a2b48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc00a2ad8, flags=0, res=0xffffffffc00a2b48)
at /usr/src/sys/kern/sys_generic.c:238
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#11 0xffffffff803968c1 in
sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc00a2bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 34 (pid 643/0, sshd):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8069038e "select", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80397901 in doselect (nd=<value optimized out>, in=0x8005801b8,
ou=0x0, ex=0x0, tv=0x0, res=0xffffffffbfa93b48)
at /usr/src/sys/kern/sys_generic.c:965
#4 0xffffffff80397c1f in sys_select (uap=0xffffffffbfa93b48) at
/usr/src/sys/kern/sys_generic.c:780
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa93bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x0000000000000006 in ?? ()
#8 0x00000008005801b8 in ?? ()
#9 0x0000000000000000 in ?? ()

Thread 33 (pid 828/0, su):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc3758, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=829, status=0xffffffffbfa9fa64,
options=2, rusage=0x0, res=0xffffffffbfa9fb48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffbfa9fb48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa9fbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x000000000000033d in ?? ()
#8 0x00007ffffffff74c in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 32 (pid 829/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff9cc17258, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0282a64,
options=2, rusage=0x0, res=0xffffffffc0282b48)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit--- at
/usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc0282b48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc0282bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x00007ffffffff764 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 31 (pid 94579/0, vi):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffc02db1d8, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffffc02db1c8, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffffc02db1c8, uio=0xffffffffc0302ad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a81d4 in ptsread (ap=0xffffffffc03029a8) at
/usr/src/sys/kern/tty_pty.c:396
#6 0xffffffff8034d05c in dev_dread (dev=0xffffffff9f5a9f10, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#7 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb27000,
uio=0xffffffffc0302ad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#8 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc0302ad8, flags=0,
res=0xffffffffc0302b48)
at /usr/src/sys/sys/file2.h:57
#9 kern_preadv (fd=0, auio=0xffffffffc0302ad8, flags=0, res=0xffffffffc0302b48)
at /usr/src/sys/kern/sys_generic.c:238
#10 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#11 0xffffffff805be85e in syscall2 (frame=0xffffffffc0302bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#12 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#13 0x0000000000000000 in ?? ()

Thread 30 (pid 94595/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff9cc188d8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc3b2da64,
options=2, rusage=0x0, res=0xffffffffc3b2db48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc3b2db48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc3b2dbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#7 0x00000000ffffffff in
?? ()
#8 0x00007ffffffff634 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 29 (pid 94608/0, mail):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffc4020f00, flags=1280,
wmesg=0xffffffff80642c0b "piperd", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80398d6b in pipe_read (fp=0xffffffffbfb27340,
uio=0xffffffffc63a9ad8, cred=<value optimized out>,
fflags=<value optimized out>) at /usr/src/sys/kern/sys_pipe.c:642
#4 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc63a9ad8, flags=0,
res=0xffffffffc63a9b48)
at /usr/src/sys/sys/file2.h:57
#5 kern_preadv (fd=0, auio=0xffffffffc63a9ad8, flags=0, res=0xffffffffc63a9b48)
at /usr/src/sys/kern/sys_generic.c:238
#6 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#7 0xffffffff805be85e in syscall2 (frame=0xffffffffc63a9bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#8 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#9 0x0000000000000000 in ?? ()

Thread 28 (pid 94639/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc4958, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc406da64,
options=2, rusage=0x0, res=0xffffffffc406db48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc406db48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc406dbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x00007ffffffff744 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 27 (pid 94642/0, hammer):
#0 _get_mycpu () at ./machine/thread.h:73
#1 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:657
#2 0xffffffff8036ba66 in boot (howto=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:378
#3 0xffffffff8036be0a in panic (fmt=0xffffffff805e9d54 "from debugger") at
/usr/src/sys/kern/kern_shutdown.c:813
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#4 0xffffffff80188325 in
db_panic (addr=<value optimized out>, have_addr=0, count=0, modif=0x0)
at /usr/src/sys/ddb/db_command.c:448
#5 0xffffffff801889db in db_command () at /usr/src/sys/ddb/db_command.c:344
#6 db_command_loop () at /usr/src/sys/ddb/db_command.c:470
#7 0xffffffff8018b751 in db_trap (type=<value optimized out>, code=<value
optimized out>) at /usr/src/sys/ddb/db_trap.c:71
#8 0xffffffff805b9ad9 in kdb_trap (type=3, code=0, regs=0xffffffffc53ed0c8)
at /usr/src/sys/platform/pc64/x86_64/db_interface.c:176
#9 0xffffffff805be1b5 in trap (frame=0xffffffffc53ed0c8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:686
#10 0xffffffff805b7f5e in calltrap () at
/usr/src/sys/platform/pc64/x86_64/exception.S:179
#11 0xffffffff808f6658 in main_console.9668 ()
#12 0x0000000000000780 in ?? ()
#13 0xffffffffbfd5c6b0 in ?? ()
#14 0x0000000000000000 in ?? ()

Thread 26 (kernel idle_1):
#0 0xffffffff80acd36f in acpi_cpu_idle () at
/usr/src/sys/dev/acpica5/acpi_cpu_cstate.c:865
#1 0xffffffff805bd20f in cpu_idle () at
/usr/src/sys/platform/pc64/x86_64/machdep.c:944
#2 0x0000000000000000 in ?? ()

Thread 25 (kernel softclock 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80383d90 in softclock_handler (arg=<value optimized out>) at
/usr/src/sys/kern/kern_timeout.c:313
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 24 (kernel usched 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff80372737 in sched_thread (dummy=<value optimized out>) at
/usr/src/sys/kern/usched_bsd4.c:1169
#3 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 23 (kernel dsched 1):
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()

Thread 22 (kernel netisr_cpu 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2 0xffffffff8037e2ee in
lwkt_sleep (wmesg=0xffffffff8063face "waitport", flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff80891f88,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 21 (kernel ifnet 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff8088e248,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 20 (kernel taskq_cpu 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f5c0 in tsleep (ident=0xffffffff9334bb18, flags=1024,
wmesg=0xffffffff80642947 "tqthr", timo=0)
at /usr/src/sys/kern/kern_synch.c:626
#3 0xffffffff8037fcb9 in ssleep (ident=0xffffffff9334bb18,
spin=0xffffffff9334bb50, flags=0,
wmesg=0xffffffff80642947 "tqthr", timo=0) at /usr/src/sys/kern/kern_synch.c:707
#4 0xffffffff803960b1 in TQ_SLEEP (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:98
#5 taskqueue_thread_loop (arg=<value optimized out>) at
/usr/src/sys/kern/subr_taskqueue.c:374
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 19 (kernel udp_thread 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bdcc8,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803faeb4 in lwkt_waitport (arg=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 netmsg_service_loop (arg=<value optimized out>) at /usr/src/sys/net/netisr.c:332
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---Backtrace stopped:
previous frame inner to this frame (corrupt stack?)

Thread 18 (kernel tcp_thread 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff808bc148,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff804375e3 in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 tcpmsg_service_loop (dummy=<value optimized out>) at
/usr/src/sys/netinet/tcp_subr.c:423
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 17 (kernel rtable_cpu 1):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037e2ee in lwkt_sleep (wmesg=0xffffffff8063face "waitport",
flags=<value optimized out>)
at /usr/src/sys/kern/kern_synch.c:790
#3 0xffffffff8037aea4 in lwkt_thread_waitport (port=0xffffffff83652420,
flags=0) at /usr/src/sys/kern/lwkt_msgport.c:717
#4 0xffffffff803fcb3d in lwkt_waitport (dummy=<value optimized out>) at
/usr/src/sys/sys/msgport2.h:85
#5 rtable_service_loop (dummy=<value optimized out>) at
/usr/src/sys/net/route.c:199
#6 0xffffffff803766a9 in lwkt_deschedule_self (td=0x0) at
/usr/src/sys/kern/lwkt_thread.c:271
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 16 (pid 212/0, dhclient):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8064e511 "poll", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff8039801c in sys_poll (uap=0xffffffffbfef8b48) at
/usr/src/sys/kern/sys_generic.c:1096
#4 0xffffffff805be85e in syscall2 (frame=0xffffffffbfef8bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#5 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#6 0x00007ffffffffa90 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00000000ffffffff in ?? ()
#9 0x0000000000000000 in ?? ()

Thread 15 (pid 226/0, dhclient):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#2 0xffffffff8037f595 in
tsleep (ident=0xffffffff8087bc30, flags=-1079445272, wmesg=0xffffffff8064e511
"poll", timo=8640000)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff8039801c in sys_poll (uap=0xffffffffbfa8fb48) at
/usr/src/sys/kern/sys_generic.c:1096
#4 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa8fbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#5 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#6 0x00007ffffffffa90 in ?? ()
#7 0x0000000000000002 in ?? ()
#8 0x000000007ffffd78 in ?? ()
#9 0x00000008005a00c0 in ?? ()
#10 0x0000000000000005 in ?? ()
#11 0x0000000100000016 in ?? ()
#12 0x00000000000000d1 in ?? ()
#13 0x00000008006c00c0 in ?? ()
#14 0x000000007ffffd78 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x00000008006d0000 in ?? ()
#17 0x0000000000000002 in ?? ()
#18 0x000000000020c49b in ?? ()
#19 0x00000000005c6a78 in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 14 (pid 722/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362b780, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362b770, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362b770, uio=0xffffffffbfa17ad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffbfa179a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde550, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26b88,
uio=0xffffffffbfa17ad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffbfa17ad8, flags=0,
res=0xffffffffbfa17b48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffbfa17ad8, flags=0, res=0xffffffffbfa17b48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa17bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#13 0xffffffff805b8193 in
Xfast_syscall () at /usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 13 (pid 725/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362bfa0, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362bf90, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362bf90, uio=0xffffffffc006fad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffc006f9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde820, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26d28,
uio=0xffffffffc006fad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc006fad8, flags=0,
res=0xffffffffc006fb48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc006fad8, flags=0, res=0xffffffffc006fb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc006fbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 12 (pid 723/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362be00, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362bdf0, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362bdf0, uio=0xffffffffbfa9bad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffbfa9b9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cdde640, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26df8,
uio=0xffffffffbfa9bad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffbfa9bad8, flags=0,
res=0xffffffffbfa9bb48)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit--- at
/usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffbfa9bad8, flags=0, res=0xffffffffbfa9bb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa9bbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 11 (pid 728/0, getty):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8362d800, flags=256,
wmesg=0xffffffff80643b5b "ttyin", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803a0e22 in ttysleep (tp=0xffffffff8362d7f0, chan=0x0, slpflags=0,
wmesg=0x0, timo=0)
at /usr/src/sys/kern/tty.c:2583
#4 0xffffffff803a3f7b in ttread (tp=0xffffffff8362d7f0, uio=0xffffffffc007cad8,
flag=<value optimized out>)
at /usr/src/sys/kern/tty.c:1737
#5 0xffffffff803a0cff in ttyread (ap=<value optimized out>) at
/usr/src/sys/kern/tty.c:2683
#6 0xffffffff805ac1c8 in scread (ap=0xffffffffc007c9a8) at
/usr/src/sys/dev/misc/syscons/syscons.c:574
#7 0xffffffff8034d05c in dev_dread (dev=0xffffffff9cddeaf0, uio=0x0, ioflag=0)
at /usr/src/sys/kern/kern_device.c:140
#8 0xffffffff8050f17f in devfs_specf_read (fp=0xffffffffbfb26f30,
uio=0xffffffffc007cad8, cred=<value optimized out>,
flags=0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1051
#9 0xffffffff8039671e in dofileread (fd=0, auio=0xffffffffc007cad8, flags=0,
res=0xffffffffc007cb48)
at /usr/src/sys/sys/file2.h:57
#10 kern_preadv (fd=0, auio=0xffffffffc007cad8, flags=0, res=0xffffffffc007cb48)
at /usr/src/sys/kern/sys_generic.c:238
#11 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#12 0xffffffff805be85e in syscall2 (frame=0xffffffffc007cbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#13 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#14 0x0000000000000000 in ?? ()

Thread 10 (pid 823/0, sshd):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffbfd13ed0, flags=256,
wmesg=0xffffffff80644f6c "sbwait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff803b3445 in ssb_wait (ssb=0x0) at /usr/src/sys/kern/uipc_socket2.c:85
#4 0xffffffff803b0e9b in soreceive (so=0xffffffffbfd13e50, psa=0x0,
uio=0xffffffffbfa1aad8, sio=0x0, controlp=0x0,
flagsp=0xffffffffbfa1aa0c) at /usr/src/sys/kern/uipc_socket.c:910
#5 0xffffffff8039d608 in so_pru_soreceive (fp=0xffffffffbfb27138,
uio=0xffffffffbfa1aad8, cred=<value optimized out>,
fflags=0) at /usr/src/sys/sys/socketops.h:72
#6 soo_read (fp=0xffffffffbfb27138, uio=0xffffffffbfa1aad8, cred=<value
optimized out>, fflags=0)
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit--- at
/usr/src/sys/kern/sys_socket.c:90
#7 0xffffffff8039671e in dofileread (fd=5, auio=0xffffffffbfa1aad8, flags=0,
res=0xffffffffbfa1ab48)
at /usr/src/sys/sys/file2.h:57
#8 kern_preadv (fd=5, auio=0xffffffffbfa1aad8, flags=0, res=0xffffffffbfa1ab48)
at /usr/src/sys/kern/sys_generic.c:238
#9 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#10 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa1abf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#11 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#12 0x0000000000000005 in ?? ()
#13 0x00007ffffffff210 in ?? ()
#14 0x0000000000000004 in ?? ()
#15 0x0000000000000004 in ?? ()
#16 0x0000000000001000 in ?? ()
#17 0x00007fffffffe93f in ?? ()
#18 0x0000000000000003 in ?? ()
#19 0x0000000000000000 in ?? ()

Thread 9 (pid 826/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff9cc16dd8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0233a64,
options=2, rusage=0x0, res=0xffffffffc0233b48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc0233b48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc0233bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x00007ffffffff574 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (pid 1/0, init):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff9cc11858, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0x0, options=0, rusage=0x0,
res=0xffffffff9cc42b48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffff9cc42b48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffff9cc42bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#6 0xffffffff805b8193 in
Xfast_syscall () at /usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x0000000000000000 in ?? ()

Thread 7 (pid 825/0, sshd):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8069038e "select", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80397901 in doselect (nd=<value optimized out>, in=0x800580210,
ou=0x800580208, ex=0x0, tv=0x0,
res=0xffffffffc023cb48) at /usr/src/sys/kern/sys_generic.c:965
#4 0xffffffff80397c1f in sys_select (uap=0xffffffffc023cb48) at
/usr/src/sys/kern/sys_generic.c:780
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc023cbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x000000000000000a in ?? ()
#8 0x0000000800580210 in ?? ()
#9 0x0000000800580208 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 6 (pid 94593/0, cron):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffbfbe0f00, flags=1280,
wmesg=0xffffffff80642c0b "piperd", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80398d6b in pipe_read (fp=0xffffffff9ce6a128,
uio=0xffffffffc02adad8, cred=<value optimized out>,
fflags=<value optimized out>) at /usr/src/sys/kern/sys_pipe.c:642
#4 0xffffffff8039671e in dofileread (fd=5, auio=0xffffffffc02adad8, flags=0,
res=0xffffffffc02adb48)
at /usr/src/sys/sys/file2.h:57
#5 kern_preadv (fd=5, auio=0xffffffffc02adad8, flags=0, res=0xffffffffc02adb48)
at /usr/src/sys/kern/sys_generic.c:238
#6 0xffffffff803968c1 in sys_read (uap=0x0) at /usr/src/sys/kern/sys_generic.c:114
#7 0xffffffff805be85e in syscall2 (frame=0xffffffffc02adbf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#8 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#9 0x0000000000000005 in ?? ()
#10 0x0000000800550000 in ?? ()
#11 0x0000000000004000 in ?? ()
#12 0x0000000800550000 in ?? ()
#13 0x00000000000000c0 in ?? ()
#14 0x00000000ffffffff in ?? ()
#15 0x0000000000000003 in ?? ()
#16 0x0000000800a80220 in ?? ()
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#17 0x00000008005900e8 in
?? ()
#18 0x0000000000000000 in ?? ()

Thread 5 (pid 94597/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff9cc17fd8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0297a64,
options=2, rusage=0x0, res=0xffffffffc0297b48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc0297b48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc0297bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x00007ffffffff654 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 4 (pid 94604/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc4058, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc0293a64,
options=2, rusage=0x0, res=0xffffffffc0293b48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc0293b48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc0293bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x00007ffffffff1e4 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 3 (pid 94603/0, sh):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffffbfcc3bd8, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=-1, status=0xffffffffc02bea64,
options=2, rusage=0x0, res=0xffffffffc02beb48)
at /usr/src/sys/kern/kern_exit.c:869
---Type <return> to continue, or q <return> to quit---
---Type <return> to continue, or q <return> to quit---#4 0xffffffff803578be in
sys_wait4 (uap=0xffffffffc02beb48) at /usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc02bebf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000ffffffff in ?? ()
#8 0x00007fffffffef34 in ?? ()
#9 0x0000000000000002 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 2 (pid 349/0, syslogd):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff8087bc30, flags=256,
wmesg=0xffffffff8069038e "select", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80397901 in doselect (nd=<value optimized out>, in=0x8005900c8,
ou=0x0, ex=0x0, tv=0x0, res=0xffffffffbfa97b48)
at /usr/src/sys/kern/sys_generic.c:965
#4 0xffffffff80397c1f in sys_select (uap=0xffffffffbfa97b48) at
/usr/src/sys/kern/sys_generic.c:780
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffbfa97bf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x0000000000000007 in ?? ()
#8 0x00000008005900c8 in ?? ()
#9 0x0000000000000000 in ?? ()

Thread 1 (pid 94640/0, hammer):
#0 _crit_exit_quick () at /usr/src/sys/sys/thread2.h:177
#1 lwkt_switch () at /usr/src/sys/kern/lwkt_thread.c:790
#2 0xffffffff8037f595 in tsleep (ident=0xffffffff9cc18d58, flags=256,
wmesg=0xffffffff805f845d "wait", timo=0)
at /usr/src/sys/kern/kern_synch.c:615
#3 0xffffffff80357813 in kern_wait (pid=94642, status=0xffffffffc030ea64,
options=0, rusage=0x0, res=0xffffffffc030eb48)
at /usr/src/sys/kern/kern_exit.c:869
#4 0xffffffff803578be in sys_wait4 (uap=0xffffffffc030eb48) at
/usr/src/sys/kern/kern_exit.c:671
#5 0xffffffff805be85e in syscall2 (frame=0xffffffffc030ebf8) at
/usr/src/sys/platform/pc64/x86_64/trap.c:1168
#6 0xffffffff805b8193 in Xfast_syscall () at
/usr/src/sys/platform/pc64/x86_64/exception.S:304
#7 0x00000000000171b2 in ?? ()
#8 0x00007fffffffe684 in ?? ()
#9 0x0000000000000000 in ?? ()
(kgdb)
(kgdb) q

1. exit
   Script done on Sun Nov 8 03:10:45 2009

Hi,

On Sat, Nov 07, 2009 at 05:23:12PM +0530, Saifi Khan wrote:

After about 8 hrs of uptime on the X86_64 build, the system had
an assertion failure.

assertion: parent != NULL in hammer_cursor_removed_node

Although the backtrace is garbled, this is most likely the panic
introduced by the committed fix for http://bugs.dragonflybsd.org/issue1577,
especially since it happened while running `hammer cleanup' from
a periodic(8) job.
For some reason the search facility on our bug tracker can't find it
with a keyword like `hammer_cursor_removed_node' even with full text search.

The commit f3a4893b has moved the call to hammer_cursor_removed_node()
in btree_remove() to after the recursive call to itself, but a call to
btree_remove() moves the cursor up after deleting the sub tree, so it's
possible that it reaches the root of the tree during the recursion.
When that happens, cursor->parent is set to NULL and ondisk->parent of
the current node is also 0.

I have a bandaid for this problem (it's being tested):

HAMMER VFS - don't call hammer_cursor_removed_node when the recursion reached the root of the filesystem

---
sys/vfs/hammer/hammer_btree.c | 12 +++++++---
1 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/sys/vfs/hammer/hammer_btree.c b/sys/vfs/hammer/hammer_btree.c
index 6ee1e1a..551fb5f 100644
--- a/sys/vfs/hammer/hammer_btree.c
++ b/sys/vfs/hammer/hammer_btree.c
@ -2226,9 +2226,15 @ btree_remove(hammer_cursor_t cursor)
hammer_cursor_deleted_element(cursor->node, 0);
error = btree_remove(cursor);
if (error == 0) {
- hammer_cursor_removed_node(
- node, cursor->parent,
- cursor->parent_index);
KKASSERT(
+ cursor->parent != NULL ||
+ cursor->node->ondisk->parent == 0
+ );
+ if (cursor->parent != NULL) {
+ hammer_cursor_removed_node(
+ node, cursor->parent,
+ cursor->parent_index);
+ }
hammer_modify_node_all(cursor->trans, node);
ondisk = node->ondisk;
ondisk->type = HAMMER_BTREE_TYPE_DELETED;
--
1.6.4

:Although the backtrace is garbled, this is most likely the panic
:introduced by the committed fix for http://bugs.dragonflybsd.org/issue1577,
:especially since it happened while running `hammer cleanup' from
:a periodic(8) job.
:For some reason the search facility on our bug tracker can't find it
:with a keyword like `hammer_cursor_removed_node' even with full text search.
:
:The commit f3a4893b has moved the call to hammer_cursor_removed_node()
:in btree_remove() to after the recursive call to itself, but a call to
:btree_remove() moves the cursor up after deleting the sub tree, so it's
:possible that it reaches the root of the tree during the recursion.
:When that happens, cursor->parent is set to NULL and ondisk->parent of
:the current node is also 0.
:
:I have a bandaid for this problem (it's being tested):
:
:HAMMER VFS - don't call hammer_cursor_removed_node when the recursion reached the root of the filesystem

Ok, if the recursion is successful and winds up removing an internal node just below the root node, then when it cursors up again you are correct: It will be at the root node and cursor->parent will be NULL.

For this case what we really want it to do is call hammer_cursor_removed_node() with the original (now deleted) node as the first argument and the root node as the second argument.

We absolutely MUST call hammer_cursor_removed_node() for any node which is removed, so we can't conditionalize it as in your patch.

I think I may be calling hammer_cursor_removed_node() with the wrong node. I think I have to use cursor->node/cursor->index instead of cursor->parent/cursor->parent_index. The recursion issues a
cursor_up in the no-error case after all.

Lets see. hammer_btree_remove() recurses and hits a stop where (parent->ondisk->count > 1). So it is able to remove the node anddoes not have to remove the parent. The last thing it does is
call hammer_cursor_up():

                /*
                 * cursor->node is invalid, cursor up to make the cursor
                 * valid again.
                 */
                error = hammer_cursor_up(cursor);

So now cursor->node points at the parent which is still intact (and could end up being the root node). So on the return from the recursion we should be using cursor->node, not cursor->parent.
I'm thinking something like this (see below). UNTESTED! I will start some tests up today. It could panic instantly :-)

-Matt
Matthew Dillon
<dillon@backplane.com>

diff --git a/sys/vfs/hammer/hammer_btree.c b/sys/vfs/hammer/hammer_btree.c
index 6ee1e1a..050d4a2 100644
--- a/sys/vfs/hammer/hammer_btree.c
+++ b/sys/vfs/hammer/hammer_btree.c
@@ -2226,9 +2226,10 @@ btree_remove(hammer_cursor_t cursor)
             hammer_cursor_deleted_element(cursor->node, 0);
             error = btree_remove(cursor);
             if (error == 0) {
+                KKASSERT(node != cursor->node);
                 hammer_cursor_removed_node(
-                    node, cursor->parent,
-                    cursor->parent_index);
+                    node, cursor->node,
+                    cursor->index);
                 hammer_modify_node_all(cursor->trans, node);
                 ondisk = node->ondisk;
                 ondisk->type = HAMMER_BTREE_TYPE_DELETED;

: I'm thinking something like this (see below). UNTESTED! I will
: start some tests up today. It could panic instantly :-)

I will have to run some additional tests.  Two issues came up but
    I'm not sure they are related to the patch or not.

• I did a rm -rf of around 7 million files across a few tens of
  thousands of directories. The rm -rf missed two files. The symptom
  being it couldn't remove the directory it thought was empty but
  which was not empty. ls -lR showed the directory as still
  containing files.

  A second rm -rf got rid of the files. I don't know if this is
  patch-related or not, it could be a pre-existing bug that I just
  found. Directory scans are based on 64-bit hash keys and should
  not lose their place regardless of whatever else is going on in
  the directory (like file deletions and creations). It should not
  have missed the files.

• I did a bulkbuild and had some odd build issues, but I will have
  to do a second bulkbuild to try to determine if it is related to
  HAMMER or just something going on with the bulkbuild.

  It only built around 3000 or so packages and got stuck somewhere
  along the way, but the screen output showed it had run through
  all 8000+ packages.

  -Matt

Thank you Matt and Tomokazu-san for your reply.

Yes, the backtrace is garbled when it is dumped to the screen.
Would it help if i uploaded the core files somewhere ?

i also took two photographs when the panic happened. Here are the links
http://i818.photobucket.com/albums/zz107/saifikhan/dragonfly/DSC00101.jpg
http://i818.photobucket.com/albums/zz107/saifikhan/dragonfly/DSC00102.jpg

i would like to help with testing the patches that you propose.
Is there a link from where i could download the test cases ?

thanks
Saifi.

Hi all,

This is the patch I applied:

diff --git a/sys/vfs/hammer/hammer_btree.c b/sys/vfs/hammer/hammer_btree.c
index 6ee1e1a..7b65610 100644
--- a/sys/vfs/hammer/hammer_btree.c
++ b/sys/vfs/hammer/hammer_btree.c
@ -2226,9 +2226,10 @ btree_remove(hammer_cursor_t cursor)
                       hammer_cursor_deleted_element(cursor->node, 0);
                       error = btree_remove(cursor);
                       if (error == 0) {
                              KKASSERT;
                               hammer_cursor_removed_node(
-                                       node, cursor->parent,
-                                       cursor->parent_index);
+                                      node, cursor->parent,
+                                      cursor->parent_index);
                               hammer_modify_node_all(cursor->trans, node);
                               ondisk = node->ondisk;
                               ondisk->type = HAMMER_BTREE_TYPE_DELETED;

See below for more details. Kernel core dump on
http://leaf.dragonflybsd.org/~tuxillo/bugs/1597/

Unread portion of the kernel message buffer:
panic: assertion: parent != NULL in hammer_cursor_removed_node
Trace beginning at frame 0xccf6a68c
panic(ccf6a6b0,0,0,ccf9d238,ccf6a6c8) at panic+0x8c
panic(c057dc43,c059c7ab,c05693b3,1f,0) at panic+0x8c
hammer_cursor_removed_node(ccf9d238,0,0,ccf6a818,ccf9df58) at
hammer_cursor_removed_node+0x27
btree_remove(ccf9d238,0,ccf6a818,c1b07000,0) at btree_remove+0x14d
btree_remove(0,0,c1b07040,c16bf898,ccf6a7b0) at btree_remove+0x115
hammer_btree_delete(ccf6a818) at hammer_btree_delete+0x1ab
hammer_delete_at_cursor(ccf6a818,2,0,0,4afbe918) at
hammer_delete_at_cursor+0x327
hammer_ioc_prune(ccf6aa74,c9c1d450,c16bf800,4,ccf6a956) at
hammer_ioc_prune+0x4c1
hammer_ioctl(c9c1d450,c0fc6801,c16bf800,1,c15876f8) at hammer_ioctl+0x1b8
hammer_vop_ioctl(ccf6aad0,c0649260,c16d1eb0,ccf6ab00,c032a999) at
hammer_vop_ioctl+0x2f
vop_ioctl(c16d1eb0,c16f5128,c0fc6801,c16bf800,1) at vop_ioctl+0x3e
vn_ioctl(c8eadcf8,c0fc6801,c16bf800,c15876f8,ccf6acf0) at vn_ioctl+0xe0
mapped_ioctl(3,c0fc6801,280b0188,0,ccf6acf0) at mapped_ioctl+0x3e7
sys_ioctl(ccf6acf0,6,9574,0,c9b8e898) at sys_ioctl+0x17
syscall2(ccf6ad40) at syscall2+0x1ef
Xint0x80_syscall() at Xint0x80_syscall+0x36
Debugger("panic")
Uptime: 5m2s

dumping to dev ad0s1b, blockno 262144
dump 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241
240 239 238 237 236 235 234 233 232 231 230 22
9 228 227 226 225 224 223 222 221
220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204
203 202 201 2
00 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184
183 182 181 180 179 178 177 176 175 174 173 172
171 170 169 168 167 166 165 164
163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147
146 145 144 143
142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127
126 125 124 123 122 121 120 119 118 117 116 115 11
4 113 112 111 110 109 108 107
106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86
85 84 83 82 81
80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60
59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 4
2 41 40 39 38 37 36 35 34
33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11
10 9 8 7 6 5 4 3 2 1

And the backtrace from KGDB:

(kgdb) bt
#0 _get_mycpu () at ./machine/thread.h:83
#1 dumpsys () at /build/home/tuxillo/dfbsd/sys/kern/kern_shutdown.c:657
#2 0xc031e0a2 in boot (howto=260) at
/build/home/tuxillo/dfbsd/sys/kern/kern_shutdown.c:378
#3 0xc031e1c3 in panic (fmt=0xc057dc43 "assertion: %s in %s")
at /build/home/tuxillo/dfbsd/sys/kern/kern_shutdown.c:813
#4 0xc048aa26 in hammer_cursor_removed_node (node=0xccf9d238,
parent=0x0, index=0)
at /build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_cursor.c:722
#5 0xc04885bf in btree_remove (cursor=0xccf6a818) at
/build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_btree.c:2230
#6 0xc0488587 in btree_remove (cursor=0xccf6a818) at
/build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_btree.c:2227
#7 0xc048899b in hammer_btree_delete (cursor=0xccf6a818)
at /build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_btree.c:896
#8 0xc049809e in hammer_delete_at_cursor (cursor=0xccf6a818,
delete_flags=<value optimized out>, delete_tid=0,
delete_ts=1258023192, track=0, stat_bytes=0xc16bf8b0)
at /build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_object.c:2488
#9 0xc049d209 in hammer_ioc_prune (trans=0xccf6aa74, ip=0xc9c1d450,
prune=0xc16bf800)
at /build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_prune.c:184
#10 0xc0493894 in hammer_ioctl (ip=0xc9c1d450, com=3237767169,
data=0xc16bf800 "\001", fflag=1, cred=0xc15876f8)
at /build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_ioctl.c:76
#11 0xc04a5181 in hammer_vop_ioctl (ap=0xccf6aad0) at
/build/home/tuxillo/dfbsd/sys/vfs/hammer/hammer_vnops.c:2304
#12 0xc0375bd1 in vop_ioctl (ops=0xc16d1eb0, vp=0xc16f5128,
command=3237767169, data=0xc16bf800 "\001", fflag=1,
cred=0xc15876f8, msg=0xccf6acf0) at
/build/home/tuxillo/dfbsd/sys/kern/vfs_vopops.c:389
#13 0xc03756f5 in vn_ioctl (fp=0xc8eadcf8, com=3237767169,
data=0xc16bf800 "\001", ucred=0xc15876f8,
msg=0xccf6acf0) at /build/home/tuxillo/dfbsd/sys/kern/vfs_vnops.c:938
#14 0xc033e8af in fo_ioctl (fd=3, com=3237767169, uspc_data=0x280b0188
<Address 0x280b0188 out of bounds>,
map=0x0, msg=0xccf6acf0) at /build/home/tuxillo/dfbsd/sys/sys/file2.h:88
#15 mapped_ioctl (fd=3, com=3237767169, uspc_data=0x280b0188 <Address
0x280b0188 out of bounds>, map=0x0,
msg=0xccf6acf0) at /build/home/tuxillo/dfbsd/sys/kern/sys_generic.c:697
#16 0xc033e938 in sys_ioctl (uap=0xccf6acf0) at
/build/home/tuxillo/dfbsd/sys/kern/sys_generic.c:521
#17 0xc0530dd3 in syscall2 (frame=0xccf6ad40) at
/build/home/tuxillo/dfbsd/sys/platform/pc32/i386/trap.c:1339
#18 0xc05212a6 in Xint0x80_syscall () at
/build/home/tuxillo/dfbsd/sys/platform/pc32/i386/exception.s:876
#19 0xccf6ad40 in ?? ()
#20 0x0000007b in ?? ()
#21 0x0000002f in ?? ()
#22 0x0000002f in ?? ()
#23 0x0000002f in ?? ()
#24 0x280b0080 in ?? ()

Please let me know if you need me to test anything else.

Cheers,
Antonio Huete

On Thu, Nov 12, 2009 at 12:47:21PM +0100, Antonio Huete Jimenez wrote:

This is the patch I applied:

diff --git a/sys/vfs/hammer/hammer_btree.c b/sys/vfs/hammer/hammer_btree.c
index 6ee1e1a..7b65610 100644
--- a/sys/vfs/hammer/hammer_btree.c
++ b/sys/vfs/hammer/hammer_btree.c
@ -2226,9 +2226,10 @ btree_remove(hammer_cursor_t cursor)
hammer_cursor_deleted_element(cursor->node, 0);
error = btree_remove(cursor);
if (error == 0) {
KKASSERT;
hammer_cursor_removed_node(
- node, cursor->parent,
- cursor->parent_index);
+ node, cursor->parent,
+ cursor->parent_index);
hammer_modify_node_all(cursor->trans, node);
ondisk = node->ondisk;
ondisk->type = HAMMER_BTREE_TYPE_DELETED;

Maybe you applied the patch by hand because the mailer mangled the spaces?
You should have changed this part

hammer_cursor_removed_node(
        node, cursor->parent,
        cursor->parent_index);

to this

hammer_cursor_removed_node(
        node, cursor->node,
        cursor->index);

I haven't seen a big problem after applying Matt's patch.

Cheers.

Hello,

I've hitted the same assertion, trying to 'hammer version-upgrade / 3'. Now, I
can reliably trigger this panic when doing a hammer cleanup.

Let me know if you need more info. I can probably arrange to upload the dump
somewhere in the following days if you need it, just tell me.

uname a
DragonFly libellule.jos.net 2.5.1-DEVELOPMENT DragonFly v2.5.1.352.g37653
DEVELOPMENT #1: Sun Dec 6 20:08:01 CET 2009
root@libellule.jos.net:/usr/obj/usr/src/sys/GENERIC i386

libellule# hammer cleanup
cleanup / - handle PFS #0 using /var/hammer/root
snapshots - skip
prune - skip
rebalance - skip
reblock - skip
recopy - skip
cleanup /var - handle PFS #1 using /var/hammer/var
snapshots - skip
prune - skip
rebalance - skip
reblock - skip
recopy - skip
cleanup /tmp - handle PFS #2 using /var/hammer/tmp
snapshots - disabled
prune - skip
rebalance - skip
reblock - skip
recopy - skip
cleanup /usr - handle PFS #3 using /var/hammer/usr
snapshots - skip
prune - skip
rebalance - skip
reblock - skip
recopy - skip
cleanup /home - handle PFS #4 using /var/hammer/home
snapshots - run
prune - run
rebalance - run..
reblock - run....
recopy - run....
cleanup /usr/obj -(migrating) HAMMER UPGRADE: Moving snapshots
Moving snapshots from /usr/obj/snapshots to /var/hammer/usr/obj
handle PFS #5 using /var/hammer/usr/obj
snapshots - disabled
prune - run
<PANIC!>

libellule# kgdb kernel.1 vmcore.1
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-dragonfly".
For bug reporting instructions, please see:
<http://bugs.dragonflybsd.org/&gt;...
Reading symbols from /var/crash/kernel.1...done.

Unread portion of the kernel message buffer:
panic: assertion: parent != NULL in hammer_cursor_removed_node
Trace beginning at frame 0xdece6684
panic(dece66a8,0,0,dee36cf8,dece66c0) at panic+0x8c
panic(c057df63,c059cadf,c05696d3,dee36b18,0) at panic+0x8c
hammer_cursor_removed_node(dee36cf8,0,0,dece6818,0) at
hammer_cursor_removed_node+0x27
btree_remove(dee36cf8,0,dee36e38,dee36e38,dee36e38) at btree_remove+0x12f
btree_remove(0,0,c82c2040,c458ff98,dece67b0) at btree_remove+0x114
hammer_btree_delete(dece6818) at hammer_btree_delete+0x1ab
hammer_delete_at_cursor(dece6818,2,0,0,4b1d6cb3) at
hammer_delete_at_cursor+0x327
hammer_ioc_prune(dece6a74,da83c2d0,c458ff00,4,dece6956) at
hammer_ioc_prune+0x4c1
hammer_ioctl(da83c2d0,c0fc6801,c458ff00,1,c44674c8) at hammer_ioctl+0x1b8
hammer_vop_ioctl(dece6ad0,c0649380,db15eed0,4b1d6cb3,26a27419) at
hammer_vop_ioctl+0x2f
vop_ioctl(db15eed0,c45d44a8,c0fc6801,c458ff00,1) at vop_ioctl+0x3e
vn_ioctl(d784eb98,c0fc6801,c458ff00,c44674c8,dece6cf0) at vn_ioctl+0xe0
mapped_ioctl(4,c0fc6801,280b0188,0,dece6cf0) at mapped_ioctl+0x3e7
sys_ioctl(dece6cf0,6,0,0,da78c018) at sys_ioctl+0x17
syscall2(dece6d40) at syscall2+0x1ef
Xint0x80_syscall() at Xint0x80_syscall+0x36
Debugger("panic")
Uptime: 1h18m39s

dumping to dev ad0s1b, blockno 4214784
dump 2038 2037 2036 2035 2034 2033 2032 2031 2030 2029 2028 2027 2026 2025 2024
2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008
2007 2006 2005 2004 2003 2002 2001 2000 1999 1998 1997 1996 1995 1994 1993 1992
1991 1990 1989 1988 1987 1986 1985 1984 1983 1982 1981 1980 1979 1978 1977 1976
1975 1974 1973 1972 1971 1970 1969 1968 1967 1966 1965 1964 1963 1962 1961 1960
1959 1958 1957 1956 1955 1954 1953 1952 1951 1950 1949 1948 1947 1946 1945 1944
1943 1942 1941 1940 1939 1938 1937 1936 1935 1934 1933 1932 1931 1930 1929 1928
1927 1926 1925 1924 1923 1922 1921 1920 1919 1918 1917 1916 1915 1914 1913 1912
1911 1910 1909 1908 1907 1906 1905 1904 1903 1902 1901 1900 1899 1898 1897 1896
1895 1894 1893 1892 1891 1890 1889 1888 1887 1886 1885 1884 1883 1882 1881 1880
1879 1878 1877 1876 1875 1874 1873 1872 1871 1870 1869 1868 1867 1866 1865 1864
1863 1862 1861 1860 1859 1858 1857 1856 1855 1854 1853 1852 1851 1850 1849 1848
1847 1846 1845 1844 1843 1842 1841 1840 1839 1838 1837 1836 1835 1834 1833 1832
1831 1830 1829 1828 1827 1826 1825 1824 1823 1822 1821 1820 1819 1818 1817 1816
1815 1814 1813 1812 1811 1810 1809 1808 1807 1806 1805 1804 1803 1802 1801 1800
1799 1798 1797 1796 1795 1794 1793 1792 1791 1790 1789 1788 1787 1786 1785 1784
1783 1782 1781 1780 1779 1778 1777 1776 1775 1774 1773 1772 1771 1770 1769 1768
1767 1766 1765 1764 1763 1762 1761 1760 1759 1758 1757 1756 1755 1754 1753 1752
1751 1750 1749 1748 1747 1746 1745 1744 1743 1742 1741 1740 1739 1738 1737 1736
1735 1734 1733 1732 1731 1730 1729 1728 1727 1726 1725 1724 1723 1722 1721 1720
1719 1718 1717 1716 1715 1714 1713 1712 1711 1710 1709 1708 1707 1706 1705 1704
1703 1702 1701 1700 1699 1698 1697 1696 1695 1694 1693 1692 1691 1690 1689 1688
1687 1686 1685 1684 1683 1682 1681 1680 1679 1678 1677 1676 1675 1674 1673 1672
1671 1670 1669 1668 1667 1666 1665 1664 1663 1662 1661 1660 1659 1658 1657 1656
1655 1654 1653 1652 1651 1650 1649 1648 1647 1646 1645 1644 1643 1642 1641 1640
1639 1638 1637 1636 1635 1634 1633 1632 1631 1630 1629 1628 1627 1626 1625 1624
1623 1622 1621 1620 1619 1618 1617 1616 1615 1614 1613 1612 1611 1610 1609 1608
1607 1606 1605 1604 1603 1602 1601 1600 1599 1598 1597 1596 1595 1594 1593 1592
1591 1590 1589 1588 1587 1586 1585 1584 1583 1582 1581 1580 1579 1578 1577 1576
1575 1574 1573 1572 1571 1570 1569 1568 1567 1566 1565 1564 1563 1562 1561 1560
1559 1558 1557 1556 1555 1554 1553 1552 1551 1550 1549 1548 1547 1546 1545 1544
1543 1542 1541 1540 1539 1538 1537 1536 1535 1534 1533 1532 1531 1530 1529 1528
1527 1526 1525 1524 1523 1522 1521 1520 1519 1518 1517 1516 1515 1514 1513 1512
1511 1510 1509 1508 1507 1506 1505 1504 1503 1502 1501 1500 1499 1498 1497 1496
1495 1494 1493 1492 1491 1490 1489 1488 1487 1486 1485 1484 1483 1482 1481 1480
1479 1478 1477 1476 1475 1474 1473 1472 1471 1470 1469 1468 1467 1466 1465 1464
1463 1462 1461 1460 1459 1458 1457 1456 1455 1454 1453 1452 1451 1450 1449 1448
1447 1446 1445 1444 1443 1442 1441 1440 1439 1438 1437 1436 1435 1434 1433 1432
1431 1430 1429 1428 1427 1426 1425 1424 1423 1422 1421 1420 1419 1418 1417 1416
1415 1414 1413 1412 1411 1410 1409 1408 1407 1406 1405 1404 1403 1402 1401 1400
1399 1398 1397 1396 1395 1394 1393 1392 1391 1390 1389 1388 1387 1386 1385 1384
1383 1382 1381 1380 1379 1378 1377 1376 1375 1374 1373 1372 1371 1370 1369 1368
1367 1366 1365 1364 1363 1362 1361 1360 1359 1358 1357 1356 1355 1354 1353 1352
1351 1350 1349 1348 1347 1346 1345 1344 1343 1342 1341 1340 1339 1338 1337 1336
1335 1334 1333 1332 1331 1330 1329 1328 1327 1326 1325 1324 1323 1322 1321 1320
1319 1318 1317 1316 1315 1314 1313 1312 1311 1310 1309 1308 1307 1306 1305 1304
1303 1302 1301 1300 1299 1298 1297 1296 1295 1294 1293 1292 1291 1290 1289 1288
1287 1286 1285 1284 1283 1282 1281 1280 1279 1278 1277 1276 1275 1274 1273 1272
1271 1270 1269 1268 1267 1266 1265 1264 1263 1262 1261 1260 1259 1258 1257 1256
1255 1254 1253 1252 1251 1250 1249 1248 1247 1246 1245 1244 1243 1242 1241 1240
1239 1238 1237 1236 1235 1234 1233 1232 1231 1230 1229 1228 1227 1226 1225 1224
1223 1222 1221 1220 1219 1218 1217 1216 1215 1214 1213 1212 1211 1210 1209 1208
1207 1206 1205 1204 1203 1202 1201 1200 1199 1198 1197 1196 1195 1194 1193 1192
1191 1190 1189 1188 1187 1186 1185 1184 1183 1182 1181 1180 1179 1178 1177 1176
1175 1174 1173 1172 1171 1170 1169 1168 1167 1166 1165 1164 1163 1162 1161 1160
1159 1158 1157 1156 1155 1154 1153 1152 1151 1150 1149 1148 1147 1146 1145 1144
1143 1142 1141 1140 1139 1138 1137 1136 1135 1134 1133 1132 1131 1130 1129 1128
1127 1126 1125 1124 1123 1122 1121 1120 1119 1118 1117 1116 1115 1114 1113 1112
1111 1110 1109 1108 1107 1106 1105 1104 1103 1102 1101 1100 1099 1098 1097 1096
1095 1094 1093 1092 1091 1090 1089 1088 1087 1086 1085 1084 1083 1082 1081 1080
1079 1078 1077 1076 1075 1074 1073 1072 1071 1070 1069 1068 1067 1066 1065 1064
1063 1062 1061 1060 1059 1058 1057 1056 1055 1054 1053 1052 1051 1050 1049 1048
1047 1046 1045 1044 1043 1042 1041 1040 1039 1038 1037 1036 1035 1034 1033 1032
1031 1030 1029 1028 1027 1026 1025 1024 1023 1022 1021 1020 1019 1018 1017 1016
1015 1014 1013 1012 1011 1010 1009 1008 1007 1006 1005 1004 1003 1002 1001 1000
999 998 997 996 995 994 993 992 991 990 989 988 987 986 985 984 983 982 981 980
979 978 977 976 975 974 973 972 971 970 969 968 967 966 965 964 963 962 961 960
959 958 957 956 955 954 953 952 951 950 949 948 947 946 945 944 943 942 941 940
939 938 937 936 935 934 933 932 931 930 929 928 927 926 925 924 923 922 921 920
919 918 917 916 915 914 913 912 911 910 909 908 907 906 905 904 903 902 901 900
899 898 897 896 895 894 893 892 891 890 889 888 887 886 885 884 883 882 881 880
879 878 877 876 875 874 873 872 871 870 869 868 867 866 865 864 863 862 861 860
859 858 857 856 855 854 853 852 851 850 849 848 847 846 845 844 843 842 841 840
839 838 837 836 835 834 833 832 831 830 829 828 827 826 825 824 823 822 821 820
819 818 817 816 815 814 813 812 811 810 809 808 807 806 805 804 803 802 801 800
799 798 797 796 795 794 793 792 791 790 789 788 787 786 785 784 783 782 781 780
779 778 777 776 775 774 773 772 771 770 769 768 767 766 765 764 763 762 761 760
759 758 757 756 755 754 753 752 751 750 749 748 747 746 745 744 743 742 741 740
739 738 737 736 735 734 733 732 731 730 729 728 727 726 725 724 723 722 721 720
719 718 717 716 715 714 713 712 711 710 709 708 707 706 705 704 703 702 701 700
699 698 697 696 695 694 693 692 691 690 689 688 687 686 685 684 683 682 681 680
679 678 677 676 675 674 673 672 671 670 669 668 667 666 665 664 663 662 661 660
659 658 657 656 655 654 653 652 651 650 649 648 647 646 645 644 643 642 641 640
639 638 637 636 635 634 633 632 631 630 629 628 627 626 625 624 623 622 621 620
619 618 617 616 615 614 613 612 611 610 609 608 607 606 605 604 603 602 601 600
599 598 597 596 595 594 593 592 591 590 589 588 587 586 585 584 583 582 581 580
579 578 577 576 575 574 573 572 571 570 569 568 567 566 565 564 563 562 561 560
559 558 557 556 555 554 553 552 551 550 549 548 547 546 545 544 543 542 541 540
539 538 537 536 535 534 533 532 531 530 529 528 527 526 525 524 523 522 521 520
519 518 517 516 515 514 513 512 511 510 509 508 507 506 505 504 503 502 501 500
499 498 497 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480
479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460
459 458 457 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440
439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420
419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400
399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 380
379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360
359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340
339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322 321 320
319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300
299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280
279 278 277 276 275 274 273 272 271 270 269 268 267 266 265 264 263 262 261 260
259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240
239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220
219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200
199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180
179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160
159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140
139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120
119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100
99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73
72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46
45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19
18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Reading symbols from /boot/modules/acpi.ko...done.
Loaded symbols for /boot/modules/acpi.ko
Reading symbols from /boot/modules/null.ko...done.
Loaded symbols for /boot/modules/null.ko
get_mycpu () at ./machine/thread.h:83
83 __asm ("movl %%fs:globaldata,%0" : "=r" (gd) : "m"(_mycpu__dummy));
(kgdb) bt
#0 _get_mycpu () at ./machine/thread.h:83
#1 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:657
#2 0xc031e1e2 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:378
#3 0xc031e303 in panic (fmt=0xc057df63 "assertion: %s in %s") at
/usr/src/sys/kern/kern_shutdown.c:813
#4 0xc048acc6 in hammer_cursor_removed_node (node=0xdee36cf8, parent=0x0,
index=0) at /usr/src/sys/vfs/hammer/hammer_cursor.c:722
#5 0xc048886d in btree_remove (cursor=0xdece6818) at
/usr/src/sys/vfs/hammer/hammer_btree.c:2229
#6 0xc0488852 in btree_remove (cursor=0xdece6818) at
/usr/src/sys/vfs/hammer/hammer_btree.c:2227
#7 0xc0488c51 in hammer_btree_delete (cursor=0xdece6818) at
/usr/src/sys/vfs/hammer/hammer_btree.c:896
#8 0xc0498382 in hammer_delete_at_cursor (cursor=0xdece6818,
delete_flags=<value optimized out>, delete_tid=0, delete_ts=1260219571, track=0,
stat_bytes=0xc458ffb0) at /usr/src/sys/vfs/hammer/hammer_object.c:2488
#9 0xc049d4dd in hammer_ioc_prune (trans=0xdece6a74, ip=0xda83c2d0,
prune=0xc458ff00) at /usr/src/sys/vfs/hammer/hammer_prune.c:184
#10 0xc0493b94 in hammer_ioctl (ip=0xda83c2d0, com=3237767169, data=0xc458ff00
"\001", fflag=1, cred=0xc44674c8)
at /usr/src/sys/vfs/hammer/hammer_ioctl.c:76
#11 0xc04a544d in hammer_vop_ioctl (ap=0xdece6ad0) at
/usr/src/sys/vfs/hammer/hammer_vnops.c:2304
#12 0xc0375ce5 in vop_ioctl (ops=0xdb15eed0, vp=0xc45d44a8, command=3237767169,
data=0xc458ff00 "\001", fflag=1, cred=0xc44674c8, msg=0xdece6cf0)
at /usr/src/sys/kern/vfs_vopops.c:389
#13 0xc0375809 in vn_ioctl (fp=0xd784eb98, com=3237767169, data=0xc458ff00
"\001", ucred=0xc44674c8, msg=0xdece6cf0)
at /usr/src/sys/kern/vfs_vnops.c:938
#14 0xc033e9ef in fo_ioctl (fd=4, com=3237767169, uspc_data=0x280b0188 <Address
0x280b0188 out of bounds>, map=0x0, msg=0xdece6cf0)
at /usr/src/sys/sys/file2.h:88
#15 mapped_ioctl (fd=4, com=3237767169, uspc_data=0x280b0188 <Address 0x280b0188
out of bounds>, map=0x0, msg=0xdece6cf0)
at /usr/src/sys/kern/sys_generic.c:697
#16 0xc033ea78 in sys_ioctl (uap=0xdece6cf0) at
/usr/src/sys/kern/sys_generic.c:521
#17 0xc05310b3 in syscall2 (frame=0xdece6d40) at
/usr/src/sys/platform/pc32/i386/trap.c:1339
#18 0xc0521586 in Xint0x80_syscall () at
/usr/src/sys/platform/pc32/i386/exception.s:876
#19 0xdece6d40 in ?? ()
#20 0x0000007b in ?? ()
#21 0x0000002f in ?? ()
#22 0x0001002f in ?? ()
#23 0xbfbf002f in ?? ()
#24 0x280b0080 in ?? ()
#25 0x00000000 in ?? ()
(kgdb)

Hi,

This is a known issue. Matt gave us a patch to fix this:
http://leaf.dragonflybsd.org/~tuxillo/archive/0001-HAMMER-Fix-cleanup.patch
Just fetch it and 'git am 0001-HAMMER-Fix-cleanup.patch' to apply to
your local checkout and then quickkernel.

I don't know what's blocking to commit this fix.

Btw, you can take advantage of Alex's minidump port in HEAD to avoid
such huge dumps :-)

Cheers,
Antonio Huete

2009/12/7 jb (via DragonFly issue tracker) <sinknull@leaf.dragonflybsd.org>:

jb <zeroaleph@gmail.com> added the comment:

Hello,

I've hitted the same assertion, trying to 'hammer version-upgrade / 3'. Now, I
can reliably trigger this panic when doing a hammer cleanup.

Let me know if you need more info. I can probably arrange to upload the dump
somewhere in the following days if you need it, just tell me.

uname a
DragonFly libellule.jos.net 2.5.1-DEVELOPMENT DragonFly v2.5.1.352.g37653
DEVELOPMENT #1: Sun Dec  6 20:08:01 CET 2009
root@libellule.jos.net:/usr/obj/usr/src/sys/GENERIC  i386

libellule# hammer cleanup
cleanup /                    - handle PFS #0 using /var/hammer/root
          snapshots - skip
              prune - skip
          rebalance - skip
            reblock - skip
             recopy - skip
cleanup /var                 - handle PFS #1 using /var/hammer/var
          snapshots - skip
              prune - skip
          rebalance - skip
            reblock - skip
             recopy - skip
cleanup /tmp                 - handle PFS #2 using /var/hammer/tmp
          snapshots - disabled
              prune - skip
          rebalance - skip
            reblock - skip
             recopy - skip
cleanup /usr                 - handle PFS #3 using /var/hammer/usr
          snapshots - skip
              prune - skip
          rebalance - skip
            reblock - skip
             recopy - skip
cleanup /home                - handle PFS #4 using /var/hammer/home
          snapshots - run
              prune - run
          rebalance - run..
            reblock - run....
             recopy - run....
cleanup /usr/obj             -(migrating)  HAMMER UPGRADE: Moving snapshots
       Moving snapshots from /usr/obj/snapshots to /var/hammer/usr/obj
 handle PFS #5 using /var/hammer/usr/obj
          snapshots - disabled
              prune - run
<PANIC!>

libellule# kgdb kernel.1 vmcore.1
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-dragonfly".
For bug reporting instructions, please see:
<http://bugs.dragonflybsd.org/&gt;...
Reading symbols from /var/crash/kernel.1...done.

Unread portion of the kernel message buffer:
panic: assertion: parent != NULL in hammer_cursor_removed_node
Trace beginning at frame 0xdece6684
panic(dece66a8,0,0,dee36cf8,dece66c0) at panic+0x8c
panic(c057df63,c059cadf,c05696d3,dee36b18,0) at panic+0x8c
hammer_cursor_removed_node(dee36cf8,0,0,dece6818,0) at
hammer_cursor_removed_node+0x27
btree_remove(dee36cf8,0,dee36e38,dee36e38,dee36e38) at btree_remove+0x12f
btree_remove(0,0,c82c2040,c458ff98,dece67b0) at btree_remove+0x114
hammer_btree_delete(dece6818) at hammer_btree_delete+0x1ab
hammer_delete_at_cursor(dece6818,2,0,0,4b1d6cb3) at
hammer_delete_at_cursor+0x327
hammer_ioc_prune(dece6a74,da83c2d0,c458ff00,4,dece6956) at
hammer_ioc_prune+0x4c1
hammer_ioctl(da83c2d0,c0fc6801,c458ff00,1,c44674c8) at hammer_ioctl+0x1b8
hammer_vop_ioctl(dece6ad0,c0649380,db15eed0,4b1d6cb3,26a27419) at
hammer_vop_ioctl+0x2f
vop_ioctl(db15eed0,c45d44a8,c0fc6801,c458ff00,1) at vop_ioctl+0x3e
vn_ioctl(d784eb98,c0fc6801,c458ff00,c44674c8,dece6cf0) at vn_ioctl+0xe0
mapped_ioctl(4,c0fc6801,280b0188,0,dece6cf0) at mapped_ioctl+0x3e7
sys_ioctl(dece6cf0,6,0,0,da78c018) at sys_ioctl+0x17
syscall2(dece6d40) at syscall2+0x1ef
Xint0x80_syscall() at Xint0x80_syscall+0x36
Debugger("panic")
Uptime: 1h18m39s

dumping to dev ad0s1b, blockno 4214784
dump 2038 2037 2036 2035 2034 2033 2032 2031 2030 2029 2028 2027 2026 2025 2024
2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008
2007 2006 2005 2004 2003 2002 2001 2000 1999 1998 1997 1996 1995 1994 1993 1992
1991 1990 1989 1988 1987 1986 1985 1984 1983 1982 1981 1980 1979 1978 1977 1976
1975 1974 1973 1972 1971 1970 1969 1968 1967 1966 1965 1964 1963 1962 1961 1960
1959 1958 1957 1956 1955 1954 1953 1952 1951 1950 1949 1948 1947 1946 1945 1944
1943 1942 1941 1940 1939 1938 1937 1936 1935 1934 1933 1932 1931 1930 1929 1928
1927 1926 1925 1924 1923 1922 1921 1920 1919 1918 1917 1916 1915 1914 1913 1912
1911 1910 1909 1908 1907 1906 1905 1904 1903 1902 1901 1900 1899 1898 1897 1896
1895 1894 1893 1892 1891 1890 1889 1888 1887 1886 1885 1884 1883 1882 1881 1880
1879 1878 1877 1876 1875 1874 1873 1872 1871 1870 1869 1868 1867 1866 1865 1864
1863 1862 1861 1860 1859 1858 1857 1856 1855 1854 1853 1852 1851 1850 1849 1848
1847 1846 1845 1844 1843 1842 1841 1840 1839 1838 1837 1836 1835 1834 1833 1832
1831 1830 1829 1828 1827 1826 1825 1824 1823 1822 1821 1820 1819 1818 1817 1816
1815 1814 1813 1812 1811 1810 1809 1808 1807 1806 1805 1804 1803 1802 1801 1800
1799 1798 1797 1796 1795 1794 1793 1792 1791 1790 1789 1788 1787 1786 1785 1784
1783 1782 1781 1780 1779 1778 1777 1776 1775 1774 1773 1772 1771 1770 1769 1768
1767 1766 1765 1764 1763 1762 1761 1760 1759 1758 1757 1756 1755 1754 1753 1752
1751 1750 1749 1748 1747 1746 1745 1744 1743 1742 1741 1740 1739 1738 1737 1736
1735 1734 1733 1732 1731 1730 1729 1728 1727 1726 1725 1724 1723 1722 1721 1720
1719 1718 1717 1716 1715 1714 1713 1712 1711 1710 1709 1708 1707 1706 1705 1704
1703 1702 1701 1700 1699 1698 1697 1696 1695 1694 1693 1692 1691 1690 1689 1688
1687 1686 1685 1684 1683 1682 1681 1680 1679 1678 1677 1676 1675 1674 1673 1672
1671 1670 1669 1668 1667 1666 1665 1664 1663 1662 1661 1660 1659 1658 1657 1656
1655 1654 1653 1652 1651 1650 1649 1648 1647 1646 1645 1644 1643 1642 1641 1640
1639 1638 1637 1636 1635 1634 1633 1632 1631 1630 1629 1628 1627 1626 1625 1624
1623 1622 1621 1620 1619 1618 1617 1616 1615 1614 1613 1612 1611 1610 1609 1608
1607 1606 1605 1604 1603 1602 1601 1600 1599 1598 1597 1596 1595 1594 1593 1592
1591 1590 1589 1588 1587 1586 1585 1584 1583 1582 1581 1580 1579 1578 1577 1576
1575 1574 1573 1572 1571 1570 1569 1568 1567 1566 1565 1564 1563 1562 1561 1560
1559 1558 1557 1556 1555 1554 1553 1552 1551 1550 1549 1548 1547 1546 1545 1544
1543 1542 1541 1540 1539 1538 1537 1536 1535 1534 1533 1532 1531 1530 1529 1528
1527 1526 1525 1524 1523 1522 1521 1520 1519 1518 1517 1516 1515 1514 1513 1512
1511 1510 1509 1508 1507 1506 1505 1504 1503 1502 1501 1500 1499 1498 1497 1496
1495 1494 1493 1492 1491 1490 1489 1488 1487 1486 1485 1484 1483 1482 1481 1480
1479 1478 1477 1476 1475 1474 1473 1472 1471 1470 1469 1468 1467 1466 1465 1464
1463 1462 1461 1460 1459 1458 1457 1456 1455 1454 1453 1452 1451 1450 1449 1448
1447 1446 1445 1444 1443 1442 1441 1440 1439 1438 1437 1436 1435 1434 1433 1432
1431 1430 1429 1428 1427 1426 1425 1424 1423 1422 1421 1420 1419 1418 1417 1416
1415 1414 1413 1412 1411 1410 1409 1408 1407 1406 1405 1404 1403 1402 1401 1400
1399 1398 1397 1396 1395 1394 1393 1392 1391 1390 1389 1388 1387 1386 1385 1384
1383 1382 1381 1380 1379 1378 1377 1376 1375 1374 1373 1372 1371 1370 1369 1368
1367 1366 1365 1364 1363 1362 1361 1360 1359 1358 1357 1356 1355 1354 1353 1352
1351 1350 1349 1348 1347 1346 1345 1344 1343 1342 1341 1340 1339 1338 1337 1336
1335 1334 1333 1332 1331 1330 1329 1328 1327 1326 1325 1324 1323 1322 1321 1320
1319 1318 1317 1316 1315 1314 1313 1312 1311 1310 1309 1308 1307 1306 1305 1304
1303 1302 1301 1300 1299 1298 1297 1296 1295 1294 1293 1292 1291 1290 1289 1288
1287 1286 1285 1284 1283 1282 1281 1280 1279 1278 1277 1276 1275 1274 1273 1272
1271 1270 1269 1268 1267 1266 1265 1264 1263 1262 1261 1260 1259 1258 1257 1256
1255 1254 1253 1252 1251 1250 1249 1248 1247 1246 1245 1244 1243 1242 1241 1240
1239 1238 1237 1236 1235 1234 1233 1232 1231 1230 1229 1228 1227 1226 1225 1224
1223 1222 1221 1220 1219 1218 1217 1216 1215 1214 1213 1212 1211 1210 1209 1208
1207 1206 1205 1204 1203 1202 1201 1200 1199 1198 1197 1196 1195 1194 1193 1192
1191 1190 1189 1188 1187 1186 1185 1184 1183 1182 1181 1180 1179 1178 1177 1176
1175 1174 1173 1172 1171 1170 1169 1168 1167 1166 1165 1164 1163 1162 1161 1160
1159 1158 1157 1156 1155 1154 1153 1152 1151 1150 1149 1148 1147 1146 1145 1144
1143 1142 1141 1140 1139 1138 1137 1136 1135 1134 1133 1132 1131 1130 1129 1128
1127 1126 1125 1124 1123 1122 1121 1120 1119 1118 1117 1116 1115 1114 1113 1112
1111 1110 1109 1108 1107 1106 1105 1104 1103 1102 1101 1100 1099 1098 1097 1096
1095 1094 1093 1092 1091 1090 1089 1088 1087 1086 1085 1084 1083 1082 1081 1080
1079 1078 1077 1076 1075 1074 1073 1072 1071 1070 1069 1068 1067 1066 1065 1064
1063 1062 1061 1060 1059 1058 1057 1056 1055 1054 1053 1052 1051 1050 1049 1048
1047 1046 1045 1044 1043 1042 1041 1040 1039 1038 1037 1036 1035 1034 1033 1032
1031 1030 1029 1028 1027 1026 1025 1024 1023 1022 1021 1020 1019 1018 1017 1016
1015 1014 1013 1012 1011 1010 1009 1008 1007 1006 1005 1004 1003 1002 1001 1000
999 998 997 996 995 994 993 992 991 990 989 988 987 986 985 984 983 982 981 980
979 978 977 976 975 974 973 972 971 970 969 968 967 966 965 964 963 962 961 960
959 958 957 956 955 954 953 952 951 950 949 948 947 946 945 944 943 942 941 940
939 938 937 936 935 934 933 932 931 930 929 928 927 926 925 924 923 922 921 920
919 918 917 916 915 914 913 912 911 910 909 908 907 906 905 904 903 902 901 900
899 898 897 896 895 894 893 892 891 890 889 888 887 886 885 884 883 882 881 880
879 878 877 876 875 874 873 872 871 870 869 868 867 866 865 864 863 862 861 860
859 858 857 856 855 854 853 852 851 850 849 848 847 846 845 844 843 842 841 840
839 838 837 836 835 834 833 832 831 830 829 828 827 826 825 824 823 822 821 820
819 818 817 816 815 814 813 812 811 810 809 808 807 806 805 804 803 802 801 800
799 798 797 796 795 794 793 792 791 790 789 788 787 786 785 784 783 782 781 780
779 778 777 776 775 774 773 772 771 770 769 768 767 766 765 764 763 762 761 760
759 758 757 756 755 754 753 752 751 750 749 748 747 746 745 744 743 742 741 740
739 738 737 736 735 734 733 732 731 730 729 728 727 726 725 724 723 722 721 720
719 718 717 716 715 714 713 712 711 710 709 708 707 706 705 704 703 702 701 700
699 698 697 696 695 694 693 692 691 690 689 688 687 686 685 684 683 682 681 680
679 678 677 676 675 674 673 672 671 670 669 668 667 666 665 664 663 662 661 660
659 658 657 656 655 654 653 652 651 650 649 648 647 646 645 644 643 642 641 640
639 638 637 636 635 634 633 632 631 630 629 628 627 626 625 624 623 622 621 620
619 618 617 616 615 614 613 612 611 610 609 608 607 606 605 604 603 602 601 600
599 598 597 596 595 594 593 592 591 590 589 588 587 586 585 584 583 582 581 580
579 578 577 576 575 574 573 572 571 570 569 568 567 566 565 564 563 562 561 560
559 558 557 556 555 554 553 552 551 550 549 548 547 546 545 544 543 542 541 540
539 538 537 536 535 534 533 532 531 530 529 528 527 526 525 524 523 522 521 520
519 518 517 516 515 514 513 512 511 510 509 508 507 506 505 504 503 502 501 500
499 498 497 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480
479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460
459 458 457 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440
439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420
419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400
399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 380
379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360
359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340
339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322 321 320
319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300
299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280
279 278 277 276 275 274 273 272 271 270 269 268 267 266 265 264 263 262 261 260
259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240
239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220
219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200
199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180
179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160
159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140
139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120
119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100
99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73
72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46
45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19
18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Reading symbols from /boot/modules/acpi.ko...done.
Loaded symbols for /boot/modules/acpi.ko
Reading symbols from /boot/modules/null.ko...done.
Loaded symbols for /boot/modules/null.ko
get_mycpu () at ./machine/thread.h:83
83          _asm ("movl %%fs:globaldata,%0" : "=r" (gd) : "m"(_mycpu_dummy));
(kgdb) bt
#0  _get_mycpu () at ./machine/thread.h:83
#1  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:657
#2  0xc031e1e2 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:378
#3  0xc031e303 in panic (fmt=0xc057df63 "assertion: %s in %s") at
/usr/src/sys/kern/kern_shutdown.c:813
#4  0xc048acc6 in hammer_cursor_removed_node (node=0xdee36cf8, parent=0x0,
index=0) at /usr/src/sys/vfs/hammer/hammer_cursor.c:722
#5  0xc048886d in btree_remove (cursor=0xdece6818) at
/usr/src/sys/vfs/hammer/hammer_btree.c:2229
#6  0xc0488852 in btree_remove (cursor=0xdece6818) at
/usr/src/sys/vfs/hammer/hammer_btree.c:2227
#7  0xc0488c51 in hammer_btree_delete (cursor=0xdece6818) at
/usr/src/sys/vfs/hammer/hammer_btree.c:896
#8  0xc0498382 in hammer_delete_at_cursor (cursor=0xdece6818,
delete_flags=<value optimized out>, delete_tid=0, delete_ts=1260219571, track=0,
   stat_bytes=0xc458ffb0) at /usr/src/sys/vfs/hammer/hammer_object.c:2488
#9  0xc049d4dd in hammer_ioc_prune (trans=0xdece6a74, ip=0xda83c2d0,
prune=0xc458ff00) at /usr/src/sys/vfs/hammer/hammer_prune.c:184
#10 0xc0493b94 in hammer_ioctl (ip=0xda83c2d0, com=3237767169, data=0xc458ff00
"\001", fflag=1, cred=0xc44674c8)
   at /usr/src/sys/vfs/hammer/hammer_ioctl.c:76
#11 0xc04a544d in hammer_vop_ioctl (ap=0xdece6ad0) at
/usr/src/sys/vfs/hammer/hammer_vnops.c:2304
#12 0xc0375ce5 in vop_ioctl (ops=0xdb15eed0, vp=0xc45d44a8, command=3237767169,
data=0xc458ff00 "\001", fflag=1, cred=0xc44674c8, msg=0xdece6cf0)
   at /usr/src/sys/kern/vfs_vopops.c:389
#13 0xc0375809 in vn_ioctl (fp=0xd784eb98, com=3237767169, data=0xc458ff00
"\001", ucred=0xc44674c8, msg=0xdece6cf0)
   at /usr/src/sys/kern/vfs_vnops.c:938
#14 0xc033e9ef in fo_ioctl (fd=4, com=3237767169, uspc_data=0x280b0188 <Address
0x280b0188 out of bounds>, map=0x0, msg=0xdece6cf0)
   at /usr/src/sys/sys/file2.h:88
#15 mapped_ioctl (fd=4, com=3237767169, uspc_data=0x280b0188 <Address 0x280b0188
out of bounds>, map=0x0, msg=0xdece6cf0)
   at /usr/src/sys/kern/sys_generic.c:697
#16 0xc033ea78 in sys_ioctl (uap=0xdece6cf0) at
/usr/src/sys/kern/sys_generic.c:521
#17 0xc05310b3 in syscall2 (frame=0xdece6d40) at
/usr/src/sys/platform/pc32/i386/trap.c:1339
#18 0xc0521586 in Xint0x80_syscall () at
/usr/src/sys/platform/pc32/i386/exception.s:876
#19 0xdece6d40 in ?? ()
#20 0x0000007b in ?? ()
#21 0x0000002f in ?? ()
#22 0x0001002f in ?? ()
#23 0xbfbf002f in ?? ()
#24 0x280b0080 in ?? ()
#25 0x00000000 in ?? ()
(kgdb)

_____________________________________________
DragonFly issue tracker <bugs@lists.dragonflybsd.org>
<http://bugs.dragonflybsd.org/issue1597>
_____________________________________________

:
:Hi,
:
:This is a known issue. Matt gave us a patch to fix this:
:http://leaf.dragonflybsd.org/~tuxillo/archive/0001-HAMMER-Fix-cleanup.patch
:Just fetch it and 'git am 0001-HAMMER-Fix-cleanup.patch' to apply to
:your local checkout and then quickkernel.
:
:I don't know what's blocking to commit this fix.
:
:Btw, you can take advantage of Alex's minidump port in HEAD to avoid
:such huge dumps :-)
:
:Cheers,
:Antonio Huete

There is still a another panic related to searches going past the end
of the key range (which isn't supposed to be able to happen). I don't
know if the patch makes it better or worse, if the patch is the cause
(I don't think it is), or if there are other consequences to the patch.

I was holding off the commit while trying to track it down but maybe
I should just commit that patch and treat the remaining panic as
a separate issue.

I do still need new crash dumps for the remaining panic (not the
parent != NULL panic but the other one... I think it was a key
comparison panic in the btree iteration / search code).

-Matt

Hi Matt,

No luck in getting the panic you were looking for? Do you think there is a way
we can hit it?

Cheers,
Antonio Huete