:I'll provide a dump if needed.
:
:panic(c051dc97,c0540bb7,c050506a,c2dcd000,cb281ae0) at panic+0x9f
:hammer_cursor_removed_node(cb281ae0,c8e55eb0,3,c8e55eb0,cad9cac4) at
:hammer_cursor_removed_node+0x72 btree_remove
:...
:-- 
:Gergo Szakal MD <bastyaelvtars@gmail.com>

    Got it.  It's a bug in the cursor recovery code that I added to
    support mirror propagation.  It's basically saying: "HEY MATT! YOU
    FORGOT TO CALL hammer_cursor_deleted_element() SOMEWHERE!".

    And indeed, that is what happened.  I call it properly when an
    element is deleted from a leaf in the B-Tree, but not when a
    leaf becomes empty and is deleted from its parent internal node.
    So cursors sitting in deadlock recovery on that internal node
    were not being updated.

    I'll commit a fix today, soon as I track down the bug Michael's
    CR program revealed.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>

Here is the dump anyway:

http://medica.semi.szote.u-szeged.hu/~szg/crash080710.tar.bz2

Oh my, forgot the backtrace. BTW I was deleting some files, nothing
special.

#0  dumpsys () at ./machine/thread.h:83
#1  0xc02c5a92 in boot (howto=260)
#at /usr/src/sys/kern/kern_shutdown.c:375 2  0xc02c5bb3 in panic
#(fmt=0xc0520a97 "from debugger")
#at /usr/src/sys/kern/kern_shutdown.c:800 3  0xc0164279 in db_panic
#(addr=-1068765700, have_addr=0, count=-1, modif=0xcad9c918 "")
#at /usr/src/sys/ddb/db_command.c:447 4  0xc01648e4 in db_command_loop
#() at /usr/src/sys/ddb/db_command.c:343 5  0xc0166e5c in db_trap
#(type=3, code=0) at /usr/src/sys/ddb/db_trap.c:71 6  0xc04bef4f in
#kdb_trap (type=3, code=0, regs=0xcad9ca10)
#at /usr/src/sys/platform/pc32/i386/db_interface.c:148 7  0xc04d03c2 in
#trap (frame=0xcad9ca10) at /usr/src/sys/platform/pc32/i386/trap.c:832
#8  0xc04bfc67 in calltrap ()
#at /usr/src/sys/platform/pc32/i386/exception.s:785 9  0xc04bedfc in
#Debugger (msg=0xc0534f6b "panic") at ./cpu/cpufunc.h:73 10 0xc02c5baa
#in panic (fmt=0xc051dc97 "assertion: %s in %s")
#at /usr/src/sys/kern/kern_shutdown.c:798 11 0xc0429905 in
#hammer_cursor_removed_node (node=0xcb281ae0, parent=0xc8e55eb0,
#index=3)
    at /usr/src/sys/vfs/hammer/hammer_cursor.c:694
#12 0xc04277dc in btree_remove (cursor=0xcad9cc60)
#at /usr/src/sys/vfs/hammer/hammer_btree.c:2107 13 0xc042780b in
#btree_remove (cursor=0xcad9cc60)
#at /usr/src/sys/vfs/hammer/hammer_btree.c:2123 14 0xc0427b77 in
#hammer_btree_delete (cursor=0xcad9cc60)
#at /usr/src/sys/vfs/hammer/hammer_btree.c:860 15 0xc0433a7e in
#hammer_delete_at_cursor (cursor=0xcad9cc60, delete_flags=<value
#optimized out>, stat_bytes=0x0)
    at /usr/src/sys/vfs/hammer/hammer_object.c:2023
#16 0xc0433c26 in hammer_ip_delete_record (cursor=0xcad9cc60,
#ip=0xcb2f5c50, tid=4679118898)
    at /usr/src/sys/vfs/hammer/hammer_object.c:1932
#17 0xc0434458 in hammer_ip_sync_record_cursor (cursor=0xcad9cc60,
#record=0xc1560fc8) at /usr/src/sys/vfs/hammer/hammer_object.c:1054 18
#0xc042cfe8 in hammer_sync_record_callback (record=0xc1560fc8,
#data=0xcad9cc60) at /usr/src/sys/vfs/hammer/hammer_inode.c:2054 19
#0xc0431f9d in hammer_rec_rb_tree_RB_SCAN (head=0xcb2f5d9c,
#scancmp=0xc0431eab <hammer_rec_rb_tree_SCANCMP_ALL>,
    callback=0xc042ce90 <hammer_sync_record_callback>, data=0xcad9cc60)
at /usr/src/sys/vfs/hammer/hammer_object.c:243
#20 0xc042c992 in hammer_sync_inode (ip=0xcb2f5c50)
#at /usr/src/sys/vfs/hammer/hammer_inode.c:2233 21 0xc042ac15 in
#hammer_flusher_flush_inode (ip=0xcb2f5c50, trans=0xcad87114)
#at /usr/src/sys/vfs/hammer/hammer_flusher.c:361 22 0xc042adf3 in
#hammer_flusher_slave_thread (arg=0xc1666870)
#at /usr/src/sys/vfs/hammer/hammer_flusher.c:236 23 0xc02cc80c in
#lwkt_deschedule_self (td=Cannot access memory at address 0x8
) at /usr/src/sys/kern/lwkt_thread.c:223
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

I'll provide a dump if needed.

panic(c051dc97,c0540bb7,c050506a,c2dcd000,cb281ae0) at panic+0x9f
hammer_cursor_removed_node(cb281ae0,c8e55eb0,3,c8e55eb0,cad9cac4) at
hammer_cursor_removed_node+0x72 btree_remove
(cb281ae0,0,cb282530,80000000,c2e99000) at btree_remove+0xc1
btree_remove(0,cad9cc60,c2e99040,c2e99040,cad9cb78) at btree_remove
+0xf0 hammer_btree_delete(cad9cc60,16737fe0,90000000,16,0) at
hammer_btree_delete+0x162 hammer_delete_at_cursor
(cad9cc60,3,0,cb2f5c50,0) at hammer_delete_at_cursor+0x226
hammer_ip_delete_record(cad9cc60,cb2f5c50,16e5b032,1,cb2f5c50) at
hammer_ip_delete_record+0x103 hammer_ip_sync_record_cursor
(cad9cc60,c1560fc8,0,0,c1560fc8) at hammer_ip_sync_record_cursor+0x30d
hammer_sync_record_callback(c1560fc8,cad9cc60,0,0,0) at
hammer_sync_record_callback+0x158 hammer_rec_rb_tree_RB_SCAN
(cb2f5d9c,0,c042ce90,cad9cc60) at hammer_rec_rb_tree_RB_SCAN+0xab
hammer_sync_inode(cb2f5c50,cad87100,cad87000) at hammer_sync_inode
+0x252 hammer_flusher_flush_inode
(cad87114,cad87114,c1666870,cad87000,3f) at hammer_flusher_flush_inode
+0x29 hammer_flusher_slave_thread(c1666870,0,0,0,0) at
hammer_flusher_slave_thread+0xe1 lwkt_exit() at lwkt_exit