Bug #1276

panic: assertion: volume->io.lock.refs == 0 in hammer_unload_volume

Added by thomas.nikolajsen almost 6 years ago. Updated almost 3 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

Using HEAD of yesterday, Feb 8, 2009,
I got this panic at shutdown.

Have seen same panic a few times using 2.0.1.

Uploading crash dump to leaf as .28.

-thomas
-
panic: assertion: volume->io.lock.refs == 0 in hammer_unload_volume
Trace beginning at frame 0xd28d1b4c
panic(d28d1b70,c0d9b308,c0e51004,d20d4000,d28d1b88) at panic+0x8c
panic(d20cd972,d20ce19c,d20ccfc1,1108,c0d9b308) at panic+0x8c
hammer_unload_volume(c0d9b308,0,0,0,0) at hammer_unload_volume+0x6c
hammer_vol_rb_tree_RB_SCAN(d20d400c,0,d20ba4ee,0,d20d402c) at
hammer_vol_rb_tree_RB_SCAN+0xad
hammer_free_hmp(c7a47a10,c7a47808,d28d1c2c,c01f1f89,c7a47808) at
hammer_free_hmp+0x186
hammer_vfs_unmount(c7a47808,80000,80000,c7a47828,1) at hammer_vfs_unmount+0x33
dounmount(c7a47808,80000,c7a47808,1,d28d1c7c) at dounmount+0x1a8
vfs_umountall_callback(c7a47808,0,c0416574,0,1) at vfs_umountall_callback+0x13
mountlist_scan(c01eaced,0,6) at mountlist_scan+0xe8
vfs_unmountall(0,0,1,c019def6,c0d60870) at vfs_unmountall+0x11
boot(0,d28d1d34,c0305d65,d28d1cf0,6) at boot+0x21d
sys_reboot(d28d1cf0,6,3d1c,0,c7a42dc8) at sys_reboot+0x25
syscall2(d28d1d40) at syscall2+0x1ef
Xint0x80_syscall() at Xint0x80_syscall+0x36

History

#1 Updated by dillon almost 6 years ago

:Using HEAD of yesterday, Feb 8, 2009,
:I got this panic at shutdown.
:
:Have seen same panic a few times using 2.0.1.
:
:Uploading crash dump to leaf as .28.
:
: -thomas

Upload your hammer.ko module as well, I need it to debug the hammer
part of the trace and related structures.

I just fixed a ref count issue in master that *might* be the cause.
I won't be able to rule it in our out until I look at the core.

-Matt

#2 Updated by thomas.nikolajsen almost 6 years ago

Ahh, I don't have the hammer module for this build anymore
(just installed new kernel twice); will save it next time.

-thomas

#3 Updated by thomas.nikolajsen almost 6 years ago

New crash dump using HEAD of Feb 12, 2009.

Uploading to leaf as *.29 (with all modules in use).

-thomas

#4 Updated by dillon almost 6 years ago

:Thomas Nikolajsen <> added the comment:
:
:New crash dump using HEAD of Feb 12, 2009.
:
:Uploading to leaf as *.29 (with all modules in use).
:
: -thomas

Thomas, I believe I have found it. Please try the enclosed patch.
When a buffer collision occurs under heavy loads a ref on the volume
structure was not being undone, leading to a ref count leak on the
volume structure which will result in a panic on umount.

This fix will not make it into 2.2. It needs to be tested and verified
to have fixed the problem, then I'll commit and MFC it.

-Matt
Matthew Dillon
<>

diff --git a/sys/vfs/hammer/hammer_ondisk.c b/sys/vfs/hammer/hammer_ondisk.c
index 711fe65..214e1ff 100644
--- a/sys/vfs/hammer/hammer_ondisk.c
+++ b/sys/vfs/hammer/hammer_ondisk.c
@@ -639,8 +639,10 @@ again:
* Insert the buffer into the RB tree and handle late collisions.
*/
if (RB_INSERT(hammer_buf_rb_tree, &hmp->rb_bufs_root, buffer)) {
- hammer_unref(&buffer->io.lock);
+ hammer_unref(&buffer->io.lock); /* safety */
--hammer_count_buffers;
+ hammer_rel_volume(volume, 0);
+ buffer->io.volume = NULL; /* safety */
kfree(buffer, hmp->m_misc);
goto again;
}

#5 Updated by dillon over 5 years ago

::Thomas Nikolajsen <> added the comment:
::
::New crash dump using HEAD of Feb 12, 2009.
::
::Uploading to leaf as *.29 (with all modules in use).
::
:: -thomas
:
: Thomas, I believe I have found it. Please try the enclosed patch.
: When a buffer collision occurs under heavy loads a ref on the volume
: structure was not being undone, leading to a ref count leak on the
: volume structure which will result in a panic on umount.
:
: This fix will not make it into 2.2. It needs to be tested and verified
: to have fixed the problem, then I'll commit and MFC it.
:
: -Matt
: Matthew Dillon
: <>

I've committed the patch. Did you try it and get any more crashes
on umount from that particular volume.io.refs assertion?

-Matt

#6 Updated by thomas.nikolajsen over 5 years ago

I have applied patch to 2.2.0-RELEASE, no panic seen.
Thank you for fixing, I have closed issue.

-thomas

#7 Updated by thomas.nikolajsen over 5 years ago

panic at shutdown using 2.2-RELEASE from 17th Feb 2009
with patch from this case applied;
same as 530dd4947cb729aeaa1e5b57df82ff510cb0a49b.

Seems like panic doesn't occur so often after patch applied,
but it apparently still can happen.

Btw: it seems like commit isn't MFCed to 2.2-RELEASE yet (I could do this).

Uploading to leaf as *.30 (with all modules in use).

-thomas
-
panic: assertion: volume->io.lock.refs == 0 in hammer_unload_volume
Trace beginning at frame 0xc4d59b4c
panic(c4d59b70,c0d9b308,c0e1d59c,d2064000,c4d59b88) at panic+0x8c
panic(c055c9b2,c055d1dc,c055c001,1108,c0d9b308) at panic+0x8c
hammer_unload_volume(c0d9b308,0,0,0,0) at hammer_unload_volume+0x6c
hammer_vol_rb_tree_RB_SCAN(d206400c,0,c05494fe,0,d206402c) at
hammer_vol_rb_tree_RB_SCAN+0xad
hammer_free_hmp(c7a47a10,c7a47808,c4d59c2c,c01f1f61,c7a47808) at
hammer_free_hmp+0x186
hammer_vfs_unmount(c7a47808,80000,80000,c7a47828,1) at hammer_vfs_unmount+0x33
dounmount(c7a47808,80000,c7a47808,1,c4d59c7c) at dounmount+0x1a8
vfs_umountall_callback(c7a47808,0,c0416534,0,1) at vfs_umountall_callback+0x13
mountlist_scan(c01eacc5,0,6) at mountlist_scan+0xe8
vfs_unmountall(0,0,3,c019dece,c0d66140) at vfs_unmountall+0x11
boot(4008,c4d59d34,c0305d25,c4d59cf0,6) at boot+0x21d
sys_reboot(c4d59cf0,6,98b6,0,c7a44f88) at sys_reboot+0x25
syscall2(c4d59d40) at syscall2+0x1ef
Xint0x80_syscall() at Xint0x80_syscall+0x36

#8 Updated by alexh over 4 years ago

Matt,

any update on this? did you figure out why the panic can still happen?

Cheers,
Alex Hornung

#9 Updated by ftigeot almost 3 years ago

  • Status changed from New to Resolved
  • Assignee deleted (0)

Closing due to lack of recent feedback.

#10 Updated by ftigeot almost 3 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF