Project

General

Profile

Actions

Bug #3253

closed

HAMMER2 on NFS server. "(nlock & MTX_MASK) != MTX_MASK"

Added by yellowrabbit2010 over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
11/03/2020
Due date:
% Done:

0%

Estimated time:

Description

Hello,
the client machine writes (not very fast, like 150Kb/s) data to the DragonFlyBSD NFS server with hammer2 partition.
The server dies periodically (3-4 days). These crashes do not depend on the time of continuous work (I tried to reboot several times a day), but rather just a coincidence of some circumstances.
I am using the master branch, radeon graphics driver with hardware acceleration is disabled. Crashes occur both when idle at night and when working during the day.

Here are the last three files, they refer to different commits on the master, of course, but the error is the same.

I have all the vmcore.? and kern.? files and I can put them wherever you specify.

======== kgdb -n 6 =========
Loaded symbols for /boot/kernel/radeonkmsfw_BONAIRE_vce.ko
get_mycpu () at ./machine/thread.h:69
69 __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(
_mycpu__dummy));
(kgdb) bt
#0 _get_mycpu () at ./machine/thread.h:69
#1 panic (
fmt=fmt@entry=0xffffffff80c25920 "assertion \"%s\" failed in %s at %s:%u")
at /usr/src/sys/kern/kern_shutdown.c:868
#2 0xffffffff8066cf23 in _mtx_unlock (mtx=mtx@entry=0xfffff8039daaa040)
at /usr/src/sys/kern/kern_mutex.c:725
#3 0xffffffff809464b0 in mtx_unlock (mtx=0xfffff8039daaa040)
at /usr/src/sys/sys/mutex2.h:358
#4 mtx_lock_temp_release (mtx=0xfffff8039daaa040)
at /usr/src/sys/sys/mutex2.h:491
#5 hammer2_inode_lock_temp_release (ip=0xfffff8039daa9fc0)
at /usr/src/sys/vfs/hammer2/hammer2_inode.c:496
#6 hammer2_igetv (ip=ip@entry=0xfffff8039daa9fc0,
errorp=errorp@entry=0xfffff8038047926c)
at /usr/src/sys/vfs/hammer2/hammer2_inode.c:685
#7 0xffffffff8094fc24 in hammer2_vfs_vget (mp=<optimized out>,
dvp=dvp@entry=0x0, ino=<optimized out>, ino@entry=14263250,
vpp=vpp@entry=0xfffff80380479388)
at /usr/src/sys/vfs/hammer2/hammer2_vfsops.c:1911
#8 0xffffffff80950596 in hammer2_vfs_fhtovp (mp=<optimized out>,
rootvp=<optimized out>, fhp=<optimized out>, vpp=0xfffff80380479388)
at /usr/src/sys/vfs/hammer2/hammer2_vfsops.c:2871
#9 0xffffffff807fc64f in nfsrv_fhtovp (fhp=fhp@entry=0xfffff803804793c8,
lockflag=lockflag@entry=1, mpp=mpp@entry=0xfffff80380479390,
vpp=vpp@entry=0xfffff80380479388, cred=cred@entry=0xfffff8039b2614e0,
slp=slp@entry=0xfffff8038057f240, nam=0xfffff80157724070,
rdonlyp=0xfffff80380479384, kerbflag=0, pubflag=1)
at /usr/src/sys/vfs/nfs/nfs_subs.c:1247
#10 0xffffffff807ef326 in nfsrv_write (nfsd=0xfffff8039b261420,
slp=0xfffff8038057f240, td=<optimized out>, mrq=0xfffff80380479708)
at /usr/src/sys/vfs/nfs/nfs_serv.c:1089
#11 0xffffffff807fe23f in nfssvc_nfsd (td=<optimized out>,
argp=<optimized out>, nsd=<optimized out>)
at /usr/src/sys/vfs/nfs/nfs_syscalls.c:678
#12 sys_nfssvc (sysmsg=<optimized out>, uap=<optimized out>)
at /usr/src/sys/vfs/nfs/nfs_syscalls.c:300
#13 0xffffffff80bc347e in syscall2 (frame=0xfffff803804799f8)
at /usr/src/sys/platform/pc64/x86_64/trap.c:1284
#14 0xffffffff80b8664d in Xfast_syscall ()
at /usr/src/sys/platform/pc64/x86_64/exception.S:448
#15 0x000000000000002b in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(kgdb) ====================

======== kgdb -n 4 =========
Loaded symbols for /boot/kernel/radeonkmsfw_BONAIRE_vce.ko
get_mycpu () at ./machine/thread.h:69
69 __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(
_mycpu__dummy));
(kgdb)
(kgdb) bt
#0 _get_mycpu () at ./machine/thread.h:69
#1 panic (
fmt=fmt@entry=0xffffffff80c5ec30 "assertion \"%s\" failed in %s at %s:%u")
at /usr/src/sys/kern/kern_shutdown.c:868
#2 0xffffffff8066b973 in _mtx_unlock (mtx=mtx@entry=0xfffff8031a395a40)
at /usr/src/sys/kern/kern_mutex.c:725
#3 0xffffffff809447c0 in mtx_unlock (mtx=0xfffff8031a395a40)
at /usr/src/sys/sys/mutex2.h:358
#4 mtx_lock_temp_release (mtx=0xfffff8031a395a40)
at /usr/src/sys/sys/mutex2.h:491
#5 hammer2_inode_lock_temp_release (ip=0xfffff8031a3959c0)
at /usr/src/sys/vfs/hammer2/hammer2_inode.c:496
#6 hammer2_igetv (ip=ip@entry=0xfffff8031a3959c0,
errorp=errorp@entry=0xfffff8031312e22c)
at /usr/src/sys/vfs/hammer2/hammer2_inode.c:685
#7 0xffffffff8094de84 in hammer2_vfs_vget (mp=<optimized out>,
dvp=dvp@entry=0x0, ino=<optimized out>, ino@entry=14263012,
vpp=vpp@entry=0xfffff8031312e348)
at /usr/src/sys/vfs/hammer2/hammer2_vfsops.c:1922
#8 0xffffffff8094e7c6 in hammer2_vfs_fhtovp (mp=<optimized out>,
rootvp=<optimized out>, fhp=<optimized out>, vpp=0xfffff8031312e348)
at /usr/src/sys/vfs/hammer2/hammer2_vfsops.c:2895
#9 0xffffffff807fa90f in nfsrv_fhtovp (fhp=fhp@entry=0xfffff8031312e388,
lockflag=lockflag@entry=1, mpp=mpp@entry=0xfffff8031312e350,
vpp=vpp@entry=0xfffff8031312e348, cred=cred@entry=0xfffff80325a10660,
slp=slp@entry=0xfffff8031316fac0, nam=0xfffff800e80a4130,
rdonlyp=0xfffff8031312e344, kerbflag=0, pubflag=1)
at /usr/src/sys/vfs/nfs/nfs_subs.c:1232
#10 0xffffffff807ed7d6 in nfsrv_write (nfsd=0xfffff80325a105a0,
slp=0xfffff8031316fac0, td=<optimized out>, mrq=0xfffff8031312e6c8)
at /usr/src/sys/vfs/nfs/nfs_serv.c:1089
#11 0xffffffff807fc51f in nfssvc_nfsd (td=<optimized out>,
argp=<optimized out>, nsd=<optimized out>)
at /usr/src/sys/vfs/nfs/nfs_syscalls.c:678
#12 sys_nfssvc (uap=<optimized out>) at /usr/src/sys/vfs/nfs/nfs_syscalls.c:300
#13 0xffffffff80babbeb in syscall2 (frame=0xfffff8031312e9f8)
at /usr/src/sys/platform/pc64/x86_64/trap.c:1319
#14 0xffffffff80b85d5d in Xfast_syscall ()
at /usr/src/sys/platform/pc64/x86_64/exception.S:450
#15 0x000000000000002b in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(kgdb) =============================


Files

core.txt.4 (454 KB) core.txt.4 yellowrabbit2010, 11/03/2020 02:41 PM
core.txt.5 (489 KB) core.txt.5 yellowrabbit2010, 11/03/2020 02:41 PM
core.txt.6 (431 KB) core.txt.6 yellowrabbit2010, 11/03/2020 02:42 PM
ls-crash.png (112 KB) ls-crash.png yellowrabbit2010, 11/03/2020 02:42 PM
core.txt.7 (571 KB) core.txt.7 yellowrabbit2010, 11/18/2020 01:07 PM
Actions #1

Updated by yellowrabbit2010 over 3 years ago

Master at commit 98dd7c40f68023d0feff672c7444e9c6bec0deb2
Diagnostics already looks different, but this is the same problem.

====== kgdb -n 7 =========
get_mycpu () at ./machine/thread.h:69
69 __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(
_mycpu__dummy));
(kgdb) bt
#0 _get_mycpu () at ./machine/thread.h:69
#1 panic (
fmt=fmt@entry=0xffffffff80c27220 "assertion \"%s\" failed in %s at %s:%u")
at /usr/src/sys/kern/kern_shutdown.c:868
#2 0xffffffff809485ea in mtx_unlock (mtx=<optimized out>)
at /usr/src/sys/sys/mutex2.h:348
#3 mtx_unlock (mtx=<optimized out>) at /usr/src/sys/sys/mutex2.h:343
#4 mtx_lock_temp_release (mtx=<optimized out>)
at /usr/src/sys/sys/mutex2.h:491
#5 hammer2_inode_lock_temp_release (ip=<optimized out>)
at /usr/src/sys/vfs/hammer2/hammer2_inode.c:496
#6 hammer2_igetv (ip=ip@entry=0xfffff803a01b3ec0,
errorp=errorp@entry=0xfffff801492eb26c)
at /usr/src/sys/vfs/hammer2/hammer2_inode.c:685
#7 0xffffffff809517f4 in hammer2_vfs_vget (mp=<optimized out>,
dvp=dvp@entry=0x0, ino=<optimized out>, ino@entry=14263586,
vpp=vpp@entry=0xfffff801492eb388)
at /usr/src/sys/vfs/hammer2/hammer2_vfsops.c:1911
#8 0xffffffff80952166 in hammer2_vfs_fhtovp (mp=<optimized out>,
rootvp=<optimized out>, fhp=<optimized out>, vpp=0xfffff801492eb388)
at /usr/src/sys/vfs/hammer2/hammer2_vfsops.c:2871
#9 0xffffffff807fe21f in nfsrv_fhtovp (fhp=fhp@entry=0xfffff801492eb3c8,
lockflag=lockflag@entry=1, mpp=mpp@entry=0xfffff801492eb390,
vpp=vpp@entry=0xfffff801492eb388, cred=cred@entry=0xfffff803a8caa660,
slp=slp@entry=0xfffff8015f9577c0, nam=0xfffff801572c55b0,
rdonlyp=0xfffff801492eb384, kerbflag=0, pubflag=1)
at /usr/src/sys/vfs/nfs/nfs_subs.c:1247
#10 0xffffffff807f0ef6 in nfsrv_write (nfsd=0xfffff803a8caa5a0,
slp=0xfffff8015f9577c0, td=<optimized out>, mrq=0xfffff801492eb708)
at /usr/src/sys/vfs/nfs/nfs_serv.c:1089
#11 0xffffffff807ffe0f in nfssvc_nfsd (td=<optimized out>,
argp=<optimized out>, nsd=<optimized out>)
at /usr/src/sys/vfs/nfs/nfs_syscalls.c:678
#12 sys_nfssvc (sysmsg=<optimized out>, uap=<optimized out>)
at /usr/src/sys/vfs/nfs/nfs_syscalls.c:300
#13 0xffffffff80bc4e0e in syscall2 (frame=0xfffff801492eb9f8)
at /usr/src/sys/platform/pc64/x86_64/trap.c:1284
#14 0xffffffff80b87fdd in Xfast_syscall ()
at /usr/src/sys/platform/pc64/x86_64/exception.S:448
#15 0x000000000000002b in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?) ===========

Actions #2

Updated by yellowrabbit2010 over 3 years ago

  • Status changed from New to Closed

I have not seen the crash for 20 days.

Actions

Also available in: Atom PDF