DragonFlyBSD

:Hi.
:I managed to trigger this panic while trying to find out how to
:reliably reproduce another panic (lockmgr: non-zero exclusive count).
:Just issue the following command on a machine running recent -DEVELOPMENT
:(either on i386 or x86_64):
:
:$ grep -r --mmap SomeString /usr/pkgsrc
:
:If this won't trigger the panic but you need kern/vmcore, please let me
:know and I can upload mine to my leaf account.
:
:Best Regards,
:YONETANI Tomokazu.

How much ram do you have in that machine (so I can reproduce the test).
I have a feeling that cache cycling of VM pages might tend to trigger the (second) panic more often.

non-zero exclusive counts usually mean an extra lock release or a missing lock acquisition for a lockmgr lock. It can be a little
trickly if it is a vnode lock since a completely unrelated bit of code might be causing the situation and then later on a perfectly
fine piece of code triggers it.

-Matt
Matthew Dillon
<dillon@backplane.com>

:Hi.
:I managed to trigger this panic while trying to find out how to
:reliably reproduce another panic (lockmgr: non-zero exclusive count).
:Just issue the following command on a machine running recent -DEVELOPMENT
:(either on i386 or x86_64):
:
:$ grep -r --mmap SomeString /usr/pkgsrc
:
:If this won't trigger the panic but you need kern/vmcore, please let me
:know and I can upload mine to my leaf account.
:
:Best Regards,
:YONETANI Tomokazu.

Ah!! I reproduced it easily w/ 8G ram on my 64-bit test box!

-Matt

::I managed to trigger this panic while trying to find out how to
::reliably reproduce another panic (lockmgr: non-zero exclusive count).
::Just issue the following command on a machine running recent -DEVELOPMENT
::(either on i386 or x86_64):
::
::$ grep -r --mmap SomeString /usr/pkgsrc

Ok, this grep caused a 'lockmgr: locking against myself' panic
due to the grep code doing a read() system call INSIDE the same
mmap()'d file, causing the uiomove/copyout call to overlap the
buffer cache buffer being held by the read().

Hmm. It looks like it hit a read-ahead-mark and called readrest
on a valid page. This should be an allowed operation but I'm going
to think a bit on how to fix it.. I'll probably have to add a hold_count
field to the buffer cache buf structure.

You should be able to temporarily work around the bug by commenting
out the PG_RAM test on line 1083 of vm/vm_fault.c:

#if 0
    if (fs->m->flags & PG_RAM) {
        if (debug_cluster)
            kprintf("R");
        vm_page_flag_clear(fs->m, PG_RAM);
        goto readrest;
    }
#endif

This code isn't the bug, but it is probably triggering the bug. If
that fixes this secondary issue for you you can go back to finding
the primary panic you were trying to track down and I will continue
looking into this one figuring out how to fix it properly.

-Matt

On Sun, Dec 26, 2010 at 11:09:17AM -0800, Matthew Dillon wrote:

non-zero exclusive counts usually mean an extra lock release or a
missing lock acquisition for a lockmgr lock. It can be a little
trickly if it is a vnode lock since a completely unrelated bit of
code might be causing the situation and then later on a perfectly
fine piece of code triggers it.

As for non-zero exclusive count panic, it's odd because although
lockmgr() triggered the non-zero exclusive count panic, kgdb shows
that lkp->lk_exclusivecount == 0 (shown below). Usually the panic is
followed by several more panics even during the dump routine is running,
but luckily I managed to save the crash dump and put it on my leaf account
as ~y0netan1/crash/{kern,vmcore}.21 (these are saved yesterday, so this
kernel doesn't have the patch you suggested in another message yet).

:
#14 0xffffffff802a6cc4 in panic (
    fmt=0xffffffff80508548 "lockmgr: non-zero exclusive count")
    at /usr/obj/src.test/sys/kern/kern_shutdown.c:783
#15 0xffffffff80298497 in lockmgr (lkp=0xffffffe060e72878,
    flags=<value optimized out>) at /usr/obj/src.test/sys/kern/kern_lock.c:369
#16 0xffffffff8042e387 in vm_map_find (map=0xffffffe060e72800, object=0x0,
    offset=0, addr=0xffffffe060410ac0, length=32768, align=4096, fitit=1,
    maptype=1 '\001', prot=3 '\003', max=7 '\a', cow=0)
    at /usr/obj/src.test/sys/vm/vm_map.c:1201
#17 0xffffffff8043175a in vm_mmap (map=0xffffffe060e72800,
    addr=0xffffffe060410ac0, size=<value optimized out>, prot=3 '\003',
    maxprot=7 '\a', flags=<value optimized out>, handle=0x0, foff=0)
    at /usr/obj/src.test/sys/vm/vm_mmap.c:1362
#18 0xffffffff80431d61 in kern_mmap (vms=0xffffffe060e72800,
    uaddr=<value optimized out>, ulen=<value optimized out>,
    uprot=<value optimized out>, uflags=<value optimized out>, fd=-1, upos=0,
    res=0xffffffe060410b58) at /usr/obj/src.test/sys/vm/vm_mmap.c:397
                :
(kgdb) fr 15
#15 0xffffffff80298497 in lockmgr (lkp=0xffffffe060e72878,
    flags=<value optimized out>) at /usr/obj/src.test/sys/kern/kern_lock.c:369
369                             panic("lockmgr: non-zero exclusive count");
(kgdb) p *lkp
$1 = {lk_spinlock = {lock = 0}, lk_flags = 3146240, lk_sharecount = 1,
  lk_waitcount = 3, lk_exclusivecount = 0, lk_unused1 = 0,
  lk_wmesg = 0xffffffff80537872 "thrd_sleep", lk_timo = 0,
  lk_lockholder = 0xffffffffffffffff}

The known good kernel is built from the source as of 0b38f, and known
bad kernel is from 2a418. Sometimes it was even possible to trigger
the panic by just opening the older kernel and the dump with kgdb, or
running buildkernel, but it's not consistent among ``bad'' kernels.

Here's a bug fix for the second bug... the recursive lock --mmap issue.
Please test it (with the originally suggested #if 0 workaround removed).
This fix is for HAMMER, if it works I'll do the same thing w/ NFS and UFS.

http://apollo.backplane.com/DFlyMisc/buf01.patch

We still have a recursive lock or deadlock possibility with write()'s
which isn't quite so easy to fix. I'm not quite sure what to do there.

I'll review the bisect range for the first bug and see if I can find
where the missing lock pairing is.

-Matt

:As for non-zero exclusive count panic, it's odd because although
:lockmgr() triggered the non-zero exclusive count panic, kgdb shows
:that lkp->lk_exclusivecount == 0 (shown below). Usually the panic is
:...

Hmm. The inconsistency is probalby due to the secondary panics.
Run 'dmesg -M vmcore.21 -N kern.21', two panics occured at the same
time and their kprintf output got mixed in together. Once a panic
occurs numerous subsystems will allow locks through whether they can
acquire them or not.

Now on the non-zero exclusive count issue. I don't see any mismatched
lock/unlock operations but I suspect it may be the TDF_DEADLKTREAT
code allowing an exclusive lock to worm its way inbetween an
EXCLUPGRADE request (because it doesn't block on LK_WANT_UPGRADE in
that TDF_DEADLKTREAT case).

The way to test this is to comment out the setting of TDF_DEADLKTREAT
in kern/kern_subr.c. There's just one place where it happens.

If that turns out to fix the problem then I think we may be forced
to get rid of the shared->exclusive upgrade code entirely or restrict
it to less sophisticated cases. The whole shared->exclusive priority
mechanism in lockmgr has had deadlock issues for a long time.

-Matt

I just got one of these non-zero exclusive count panics while doing a 'git
grep' in /usr/pkgsrc/. Uptime for the machine was less than 2 days, so I'm
going to try to comment out TDF_DEADLKTREAT and see if I can duplicate it over
the next few days. I'm uploading the vmcore to
http://www.theshell.com/~pavalos/crash1.tar.xz.

--Peter

On Tue, Dec 28, 2010 at 12:37:23AM +0000, Peter Avalos (via DragonFly issue tracker) wrote:

Peter Avalos <peter@theshell.com> added the comment:

I just got one of these non-zero exclusive count panics while doing a 'git
grep' in /usr/pkgsrc/. Uptime for the machine was less than 2 days, so I'm
going to try to comment out TDF_DEADLKTREAT and see if I can duplicate it over
the next few days. I'm uploading the vmcore to
http://www.theshell.com/~pavalos/crash1.tar.xz.

My vmcore.21 was also from the panic triggered by `git grep' and
that was why I tried `grep -r --mmap'.

However, even with buf01.patch applied and with the line setting
TDF_DEADLKTREAT removed, the following command can still panic
the system (you may have to run this command a few times, and/or
in a few different repositories depending on the size of the RAM
on the machine). The patches are applied over 1a92ebbba.

$ cd /usr/src
$ PAGER='dd of=/dev/null bs=4m' git grep LK_

This PC is running a 32-bit kernel and it has 2G bytes of RAM.

On Tue, Dec 28, 2010 at 04:45:59PM +0900, YONETANI Tomokazu wrote:

However, even with buf01.patch applied and with the line setting
TDF_DEADLKTREAT removed, the following command can still panic
the system (you may have to run this command a few times, and/or
in a few different repositories depending on the size of the RAM
on the machine). The patches are applied over 1a92ebbba.

I can also verify that I'm still experiencing panics with
TDF_DEADLKTREAT removed. I wasn't able to get a new vmcore, but the one
I reported about earlier is done uploading.

--Peter

On Tue, Dec 28, 2010 at 02:20:19AM -1000, Peter Avalos wrote:

On Tue, Dec 28, 2010 at 04:45:59PM +0900, YONETANI Tomokazu wrote:

However, even with buf01.patch applied and with the line setting
TDF_DEADLKTREAT removed, the following command can still panic
the system (you may have to run this command a few times, and/or
in a few different repositories depending on the size of the RAM
on the machine). The patches are applied over 1a92ebbba.

I can also verify that I'm still experiencing panics with
TDF_DEADLKTREAT removed. I wasn't able to get a new vmcore, but the one
I reported about earlier is done uploading.

Ok, 2a418930 is the first commit which triggers non-zero exclusive
count panic. 4d233d10 won't crash with `git grep' test. I don't
have e1000, so it shouldn't affect any of my PC's.

On Tue, Dec 28, 2010 at 02:20:19AM -1000, Peter Avalos wrote:

I can also verify that I'm still experiencing panics with
TDF_DEADLKTREAT removed. I wasn't able to get a new vmcore, but the one
I reported about earlier is done uploading.

I just got another one with TDF_DEADLKTREAT removed. It can be
downloaded from: http://www.theshell.com/~pavalos/crash2.tar.xz

--Peter

Hmm. Ok, and I think my other patch might have issues as well. It
is still possible for a buffer cache buffer to get invalidated while
held, though for the moment I'm going to run wieth it until I can
figure out the non-zero exclusive count issue.

-Matt

Ok, I think I fixed the non-zero exclusive count panic, please try
with the latest master (I pushed it).

It looks like a shared->exclusive upgrade can steal an exclusive lock
out from under an exclusive lock request. The exclusive lock request
uses two acquire() operations internally and the upgrade can slip in
inbetween them.

The recursive lock that my hold/drop patch addresses is still an issue.
The hold/drop patch fixes it but might have other problems (the buffer
can still be invalidated while hold). I'm still looking into that.

Please test.

-Matt

Oh, put TDF_DEADLKTREAT back in guys. That was indeed one place where
an exclusive lock could wiggle its way in but the shared->exclusive
upgrade also allows the same wiggling so my solution was to check for
the wiggle instead of remove the wiggle.

-Matt

On Wed, Dec 29, 2010 at 12:49:12AM -0800, Matthew Dillon wrote:

Oh, put TDF_DEADLKTREAT back in guys. That was indeed one place where
an exclusive lock could wiggle its way in but the shared->exclusive
upgrade also allows the same wiggling so my solution was to check for
the wiggle instead of remove the wiggle.

I've been running kernels(32bit and 64bit), built from dbe02692 with
buf01.patch applied. None of `grep --mmap' or `git grep' tests could
trigger the panic. And so far no panics occurred while running pbulk
for 24 hours.

Thanks.

On Mon, Dec 27, 2010 at 09:21:08AM -0800, Matthew Dillon wrote:

Here's a bug fix for the second bug... the recursive lock --mmap issue.
Please test it (with the originally suggested #if 0 workaround removed).
This fix is for HAMMER, if it works I'll do the same thing w/ NFS and UFS.

fetch http://apollo.backplane.com/DFlyMisc/buf01.patch

Apparently the previous versions (I haven't tried other versions than
2.8-RELEASE yet) are affected too, and the above patch seems to fix it
as well.

Hi,

On 2011/1/4 11:01 AM, YONETANI Tomokazu wrote:

On Mon, Dec 27, 2010 at 09:21:08AM -0800, Matthew Dillon wrote:

Here's a bug fix for the second bug... the recursive lock --mmap issue.
Please test it (with the originally suggested #if 0 workaround removed).
This fix is for HAMMER, if it works I'll do the same thing w/ NFS and UFS.

fetch http://apollo.backplane.com/DFlyMisc/buf01.patch

Apparently the previous versions (I haven't tried other versions than
2.8-RELEASE yet) are affected too, and the above patch seems to fix it
as well.

Matt, This patch fixes 2.8-RELEASE, but it no longer applies cleanly
to master; just resolving the conflict resulted in broken kernel.
The system stops booting in the middle of rc script because of
syntax errors in /etc/rc.subr (actually there isn't, and if I cp this
file to a non-HAMMER filesystem, no syntax error is detected).
Can you rework this patch for master (and if possible commit it too)?

Best Regards,
YONETANI Tomokazu.

:Hi,
:
:Matt, This patch fixes 2.8-RELEASE, but it no longer applies cleanly
:to master; just resolving the conflict resulted in broken kernel.
:The system stops booting in the middle of rc script because of
:syntax errors in /etc/rc.subr (actually there isn't, and if I cp this
:file to a non-HAMMER filesystem, no syntax error is detected).
:Can you rework this patch for master (and if possible commit it too)?
:
:Best Regards,
:YONETANI Tomokazu.

I have to do a little more work before the patch will
work reliably on either master or 2.8. The hold count on
the buffer does not prevent it from being ripped up by
the buffer cache (I need to add a few more places where
b_refs is tested in the buffer daemons).

-Matt
Matthew Dillon
<dillon@backplane.com>

I'm testing a revised version of the patch now and should be able to
commit it in a little bit. Basically the same thing (you may have
gotten that token lock/unlock that broke the direct patch wrong when
you manually patched it), except I adjusted the b_refs check in
getnewbuf() as well to prevent an invalidation race.

-Matt

Ok, I've pushed 283b944850fbb3421a77ef06aa121d066f5c8cad which I
believe should take care of the problem for read()s. write()'s
haven't been addressed yet but lets see how what we have works out
first.

-Matt
Matthew Dillon
<dillon@backplane.com>