Project

General

Profile

Actions
Copy link

Bug #1777

closed

vn panic: Freeing already free credential! 0xdef403f8

Added by sjg over 14 years ago. Updated over 14 years ago.

Status:
Closed
Priority:
High
Assignee:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:

Description

#0 _get_mycpu (di=0xc06ee760) at ./machine/thread.h:83
#1 md_dumpsys (di=0xc06ee760) at /usr/src/sys/platform/pc32/i386/dump_machdep.c:263
#2 0xc0320609 in dumpsys () at /usr/src/sys/kern/kern_shutdown.c:838
#3 0xc0320bdb in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:387
#4 0xc032113e in panic (fmt=0xc062402c "Freeing already free credential! %p")
at /usr/src/sys/kern/kern_shutdown.c:744
#5 0xc0319313 in crfree (cr=0xdef403f8) at /usr/src/sys/kern/kern_prot.c:1066
#6 0xdf244e94 in vnsetcred (vn=0xdef400d0, cred=0xdf030e48) at
/usr/src-misc/sys/dev/disk/vn/vn.c:701
#7 0xdf245703 in vniocattach_file (ap=0xdf24ab2c) at
/usr/src-misc/sys/dev/disk/vn/vn.c:567
#8 vnioctl (ap=0xdf24ab2c) at /usr/src-misc/sys/dev/disk/vn/vn.c:455
#9 0xc03051ce in dev_dioctl (dev=0xdaa45c08, cmd=3222029824, data=0xdf24ac10
"\207\370\277\277", fflag=1,
cred=0xdf030e48, msg=0x0) at /usr/src/sys/kern/kern_device.c:175
#10 0xc033d198 in diskioctl (ap=0xdf24ab84) at /usr/src/sys/kern/subr_disk.c:875
#11 0xc03051ce in dev_dioctl (dev=0xdaa45cd8, cmd=3222029824, data=0xdf24ac10
"\207\370\277\277", fflag=1,
cred=0xdf030e48, msg=0xdf24acf0) at /usr/src/sys/kern/kern_device.c:175
#12 0xc04b6544 in devfs_specf_ioctl (fp=0xda5161b8, com=3222029824,
data=0xdf24ac10 "\207\370\277\277",
ucred=0xdf030e48, msg=0xdf24acf0) at /usr/src/sys/vfs/devfs/devfs_vnops.c:1488
#13 0xc0346afe in fo_ioctl (fd=3, com=3222029824, uspc_data=0xbfbff69c <Address
0xbfbff69c out of bounds>, map=0x0,
msg=0xdf24acf0) at /usr/src/sys/sys/file2.h:88
#14 mapped_ioctl (fd=3, com=3222029824, uspc_data=0xbfbff69c <Address 0xbfbff69c
out of bounds>, map=0x0,
msg=0xdf24acf0) at /usr/src/sys/kern/sys_generic.c:705
#15 0xc0347051 in sys_ioctl (uap=0xdf24acf0) at /usr/src/sys/kern/sys_generic.c:527
#16 0xc0582322 in syscall2 (frame=0xdf24ad40) at
/usr/src/sys/platform/pc32/i386/trap.c:1319
#17 0xc056b176 in Xint0x80_syscall () at
/usr/src/sys/platform/pc32/i386/exception.s:876
#18 0x0000001f in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

This happens upon executing

vnconfig -c -s labels -S 2g -T vn0 /var/vkernel/rootimg.01

as per the vkernel manpage, /var/vkernel is a symlink to /home/var.vkernel and
the box is running HAMMER

ROOT on / (hammer, local)
devfs on /dev (devfs, local)
/dev/serno/9TE03QNJ.s3a on /boot (ufs, local)
/pfs/@-1:00001 on /var (null, local)
/pfs/@-1:00002 on /tmp (null, local)
/pfs/@-1:00003 on /usr (null, local)
/pfs/@-1:00004 on /home (null, local)
/pfs/@-1:00005 on /usr/obj (null, local)
/pfs/@-1:00006 on /var/crash (null, local)
/pfs/@@-1:00007 on /var/tmp (null, local)
procfs on /proc (procfs, local)

kern/vmcore are inbound to leaf

Files

Download all files
vn.patch (370 Bytes) vn.patch josepht, 06/05/2010 04:31 AM
vn.patch (370 Bytes) vn.patch josepht, 06/05/2010 10:15 PM
vn.patch (597 Bytes) vn.patch josepht, 06/05/2010 10:19 PM
Actions
Copy link
 #1

Updated by josepht over 14 years ago

I'm not sure this is the correct approach, but the attached patch
should fix this issue.

Joe

Actions
Copy link
 #2

Updated by dillon over 14 years ago

:I'm not sure this is the correct approach, but the attached patch
:should fix this issue.
:
:Joe
:...
:
:diff --git a/sys/dev/disk/vn/vn.c b/sys/dev/disk/vn/vn.c
:index 95bcf44..9f45281 100644
:--- a/sys/dev/disk/vn/vn.c
:+++ b/sys/dev/disk/vn/vn.c
:@ -697,7 +697,7 @ vnsetcred(struct vn_softc *vn, struct ucred *cred)
: * Set credits in our softc
: */
:
:- if (vn->sc_cred)
:+ if (vn->sc_cred && vn->sc_cred->cr_ref > 0)
: crfree(vn->sc_cred);
: vn->sc_cred = crdup(cred);

That can't be right.  If vn->sc_cred is set at all there must be a
    ref on it.  If there isn't something is leaking a crfree() without
    clearing it.
-Matt
                    Matthew Dillon 
                    &lt;&gt;
Actions
Copy link
 #3

Updated by josepht over 14 years ago

On Fri, Jun 04, 2010 at 09:45:37PM -0700, Matthew Dillon wrote:

:I'm not sure this is the correct approach, but the attached patch
:should fix this issue.
:
:Joe
:...
:
:diff --git a/sys/dev/disk/vn/vn.c b/sys/dev/disk/vn/vn.c
:index 95bcf44..9f45281 100644
:--- a/sys/dev/disk/vn/vn.c
:+++ b/sys/dev/disk/vn/vn.c
:@ -697,7 +697,7 @ vnsetcred(struct vn_softc *vn, struct ucred *cred)
: * Set credits in our softc
: */
:
:- if (vn->sc_cred)
:+ if (vn->sc_cred && vn->sc_cred->cr_ref > 0)
: crfree(vn->sc_cred);
: vn->sc_cred = crdup(cred);

That can't be right. If vn->sc_cred is set at all there must be a
ref on it. If there isn't something is leaking a crfree() without
clearing it.

I've been looking at the backtrace and in frame 6 vn->sc_cred is 0x00
but cr in frame 5 is non-NULL and appears to be corrupted.

Joe

Actions
Copy link
 #4

Updated by dillon over 14 years ago

:> : * Set credits in our softc
:> : */
:> :
:> :- if (vn->sc_cred)
:> :+ if (vn->sc_cred && vn->sc_cred->cr_ref > 0)
:> : crfree(vn->sc_cred);
:> : vn->sc_cred = crdup(cred);
:>
:> That can't be right. If vn->sc_cred is set at all there must be a
:> ref on it. If there isn't something is leaking a crfree() without
:> clearing it.
:
:I've been looking at the backtrace and in frame 6 vn->sc_cred is 0x00
:but cr in frame 5 is non-NULL and appears to be corrupted.
:
:Joe

Check the rest of the vn structure and see if it is corrupted.  If
   it isn't then what is probably happening is some other subsystem
   (possibly part of the call path into vn, before it gets to vn) might
   be blowing up the credential.  It might not be VN's fault.
-Matt
                    Matthew Dillon 
                    &lt;&gt;
Actions
Copy link
 #5

Updated by josepht over 14 years ago

On Sat, Jun 05, 2010 at 09:00:51AM -0700, Matthew Dillon wrote:

:> : * Set credits in our softc
:> : */
:> :
:> :- if (vn->sc_cred)
:> :+ if (vn->sc_cred && vn->sc_cred->cr_ref > 0)
:> : crfree(vn->sc_cred);
:> : vn->sc_cred = crdup(cred);
:>
:> That can't be right. If vn->sc_cred is set at all there must be a
:> ref on it. If there isn't something is leaking a crfree() without
:> clearing it.
:
:I've been looking at the backtrace and in frame 6 vn->sc_cred is 0x00
:but cr in frame 5 is non-NULL and appears to be corrupted.
:
:Joe

Check the rest of the vn structure and see if it is corrupted. If
it isn't then what is probably happening is some other subsystem
(possibly part of the call path into vn, before it gets to vn) might
be blowing up the credential. It might not be VN's fault.

The vn structure appears to NOT be corrupt. Seems like a race if
vn->sc_cred goes from NULL to non-NULL between frame 6 and 5. Could
this be related to my recent link_elf change (setting p to proc0 if p
is NULL)?

Joe

Actions
Copy link
 #6

Updated by dillon over 14 years ago

:The vn structure appears to NOT be corrupt. Seems like a race if
:vn->sc_cred goes from NULL to non-NULL between frame 6 and 5. Could
:this be related to my recent link_elf change (setting p to proc0 if p
:is NULL)?
:
:Joe

Hmm.  If devfs is racing access to that field then try changing the
    vnsetcred() code around a bit to this:
struct ucred *ocred
    ...
/*
 * Set credits in our softc
         */
        ocred = vn->sc_cred;
        vn->sc_cred = crdup(cred);
        if (ocred)
                crfree(ocred);
If the race is more involved then that (assuming it is a race), we
    may have to call crhold() and friends whenever we use vn->sc_cred.
-Matt
                    Matthew Dillon 
                    &lt;&gt;
Actions
Copy link
 #7

Updated by josepht over 14 years ago

On Sat, Jun 05, 2010 at 12:26:47PM -0700, Matthew Dillon wrote:

:The vn structure appears to NOT be corrupt. Seems like a race if
:vn->sc_cred goes from NULL to non-NULL between frame 6 and 5. Could
:this be related to my recent link_elf change (setting p to proc0 if p
:is NULL)?
:
:Joe

Hmm. If devfs is racing access to that field then try changing the
vnsetcred() code around a bit to this:

struct ucred *ocred
...

/*
  • Set credits in our softc
    */
    ocred = vn->sc_cred;
    vn->sc_cred = crdup(cred);
    if (ocred)
    crfree(ocred);

If the race is more involved then that (assuming it is a race), we
may have to call crhold() and friends whenever we use vn->sc_cred.

Sam,

Please try this patch. I was able to vnconfig a file following the
vkernel manpage without a panic. This isn't likely a long term
solution but will hopefully allow you to proceed with your GSoC work
in a vkernel. If you see this panic again let me know and I'll work
on implementing the crhold() et al. calls as Matt suggested.

Thanks,
Joe

Actions
Copy link
 #8

Updated by josepht over 14 years ago

On Sat, Jun 05, 2010 at 06:13:29PM -0400, Joe Talbott wrote:

Please try this patch. I was able to vnconfig a file following the
vkernel manpage without a panic. This isn't likely a long term
solution but will hopefully allow you to proceed with your GSoC work
in a vkernel. If you see this panic again let me know and I'll work
on implementing the crhold() et al. calls as Matt suggested.

Forgot to copy the patch to my mail server. This is the one I
intended to send.

Thanks,
Joe

Actions
Copy link
 #9

Updated by sjg over 14 years ago

Think this was entirely related to slightly out of sync kernel/modules.

Actions
Copy link

Also available in: Atom PDF