DragonFlyBSD

I was able to get crashdump!

vonbraun# kgdb kern.0 vmcore.0
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-dragonfly".
For bug reporting instructions, please see:
<http://bugs.dragonflybsd.org/&gt;...
Reading symbols from /var/crash/kern.0...done.

Unread portion of the kernel message buffer:
panic: devbuf: malloc limit exceeded
mp_lock = 00000000; cpuid = 0
Trace beginning at frame 0xffffffe019c8f710
panic() at panic+0x239
panic() at panic+0x239
kmalloc() at kmalloc+0x134
bus_dma_tag_create() at bus_dma_tag_create+0xb3
usb_block_allocmem() at usb_block_allocmem+0x1b6
usb_allocmem() at usb_allocmem+0x62
ohci_alloc_sitd() at ohci_alloc_sitd+0x46
ohci_device_isoc_transfer() at ohci_device_isoc_transfer+0x251
usbd_transfer() at usbd_transfer+0xd3
uaudio_chan_ptransfer() at uaudio_chan_ptransfer+0x149
uaudio_chan_pintr() at uaudio_chan_pintr+0x65
usb_transfer_complete() at usb_transfer_complete+0x175
ohci_softintr() at ohci_softintr+0x33c
usb_schedsoftintr() at usb_schedsoftintr+0xb
ohci_intr1() at ohci_intr1+0x261
ohci_intr() at ohci_intr+0x76
ithread_handler() at ithread_handler+0x22e
boot() called on cpu#0
Uptime: 10h11m47s
Physical memory: 2025 MB
Dumping 964 MB: 949 933 917 901 885 869 853 837 821 805 (CTRL-C to abort) 789 7
73 757 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 741 725 709 693
677 661 645 629 613 597 581 565 549 533 517 501 485 469 453 437 421 405 389 373
357 341 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37
21 5

Reading symbols from /boot/kernel/snd_uaudio.ko...done.
Loaded symbols for /boot/kernel/snd_uaudio.ko
Reading symbols from /boot/kernel/sound.ko...done.
Loaded symbols for /boot/kernel/sound.ko
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/ahci.ko...done.
Loaded symbols for /boot/kernel/ahci.ko
Reading symbols from /boot/kernel/ehci.ko...done.
Loaded symbols for /boot/kernel/ehci.ko
Reading symbols from /boot/kernel/ipfw.ko...done.
Loaded symbols for /boot/kernel/ipfw.ko
get_mycpu (di=0xffffffff8094ff80) at ./machine/thread.h:73
73 __asm ("movq %%gs:globaldata,%0" : "=r" (gd) : "m"(_mycpu__dummy));
(kgdb) bt
#0 _get_mycpu (di=0xffffffff8094ff80) at ./machine/thread.h:73
#1 md_dumpsys (di=0xffffffff8094ff80)
at /usr/src/sys/platform/pc64/x86_64/dump_machdep.c:262
#2 0xffffffff803650b6 in dumpsys () at /usr/src/sys/kern/kern_shutdown.c:880
#3 0xffffffff803657a5 in boot (howto=-2004318071)
at /usr/src/sys/kern/kern_shutdown.c:387
#4 0xffffffff80365b63 in panic (
fmt=0xffffffff80672827 "%s: malloc limit exceeded")
at /usr/src/sys/kern/kern_shutdown.c:786
#5 0xffffffff8036257c in kmalloc (size=120, type=0xffffffff808371a0,
flags=4866) at /usr/src/sys/kern/kern_slaballoc.c:508
#6 0xffffffff80608ce8 in bus_dma_tag_create (parent=0x0, alignment=32,
boundary=0, lowaddr=4294967295, highaddr=18446744073709551615, filter=0,
filterarg=0x0, maxsize=4096, nsegments=1, maxsegsz=4096, flags=2,
dmat=0xffffffe050b932b8)
at /usr/src/sys/platform/pc64/x86_64/busdma_machdep.c:266
#7 0xffffffff80590116 in usb_block_allocmem (tag=0x0, size=4096, align=32,
dmap=0xffffffe019c8f990) at /usr/src/sys/bus/usb/usb_mem.c:167
#8 0xffffffff80590253 in usb_allocmem (bus=<value optimized out>,
size=<value optimized out>, align=32, p=0xffffffe019c8f990)
at /usr/src/sys/bus/usb/usb_mem.c:236
#9 0xffffffff80589345 in ohci_alloc_sitd (sc=0xffffffe019e36868)
at /usr/src/sys/bus/usb/ohci.c:591
#10 0xffffffff8058a493 in ohci_device_isoc_enter (xfer=0xffffffe03e61d568)
at /usr/src/sys/bus/usb/ohci.c:3386
#11 ohci_device_isoc_transfer (xfer=0xffffffe03e61d568)
at /usr/src/sys/bus/usb/ohci.c:3289
#12 0xffffffff8058ecf6 in usbd_transfer (xfer=0xffffffe03e61d568)
at /usr/src/sys/bus/usb/usbdi.c:310
#13 0xffffffff80bce69b in uaudio_chan_ptransfer (ch=0xffffffe019d00790)
at /usr/src/sys/dev/sound/driver/uaudio/../../../../dev/sound/usb/uaudio.c:2
236
#14 0xffffffff80bcec07 in uaudio_chan_pintr (xfer=<value optimized out>,
priv=0xffffffe019d00938, status=<value optimized out>)
at /usr/src/sys/dev/sound/driver/uaudio/../../../../dev/sound/usb/uaudio.c:2
270
#15 0xffffffff8058ea18 in usb_transfer_complete (xfer=0xffffffe03e61d658)
at /usr/src/sys/bus/usb/usbdi.c:849
#16 0xffffffff80589f84 in ohci_softintr (v=<value optimized out>)
at /usr/src/sys/bus/usb/ohci.c:1508
#17 0xffffffff8058ce56 in usb_schedsoftintr (bus=0x0)
at /usr/src/sys/bus/usb/usb.c:893
#18 0xffffffff80588808 in ohci_intr1 (sc=0xffffffe019e36868)
at /usr/src/sys/bus/usb/ohci.c:1202
#19 0xffffffff80588ab2 in ohci_intr (p=<value optimized out>)
at /usr/src/sys/bus/usb/ohci.c:1126
#20 0xffffffff8033b4c4 in ithread_handler (arg=<value optimized out>)
at /usr/src/sys/kern/kern_intr.c:864
#21 0xffffffff80370e20 in lwkt_deschedule_self (td=0x0)
at /usr/src/sys/kern/lwkt_thread.c:281
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(kgdb) frame 6
#6 0xffffffff80608ce8 in bus_dma_tag_create (parent=0x0, alignment=32,
boundary=0, lowaddr=4294967295, highaddr=18446744073709551615, filter=0,
filterarg=0x0, maxsize=4096, nsegments=1, maxsegsz=4096, flags=2,
dmat=0xffffffe050b932b8)
at /usr/src/sys/platform/pc64/x86_64/busdma_machdep.c:266
266 newtag = kmalloc(sizeof(*newtag), M_DEVBUF, M_INTWAIT | M_ZERO);
(kgdb) q

Do you need kern* and vmcore* files? I will post KTR dump too as soon as possible.

:vasily postnicov <antonbablon@mail.ru> added the comment:
:
:I was able to get crashdump!
:...
:Do you need kern* and vmcore* files? I will post KTR dump too as soon as possible.
:
:----------
:status: unread -> chatting

Excellent!  It is clearly a memory leak but having the core file in-hand
   would be very useful.

If you could put the your kernel & vmcore files up somewhere (and the info
   file too) and email me the URL to get them that would be best.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

There is dump without KTR enabled here:

http://www.filefactory.com/file/b3hc6a8/n/dump.tar.gz

Sorry, I cannot give a direct link.

and email me the URL

Does not this forum send an email to you automatically? Anyway I am going to
send it to dillon at backplane.com.

I recompiled my kernel with KTR and KTR_ALL options, so I will give new dump
with KTR info as soon as possible.

Some debug messages (I got them while playing in Battle of Wesnoth):

usb0: isoc TD alloc failed // (and sound disappears)

Then, I sent SIGKILL to wesnoth and got endless loop of "ohci_device_isoc_abort:
early return" messages. System, surely, did not respond on anything except
hardware reset.

I do not completely sure that it is connected somehow with previous crashes
(In DragonFlyBSD 2.6 there were no endless loops, in 2.8 loops happen more often
than "normal" crashes), but all works perfectly without using USB sound card
anyway. Maybe there is a bug in OHCI driver?

Could you grab the output of vmstat -m from a system that playing music but
going to crash?

Thanks,
-- vs

Could you grab the output of vmstat -m from a system that playing music but

going to crash?

OK. Here:

Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit
IpFw/IpAcct 94 12K 0K 200838K 926 0 0
HAMMER-inodes 3363 2943K 0K134217724K 3568 0 0
HAMMER-others 3000 565K 0K 200838K 35960 0 0
acpisem 15 1K 0K 200838K 15 0 0
acpidev 95 4K 0K 200838K 95 0 0
acpitask 0 0K 0K 200838K 3 0 0
acpica 257 14K 0K 200838K 91636 0 0
acpipwr 1 1K 0K 200838K 1 0 0
mixer 1 1K 0K 200838K 1 0 0
ratefeed 2 21K 0K 200838K 32 0 0
fmtfeed 0 0K 0K 200838K 56 0 0
feeder 114 4K 0K 200838K 815 0 0
CAM SIM 8 2K 0K 200838K 8 0 0
atkbddev 2 1K 0K 200838K 2 0 0
objcache 53 8K 0K 200838K 53 0 0
objcache magazine 576 323K 0K 200838K 576 0 0
lockf 8 1K 0K 200838K 4496 0 0
ktr 1 480K 0K 200838K 1 0 0
legacydrv 2 1K 0K 200838K 2 0 0
nexusdev 3 1K 0K 200838K 3 0 0
lwbuf 27 1K 0K 200838K 27 0 0
syscons 46 187K 0K 200838K 46 0 0
crypto 1 1K 0K 200838K 1 0 0
propstng 1355 53K 0K 200838K 1355 0 0
prop string 1355 11K 0K 200838K 1438 0 0
propnmbr 1617 102K 0K 200838K 1617 0 0
pdict16 15 2K 0K 200838K 15 0 0
propdict 666 125K 0K 200838K 666 0 0
prop dictionary 666 167K 0K 200838K 703 0 0
kbdmux 6 8K 0K 200838K 6 0 0
isadev 26 2K 0K 200838K 26 0 0
USB 5978 516K 0K 200838K 5993 0 0
USBdev 14 10K 0K 200838K 37 0 0
ZONE 7 1K 0K 200838K 7 0 0
atexit 1 1K 0K 200838K 1 0 0
vmspace 61 31K 0K 200838K 61 0 0
proc-args 39 2K 0K 200838K 1539 0 0
exec-args 4 1040K 0K 200838K 4 0 0
device-mapped pages 7611 952K 0K 200838K 7611 0 0
HAMMER-mount 1 68K 0K 200838K 1 0 0
devfs 3768 590K 0K 200838K 6545 0 0
kqueue 0 0K 0K 200838K 492 0 0
UFS dirhash 12 2K 0K 200838K 12 0 0
UFS mount 3 5K 0K 200838K 3 0 0
UFS ihash 1 2048K 0K 200838K 1 0 0
FFS node 44 14K 0K134217724K 44 0 0
pagedep 1 256K 0K 200838K 1 0 0
inodedep 1 2048K 0K 200838K 1 0 0
newblk 1 1K 0K 200838K 1 0 0
p1003.1b 1 1K 0K 200838K 1 0 0
kenv 21 5K 0K 200838K 21 0 0
file desc 50 38K 0K 200838K 1441 0 0
file 177 20K 0K 200838K 18256 0 0
sigio 1 1K 0K 200838K 1 0 0
kld 115 158K 0K 200838K 131 0 0
CAM periph 5 1K 0K 200838K 19 0 0
module 323 30K 0K 200838K 323 0 0
varsym 264 10K 0K 200838K 274 0 0
scsi_cd 0 0K 0K 200838K 1 0 0
NFS daemon 1 1K 0K 200838K 1 0 0
NFS hash 1 2048K 0K 200838K 1 0 0
syncache 2 24K 0K 200838K 2 0 0
tcptemp 25 2K 0K 200838K 25 0 0
sblk 0 0K 0K 200838K 4 0 0
ipq 50 3K 0K 200838K 50 0 0
in_multi 9 1K 0K 200838K 9 0 0
igmp 1 1K 0K 200838K 1 0 0
CCMS 3734 321K 0K 200838K 3734 0 0
firewire 27 28K 0K 200838K 29 0 0
routetbl 325 55K 0K 200838K 379 0 0
faith 1 1K 0K 200838K 1 0 0
clone 3 12K 0K 200838K 3 0 0
ifaddr 75 19K 0K 200838K 77 0 0
ether_multi 30 2K 0K 200838K 30 0 0
BPF 12 9K 0K 200838K 12 0 0
MSDOSFS mount 1 2048K 0K 200838K 1 0 0
NULLFS mount 7 4K 0K 200838K 7 0 0
vnodes 3734 1867K 0K134217724K 3734 0 0
vnodeops 18 11K 0K 200838K 18 0 0
nameibufs 18 18K 0K 200838K 18 0 0
mount 12 10K 0K 200838K 15 0 0
vfscache 9172 8912K 0K 200838K 10600 0 0
BIO buffer 0 0K 0K 200838K 24 0 0
unpcb 40 9K 0K 200838K 90 0 0
fw_xfer 0 0K 0K 200838K 1 0 0
socket 57 27K 0K 200838K 1225 0 0
soname 21 1K 0K 200838K 2038 0 0
pcb 44 38K 0K 200838K 2218 0 0
tag 0 0K 0K 200838K 772 0 0
mbuf 761 381K 0K 200838K 762 0 0
mbufcl 493 986K 0K 200838K 493 0 0
mclmeta 493 8K 0K 200838K 493 0 0
ptys 257 129K 0K 200838K 257 0 0
ttys 881 114K 0K 200838K 5731 0 0
shm 12 31K 0K 200838K 33144 0 0
ISOFS mount 1 2048K 0K 200838K 1 0 0
CAM dev queue 8 1K 0K 200838K 8 0 0
sem 3 6K 0K 200838K 3 0 0
ppbusdev 3 1K 0K 200838K 3 0 0
MD disk 2 1K 0K 200838K 2 0 0
CAM XPT 79 28K 0K 200838K 270 0 0
msg 4 27K 0K 200838K 4 0 0
CAM queue 26 1K 0K 200838K 226 0 0
rman 129 14K 0K 200838K 560 0 0
pipe 55 11K 0K 200838K 664 0 0
ioctlops 0 0K 0K 200838K 24 0 0
taskqueue 8 1K 0K 200838K 8 0 0
sbuf 1 4K 0K 200838K 3 0 0
SWAP 2 1097K 0K 200838K 2 0 0
acd_driver 1 3K 0K 200838K 1 0 0
kobj 230 518K 0K 200838K 871 0 0
ata_cam 6 4K 0K 200838K 24 0 0
eventhandler 29 2K 0K 200838K 29 0 0
disk 1 1K 0K 200838K 1 0 0
bus 1177 124K 0K 200838K 10807 0 0
ar_driver 0 0K 0K 200838K 10 0 0
callout 2 4096K 0K 200838K 2 0 0
sysctl 0 0K 0K 200838K 6225 0 0
sysctloid 585 20K 0K 200838K 585 0 0
tslpque 1 16K 0K 200838K 1 0 0
dsched 1126 251K 0K 200838K 1126 0 0
ata_dma 6 2K 0K 200838K 6 0 0
lwkt message 7 2K 0K 200838K 13 0 0
thread 102 153K 0K 200838K 102 0 0
ad_driver 2 1K 0K 200838K 2 0 0
MPipe Array 2 1K 0K 200838K 2 0 0
cache 3287 290K 0K 200838K 3425 0 0
devbuf 18332 25567K 0K 200838K 25651 0 0
temp 226 13K 0K 200838K 200701 0 0
ip6ndp 9 1K 0K 200838K 9 0 0
ata_generic 259 66K 0K 200838K 552 0 0
uidinfo 7 5K 0K 200838K 20 0 0
cred 12 2K 0K 200838K 163 0 0
pgrp 34 4K 0K 200838K 154 0 0
session 28 2K 0K 200838K 84 0 0
proc 41 76K 0K 200838K 1272 0 0
lwp 55 28K 0K 200838K 1343 0 0
subproc 57 131K 0K 200838K 1303 0 0
sbp 96 9K 0K 200838K 96 0 0

Memory Totals: In Use Free Requests
64584K 0K 509328

devbuf 18332 25567K 0K 200838K 25651 0 0

Sometimes second and third values can be > 50M. It is interesting, that they can
only increase and they NEVER decrease. This time sound just stopped and third
value became constant.

And here some info from core.txt.0 (I'll use pastebin, OK?):

http://pastebin.com/eNExkXsJ

As you can see, devbuf memory is above critical value.

Some code from usb_mem.c:

/* * Do not free the memory unconditionally since we might be called * from an interrupt context and that is BAD. * XXX when should we really free?
*/
static void
usb_block_freemem(usb_dma_block_t *p) {
DPRINTFNp->size));
logmemory(blkfree, p, NULL, p->size, p->align);
crit_enter();
LIST_INSERT_HEAD(&usb_blk_freelist, p, next);
usb_blk_nfree++;
crit_exit();
}

So it, actually, does not free memory, does it?

Here is some code from NetBSD I found:

http://fxr.watson.org/fxr/source/dev/usb/usb_mem.c?v=NETBSD;im=10

As you can see, there is some function called usb_block_real_freemem, but (as I
understand) it is never compiled (it is between #if 0/#endif).

What is "interrupt context" and why it is so dangerous?

:vasily postnicov <shamaz.mazum@gmail.com> added the comment:
:
:Some code from usb_mem.c:
:
:/*
: * Do not free the memory unconditionally since we might be called
: * from an interrupt context and that is BAD.
: * XXX when should we really free?
: */
:static void
:usb_block_freemem(usb_dma_block_t *p)
:{
: DPRINTFNp->size));
: logmemory(blkfree, p, NULL, p->size, p->align);
: crit_enter();
: LIST_INSERT_HEAD(&usb_blk_freelist, p, next);
: usb_blk_nfree++;
: crit_exit();
:}
:
:So it, actually, does not free memory, does it?

Hmm.  Well, usb_block_allocmem() will reuse it, theoretically, but
    that assumes the bus_dma_tag_t is not reallocated in the mean time.
    If it is then any related memory on usb_blk_freelist will just languish
    there and never get reused.

:Here is some code from NetBSD I found:
:
:http://fxr.watson.org/fxr/source/dev/usb/usb_mem.c?v=NETBSD;im=10
:
:As you can see, there is some function called usb_block_real_freemem, but (as I
:understand) it is never compiled (it is between #if 0/#endif).
:
:What is "interrupt context" and why it is so dangerous?

The 'interrupt context' has evolved significantly over the years.
    In fact on today's operating systems most interrupt contexts are
    actually interrupt threads and it is in fact ok to free memory
    from them.

The netbsd usb_block_real_freemem() function is enclosed in
    an #if 0/#endif, so the routine is not being used.

In anycase, the usb_allocmem()/usb_freemem() code does look mighty
    suspicious to me.  There are two layers of caches in there (there is
    also a layer in usb_allocmem() and usb_freemem()... something called
    a usb_frag_freelist).

Also I don't like the way it handles dma tags in p->tag.  There's this
    comment in our usb_mem.c code:

/* XXX - override the tag, ok since we never free it */
     p->tag = tag;
     ...

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

If it is then any related memory on usb_blk_freelist will just languish

Indeed. I've compiled kernel with USB_DEBUG and set hw.usb.debug sysctl
parameter to 7. When I use soundcard there are always "usb_block_allocmem: no
free" messages in /var/log/messages. (Signals that new blocks are allocated) But
there are also few "free list size=4096" messages (Already allocated but free
blocks are used). They appear when I JUST START to listen to music (any time
when player is just started).

Any ideas how to explain and fix it?

Hm. I believe, that usb_allocmem() works fine. My usb scanner works good and
usb_allocmem() finds free fragments and blocks perfectly. Maybe ohci_alloc_*()
passes wrong arguments to usb_allocmem()? I'll try to check it out.

M_DEVBUF is used almost everywhere, it is a nightmare to find out what is
actually leaking memory.

While overhauling vn I looked at devbuf's vmstat after every mkinitrd and it
kept increasing, one leak at a time. To verify that it wasn't vn, I changed the
malloc type for the vn allocations to a different one and the leak is still
there. So something probably somewhere in the disk i/o path using M_DEVBUF is
leaking memory, too.

To reproduce just keep an eye on devbuf's allocations while running mkinitrd a
few times.

Cheers,
Alex Hornung

New dump: http://shamazmazum.users.sourceforge.net/newdump.tar.gz (DragonFlyBSD
from master (built on Jan 23), x86_64.

I've got it when tried to unload snd_uaudio module. Maybe it will help. Also,
can I know, what "softc" stands for? Is it just some device-related options?

I looked at devbuf's vmstat after every mkinitrd and it

kept increasing, one leak at a time

Yes, I see it too. Maybe I will able to find how it is connected to my problem)

#1 md_dumpsys (di=0xffffffff809d2760)
at /usr/src/sys/platform/pc64/x86_64/dump_machdep.c:262
#2 0xffffffff8037f79b in dumpsys () at /usr/src/sys/kern/kern_shutdown.c:881
#3 0xffffffff8037fe8b in boot (howto=-2004318071)
at /usr/src/sys/kern/kern_shutdown.c:388
#4 0xffffffff80380220 in panic (fmt=0xffffffff8065dca8 "%s")
at /usr/src/sys/kern/kern_shutdown.c:787
#5 0xffffffff806289ee in trap_fatal (frame=0xffffffe04d8c8698,
eva=<value optimized out>) at /usr/src/sys/platform/pc64/x86_64/trap.c:972
#6 0xffffffff80629b84 in trap (frame=0xffffffe04d8c8698)
at /usr/src/sys/platform/pc64/x86_64/trap.c:723
#7 0xffffffff80621d2e in calltrap ()
at /usr/src/sys/platform/pc64/x86_64/exception.S:180
#8 0xffffffff8062359e in db_read_bytes (addr=7020949565765610611, size=8,
data=0xffffffe04d8c8778 "d")
at /usr/src/sys/platform/pc64/x86_64/db_interface.c:244
#9 0xffffffff8019577a in db_get_value (addr=7020949565765610611, size=8,
is_signed=0) at /usr/src/sys/ddb/db_access.c:58
#10 0xffffffff80624206 in db_nextframe (addr=<value optimized out>,
have_addr=1, count=1009, modif=<value optimized out>)
at /usr/src/sys/platform/pc64/x86_64/db_trace.c:233
#11 db_stack_trace_cmd (addr=<value optimized out>, have_addr=1, count=1009,
modif=<value optimized out>)
at /usr/src/sys/platform/pc64/x86_64/db_trace.c:439
#12 0xffffffff806243e2 in print_backtrace (count=1301055352)
at /usr/src/sys/platform/pc64/x86_64/db_trace.c:451
#13 0xffffffff80380204 in panic (
fmt=0xffffffff806b1fc0 "%s called with uninitialized or corrupt sbuf")
at /usr/src/sys/kern/kern_shutdown.c:783
#14 0xffffffff803ac9de in _assert_sbuf_integrity (
fun=0x8 <Address 0x8 out of bounds>, s=<value optimized out>)
at /usr/src/sys/kern/subr_sbuf.c:92
#15 0xffffffff803aca20 in sbuf_delete (s=0x8)
at /usr/src/sys/kern/subr_sbuf.c:552
#16 0xffffffff80f664cb in uaudio_detach (self=<value optimized out>)
at /usr/src/sys/dev/sound/driver/uaudio/../../../../dev/sound/usb/uaudio.c:467
#17 0xffffffff803a047d in DEVICE_DETACH (dev=0xffffffe008bae538)
at ./device_if.h:49
#18 device_detach (dev=0xffffffe008bae538) at /usr/src/sys/kern/subr_bus.c:1728
#19 0xffffffff803a07f8 in devclass_delete_driver (busclass=0xffffffe008b45050,
driver=0xffffffff80f6dec0) at /usr/src/sys/kern/subr_bus.c:727
#20 0xffffffff803a0960 in driver_module_handler (mod=0xffffffe008b64d90,
what=1, arg=<value optimized out>) at /usr/src/sys/kern/subr_bus.c:3113
#21 0xffffffff803588b4 in module_unload (mod=0x616f6c6e75646c73)
at /usr/src/sys/kern/kern_module.c:220
#22 0xffffffff80359fc7 in linker_file_unload (file=0xffffffe008ba8868)
at /usr/src/sys/kern/kern_linker.c:487
#23 0xffffffff8035aaa2 in sys_kldunload (uap=<value optimized out>)
at /usr/src/sys/kern/kern_linker.c:825
#24 0xffffffff80629365 in syscall2 (frame=0xffffffe04d8c8c08)
at /usr/src/sys/platform/pc64/x86_64/trap.c:1182
#25 0xffffffff80621f6f in Xfast_syscall ()
at /usr/src/sys/platform/pc64/x86_64/exception.S:313
#26 0x000000000000002b in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)