DragonFlyBSD

:#6 0xc02fe396 in calltrap () at /usr/src/sys/platform/pc32/i386/exception.=
:s:783
:#7 0xc0233d36 in sack_block_lookup (scb=3D0xdace6b0c, seq=3D1554912228, sb=
:=3D0xdaa45a90) at /usr/src/sys/netinet/tcp_sack.c:128
:#8 0xc0233eda in tcp_sack_nextseg (tp=3D0xdace6a20, nextrexmt=3D0xdaa45ad0=
:, plen=3D0xdaa45ad4, lostdup=3D0xdaa45acc) at /usr/src/sys/netinet/tcp_sack=
:=2Ec:496
:#9 0xc022f603 in tcp_sack_rexmt (tp=3D0xdace6a20, th=3D<value optimized ou=

Hmm.  I see two places where a node is removed from the sackblocks list
    but lastfound is not cleared on match.  I don't know if this is the
    issue but it's the most obvious from looking at the failure.

I'll commit this tomorrow if no new developments come up.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

Index: tcp_sack.c ===================================================================
RCS file: /cvs/src/sys/netinet/tcp_sack.c,v
retrieving revision 1.6
diff u -p -r1.6 tcp_sack.c
-- tcp_sack.c 22 Apr 2007 01:13:14 0000 1.6
++ tcp_sack.c 3 Feb 2008 01:32:16 -0000
@ -176,7 +176,7 @
sb = TAILQ_FIRST(&scb>sackblocks);
while (sb && SEQ_LEQ(sb->sblk_end, th_ack)) {
nb = TAILQ_NEXT(sb, sblk_list);
- if (sb scb->lastfound)
if (scb->lastfound sb)
scb->lastfound = NULL;
TAILQ_REMOVE(&scb->sackblocks, sb, sblk_list);
free_sackblock(sb);
@ -334,6 +334,8 @ SEQ_GEQ(workingblock->sblk_end, sb-
struct sackblock *nextblock;

nextblock = TAILQ_NEXT(sb, sblk_list);
+        if (scb->lastfound == sb)
+            scb->lastfound = NULL;
         /* Remove completely overlapped block /
         TAILQ_REMOVE(&scb->sackblocks, sb, sblk_list);
         free_sackblock(sb);
@ -346,6 +348,8 @     if (sb != NULL &&
         SEQ_GEQ(workingblock->sblk_end, sb->sblk_start)) {
         / Extend new block to cover partially overlapped old block. */
         workingblock->sblk_end = sb->sblk_end;
+        if (scb->lastfound == sb)
+            scb->lastfound = NULL;
         TAILQ_REMOVE(&scb->sackblocks, sb, sblk_list);
         free_sackblock(sb);
         --scb->nblocks;

Also just got this with the same sources:

panic: zone: freeing free entry
mp_lock = 00000000; cpuid = 0
boot() called on cpu#0
Uptime: 1d11h35m59s

dumping to dev #da/0x20001, blockno 378927

(kgdb) bt
#0 dumpsys () at ./machine/thread.h:83
#1 0xc01a2ea9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:375
#2 0xc01a316c in panic (fmt=0xc034328a "zone: freeing free entry") at /usr/src/sys/kern/kern_shutdown.c:800
#3 0xc02a6aa8 in zerror (error=2) at /usr/src/sys/vm/vm_zone.c:567
#4 0xc02a6ff5 in zfree (z=0xd7049438, item=0xdb991760) at /usr/src/sys/vm/vm_zone.c:98
#5 0xc02341ac in tcp_sack_update_scoreboard (tp=0xdad397c0, to=0xdaa45be8) at /usr/src/sys/netinet/tcp_sack.c:165
#6 0xc02318d9 in tcp_input (m=0xeb7df200) at /usr/src/sys/netinet/tcp_input.c:1900
#7 0xc0229ae2 in transport_processing_oncpu (m=0xeb7df200, hlen=20, ip=<value optimized out>, nexthop=0x0) at /usr/src/sys/netinet/ip_input.c:391
#8 0xc022bae0 in ip_input (m=0xeb7df200) at /usr/src/sys/netinet/ip_input.c:1092
#9 0xc022bbb4 in ip_input_handler (msg0=0xeb7df218) at /usr/src/sys/netinet/ip_input.c:421
#10 0xc0235653 in tcpmsg_service_loop (dummy=0x0) at /usr/src/sys/netinet/tcp_subr.c:385
#11 0xc01a9fa5 in lwkt_deschedule_self (td=Cannot access memory at address 0x8
) at /usr/src/sys/kern/lwkt_thread.c:214
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Do you think it's the same problem?

FYI, the vmcores are on leaf:~pavalos/crash. The first one is *12 and
the 2nd is *13.

--Peter

:Also just got this with the same sources:
:
:panic: zone: freeing free entry
:mp_lock =3D 00000000; cpuid =3D 0
:boot() called on cpu#0
:Uptime: 1d11h35m59s
:...
:#3 0xc02a6aa8 in zerror (error=3D2) at /usr/src/sys/vm/vm_zone.c:567
:#4 0xc02a6ff5 in zfree (z=3D0xd7049438, item=3D0xdb991760) at /usr/src/sys=
:/vm/vm_zone.c:98
:#5 0xc02341ac in tcp_sack_update_scoreboard (tp=3D0xdad397c0, to=3D0xdaa45=
:be8) at /usr/src/sys/netinet/tcp_sack.c:165
:#6 0xc02318d9 in tcp_input (m=3D0xeb7df200) at /usr/src/sys/netinet/tcp_in=
:put.c:1900
:#7 0xc0229ae2 in transport_processing_oncpu (m=3D0xeb7df200, hlen=3D20, ip=
:
:Do you think it's the same problem?

Same sources prior to the patch?  It's quite possible.

I tracked this second crash to line 321 of tcp_sack.c (the kgdb backtrace
    is all wrong due to all the inlining).  It's freeing 'newblock' here,
    which should always succeed at this paricular point in the code.

I think this case can only occur if the list had previously been
    corrupted due to the hint not getting NULL'd out in those two places.

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

did this get committed?

Committed in 9e3d6c9645ed28ef5b07a9b13e380e13a86deeb8. I haven't seen this
panic in about a year, so let's call it good.