DragonFlyBSD

:hey,
:
:this is the Nth time now i'm running into free frag or free block panics.=
: either vinum is doing something very wrong or it is just interacting wi=
:th ffs on a bad way. anyways, here it comes:

It has got to be vinum.  This particular bug only occurs very
    occassionally 'in the wild'.

Is it possible to narrow it down to a particular vinum
    configuration?  e.g. mirroring vs RAID-5 vs concat ?

-Matt
                    Matthew Dillon 
                    <dillon@backplane.com>

Matthew Dillon wrote:

:hey,
:
:this is the Nth time now i'm running into free frag or free block panics.=
: either vinum is doing something very wrong or it is just interacting wi=
:th ffs on a bad way. anyways, here it comes:

It has got to be vinum. This particular bug only occurs very
occassionally 'in the wild'.

Is it possible to narrow it down to a particular vinum
configuration? e.g. mirroring vs RAID-5 vs concat ?

there is no fs on vinum yet. i was just running a raid-5 init and a mirroring rebuild (but on a differend hdd!)

cheers
simon

:there is no fs on vinum yet. i was just running a raid-5 init and a mirr=
:oring rebuild (but on a differend hdd!)
:
:cheers
: simon

Also, is the disk vinum is messing around with on the same controller
     or cable as the one that blew up?

-Matt

:..
:> occassionally 'in the wild'.
:>=20
:> Is it possible to narrow it down to a particular vinum
:> configuration? e.g. mirroring vs RAID-5 vs concat ?
:
:there is no fs on vinum yet. i was just running a raid-5 init and a mirr=
:oring rebuild (but on a differend hdd!)
:
:cheers
: simon

Ok, try this... see if it occurs without vinum, but with other nominal
     disk loading.

It could be that vinum is screwing up the geteblk or getpbuf buffers it
     obtains, and that is causing other filesystems to blow up.

-Matt

Matthew Dillon wrote:

:there is no fs on vinum yet. i was just running a raid-5 init and a mirr=
:oring rebuild (but on a differend hdd!)
Also, is the disk vinum is messing around with on the same controller
or cable as the one that blew up?

no, one is ad0 on the ata part of the chipset, the other ad4 on the sata part.

cheers
simon

Matthew Dillon wrote:

:> occassionally 'in the wild'.
:>=20
:> Is it possible to narrow it down to a particular vinum
:> configuration? e.g. mirroring vs RAID-5 vs concat ?
:
:there is no fs on vinum yet. i was just running a raid-5 init and a mirr=
:oring rebuild (but on a differend hdd!)
Ok, try this... see if it occurs without vinum, but with other nominal
disk loading.

It could be that vinum is screwing up the geteblk or getpbuf buffers it
obtains, and that is causing other filesystems to blow up.

I suspect that vinum drains lots of blks and makes other parts of the system to flush out buffer sooner or something like that. is that possible?

cheers
simon

:I suspect that vinum drains lots of blks and makes other parts of the sys=
:tem to flush out buffer sooner or something like that. is that possible?=
:
:cheers
: simon

All things are possible.  vinum is probably doing the I/O's for
    each disk in parallel.  What I like about this bug is that you seem
    to be able to reproduce it in fairly short order.

Could you post your vinum config and the vinum commands you are
    using to init it ?

-Matt

Matthew Dillon wrote:

:I suspect that vinum drains lots of blks and makes other parts of the sys=
:tem to flush out buffer sooner or something like that. is that possible?=
:
:cheers
: simon

All things are possible. vinum is probably doing the I/O's for
each disk in parallel. What I like about this bug is that you seem
to be able to reproduce it in fairly short order.

too bad it is my desktop system with 1GB of RAM... :)

Could you post your vinum config and the vinum commands you are
using to init it ?

sure, here you go:

drive v0 device /dev/ad4s1l
drive v1 device /dev/ad4s1m
drive v2 device /dev/ad4s1n
drive v3 device /dev/ad4s1o

volume test1
plex org striped 256s
sd drive v0 len 3g
sd drive v1 len 3g
plex org striped 256s
sd drive v2 len 3g
sd drive v3 len 3g

volume test2
plex org concat
sd drive v0 len 3g
sd drive v1 len 3g

volume test3
plex org raid5 256s
sd drive v0 len 3g
sd drive v1 len 3g
sd drive v2 len 3g
sd drive v3 len 3g

vinum commands:
create vinum.conf
init test3.p0
start test1.p1

(lots of hdd traffic now)

cheers
simon

So far I haven't had any luck reproducing the blkfree panic. I'm doing a
parallel buildworld and vinum init in a loop.

I did find at least one issue with the patch set... you have to use
    getpbuf() rather then trypbuf().  You can't have temporary resource 
    failures in the places where trypbuf() is being used.  This is unrelated
    to the bug, though.

If you could get a crash dump of the blkfree panic onto leaf I'd like 
    to take a look at it.

-Matt

Matthew Dillon wrote:

So far I haven't had any luck reproducing the blkfree panic. I'm doing a
parallel buildworld and vinum init in a loop.

I could not reproduce it either. I remembered something else:

Usually I did a
sync
mount -a -u -o ro

before fiddling with vinum, but /dev never re-mounted ro because of open files. The other file systems though did, but nevertheless showed inconsistencies in fsck -f afterwards. I don't know if this might be connected.

I did find at least one issue with the patch set... you have to use
getpbuf() rather then trypbuf(). You can't have temporary resource
failures in the places where trypbuf() is being used. This is unrelated
to the bug, though.

I am not sure that I understand what you mean: The original (and now used) code checks the result of geteblk(), but this can't fail anyways. So I decided to go with trypbuf() instead of getpbuf() so that those functions (all called from ioctl) could fail and not block.

If you could get a crash dump of the blkfree panic onto leaf I'd like
to take a look at it.

Will do, but it will take some time, as it is >200mb compressed.

cheers
simon

:..
:> getpbuf() rather then trypbuf(). You can't have temporary resource=
:=20
:> failures in the places where trypbuf() is being used. This is unre=
:lated
:> to the bug, though.
:
:I am not sure that I understand what you mean: The original (and now use=
:d) code checks the result of geteblk(), but this can't fail anyways. So =
:I decided to go with trypbuf() instead of getpbuf() so that those functio=
:ns (all called from ioctl) could fail and not block.

geteblk() isn't supposed to fail under normal operating conditions.
    Failures are considered, well, a fatal error.

trypbuf() CAN fail under normal operating conditions, just like
    malloc(M_NOWAIT) can fail under normal operating conditions.  Such
    failures are not a fatal error, but vinum isn't designed to treat them
    as non-fatal so what you will up with will be a bunch of odd vinum
    failures that it doesn't recover from under certain load conditions.

-Matt

:
:> If you could get a crash dump of the blkfree panic onto leaf I'd li=
:ke=20
:> to take a look at it.
:
:Will do, but it will take some time, as it is >200mb compressed.
:
:cheers
: simon

:Usually I did a
: sync
: mount -a -u -o ro
:
:before fiddling with vinum, but /dev never re-mounted ro because of open =
:files. The other file systems though did, but nevertheless showed incons=
:istencies in fsck -f afterwards. I don't know if this might be connected=
:=2E

It could be.  Going from RW -> RO is not a direction that is tested
    often.

-Matt