Bug #1135

hammer + nfs => Input/output error

Added by thomas.nikolajsen about 6 years ago. Updated over 5 years ago.

Status:ClosedStart date:
Priority:LowDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

Using HAMMER for nfs exports, nfs client often get in a state
where access to most files return `Input/output error' (EIO state).

Example:
- nfs server using HAMMER for all exports
- nfs client mount of usr/src and /home/thomas (/usr/obj local)
When building & installing world & kernel /home/thomas often get in `EIO state'.
/home/thomas is only lightly used (contrary to /usr/src).
Only known solution (besides reboot) is remounting /home/thomas.

When this happens nfs server log error (multiple times):
lookupdotdot failed 2 dvp 0xca3087d0

Is core dump needed?
(will decrease hw.physmem; 2G now)

-thomas

History

#1 Updated by dillon about 6 years ago

:Using HAMMER for nfs exports, nfs client often get in a state
:where access to most files return `Input/output error' (EIO state).
:
:Example:
: - nfs server using HAMMER for all exports
: - nfs client mount of usr/src and /home/thomas (/usr/obj local)
:When building & installing world & kernel /home/thomas often get in `EIO st=
:ate'.
:/home/thomas is only lightly used (contrary to /usr/src).
:Only known solution (besides reboot) is remounting /home/thomas.
:
:When this happens nfs server log error (multiple times):
:lookupdotdot failed 2 dvp 0xca3087d0
:
:Is core dump needed?
:(will decrease hw.physmem; 2G now)
:
: -thomas

What is the client? DragonFly? Linux? FreeBSD?

-Matt
Matthew Dillon
<>

#2 Updated by dillon about 6 years ago

:Using HAMMER for nfs exports, nfs client often get in a state
:where access to most files return `Input/output error' (EIO state).
:
:Example:
: - nfs server using HAMMER for all exports
: - nfs client mount of usr/src and /home/thomas (/usr/obj local)

Another question. Are you trying to export PFSs or are you exporting
a base HAMMER mount?

There are going to be multiple issues with PFSs.

:When this happens nfs server log error (multiple times):
:lookupdotdot failed 2 dvp 0xca3087d0
:
:Is core dump needed?
:(will decrease hw.physmem; 2G now)
:
: -thomas

Hmm. I will try to reproduce it. What is your system config?
e.g. Memory, df output, exports list, on the server. Also your
df output and fstab on the client.

That particular VOP doesn't get run very often due to caching. It
is probably (hopefully) a simple bug but I probably need to instrument
the filesystem to track it down.

A kernel core would be useful if you add code to panic the system
in the correct procedure so I get some context to work with. I have
included a patch for that below (adding temporary debugging panics
in the right places). This would be on the server side.

-Matt
Matthew Dillon
<>

Index: hammer_vnops.c
===================================================================
RCS file: /cvs/src/sys/vfs/hammer/hammer_vnops.c,v
retrieving revision 1.96
diff -u -p -r1.96 hammer_vnops.c
--- hammer_vnops.c 9 Aug 2008 07:04:16 -0000 1.96
+++ hammer_vnops.c 9 Sep 2008 23:50:12 -0000
@@ -965,6 +965,7 @@ hammer_vop_nlookupdotdot(struct vop_nloo
dip->obj_asof);
} else {
*ap->a_vpp = NULL;
+ panic("HAMMER DOTDOT1");
return ENOENT;
}
}
@@ -982,6 +983,8 @@ hammer_vop_nlookupdotdot(struct vop_nloo
*ap->a_vpp = NULL;
}
hammer_done_transaction(&trans);
+ if (error)
+ panic("HAMMER DOTDOT2");
return (error);
}

#3 Updated by thomas.nikolajsen about 6 years ago

Matthew Dillon <> wrote:
..
> What is the client? DragonFly? Linux? FreeBSD?

Client: DragonFly HEAD
Server: DragonFly RELEASE-2.0 (non Slip)

'EIO state' only on some of nfs mounts,
typically /home/thomas is affected, but /usrs/src is not

-thomas

#4 Updated by thomas.nikolajsen about 6 years ago

..
> Another question. Are you trying to export PFSs or are you exporting
> a base HAMMER mount?
>
> There are going to be multiple issues with PFSs.

Ahh, yes, I am exporting PFSs, I didn't know it isn't supported (yet)!

I will move data in HAMMER file system on my nfs server out of PFSs and retest;
it is going to be a few hours before files are moved, stay tuned.

Will send requested info if problem persists.

-thomas
PS: Now I won't bug that export of PFS root isn't possible (symlink);
but I guess that will change anyway.

#5 Updated by dillon about 6 years ago

:Ahh, yes, I am exporting PFSs, I didn't know it isn't supported (yet)!
:
:I will move data in HAMMER file system on my nfs server out of PFSs and ret=
:est;
:it is going to be a few hours before files are moved, stay tuned.
:
:Will send requested info if problem persists.
:
: -thomas
:PS: Now I won't bug that export of PFS root isn't possible (symlink);
:but I guess that will change anyway.

I have some hacks in there to try to support it but there are issues.
I definitely want to support exporting PFSs for 2.2.

I think half of the problems are related to that dotdot vop. I get
weirdness just trying to mount an exported PFS.

The other issue is that HAMMER has only one netexport structure... it
really needs one for each PFS. And I don't think NULLFS (for nullfs
mounts of the PFSs) does the pass-through quite right.

Please do re-test. I want to be sure that this is a PFS issue and not
a general HAMMER issue.

-Matt
Matthew Dillon
<>

#6 Updated by thomas.nikolajsen about 6 years ago

Retest success!

I did:
- mv files out of PFSs (took 30h; 3.5 mio files, 100GB);
and destroy PFSs
- traverse HAMMER fs: du; find . -ls;
then periodic(8) chimed in: chksetuid & locate (2+h each)
- several buildworld with local and nfs mounted /usr/obj

No `Input/output error' or `lookupdotdot failed' seen.

Most testing done using patch from issue1139;
any reason to retest without this patch?

Is it OK if I have PFSs in nfs exported files system
as long as I don't touch files in PFS from nfs client?

-thomas

#7 Updated by dillon about 6 years ago

:Is it OK if I have PFSs in nfs exported files system
:as long as I don't touch files in PFS from nfs client?
:
: -thomas

It should be ok.

-Matt
Matthew Dillon
<>

#8 Updated by corecode almost 6 years ago

can we close this?

Also available in: Atom PDF