Bug #2123

hammer is losing files

Added by schmir over 3 years ago. Updated over 3 years ago.

Status:NewStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

I'm trying to use dragonflybsd as a backup server. I'm rsyncing from
multiple remote systems to a dragonfly master. Running ls -la in one
of the synced directories in a snapshot gives me "No such file or
directory errors":

,----
| muni# pwd
|
/backup/sync/@@0x000000010fae19c0/wodka-v/pediapress.com/rootfs/usr/share/zoneinfo/posix/Etc
| muni# ls -la
| ls: GMT: No such file or directory
| ls: GMT+0: No such file or directory
| ls: GMT-0: No such file or directory
| ls: GMT0: No such file or directory
| ls: Greenwich: No such file or directory
| total 0
| drwxr-xr-x 1 root wheel 0 Aug 23 03:31 .
| drwxr-xr-x 1 root wheel 0 Aug 23 03:31 ..
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+1
| -rw-r--r-- 1 root wheel 130 Aug 23 03:31 GMT+10
| -rw-r--r-- 1 root wheel 130 Aug 23 03:31 GMT+11
| -rw-r--r-- 1 root wheel 130 Aug 23 03:31 GMT+12
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+2
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+3
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+4
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+5
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+6
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+7
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+8
| -rw-r--r-- 1 root wheel 126 Aug 23 03:31 GMT+9
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-1
| -rw-r--r-- 1 root wheel 131 Aug 23 03:31 GMT-10
| -rw-r--r-- 1 root wheel 131 Aug 23 03:31 GMT-11
| -rw-r--r-- 1 root wheel 131 Aug 23 03:31 GMT-12
| -rw-r--r-- 1 root wheel 131 Aug 23 03:31 GMT-13
| -rw-r--r-- 1 root wheel 131 Aug 23 03:31 GMT-14
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-2
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-3
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-4
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-5
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-6
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-7
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-8
| -rw-r--r-- 1 root wheel 127 Aug 23 03:31 GMT-9
| -rw-r--r-- 2 root wheel 118 Aug 23 03:31 UCT
| -rw-r--r-- 6 root wheel 118 Aug 23 03:31 UTC
| -rw-r--r-- 6 root wheel 118 Aug 23 03:31 Universal
| -rw-r--r-- 6 root wheel 118 Aug 23 03:31 Zulu
`----

The missing files are all hard links to the same file, I guess they
haven't changed during backups.

dmesg contains missing inode warnings:

,----
| muni# dmesg |tail
| HAMMER: WARNING: Missing inode for dirent "GMT0"
| obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000
| HAMMER: WARNING: Missing inode for dirent "Greenwich"
| obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000
| HAMMER: WARNING: Missing inode for dirent "Greenwich"
| obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000
| HAMMER: WARNING: Missing inode for dirent "GMT+0"
| obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000
| HAMMER: WARNING: Missing inode for dirent "GMT+0"
| obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000
`----

hammer config looks like:
,----
| muni# hammer config /backup/sync/
| # No configuration present, here are some defaults
| # you can uncomment. Also remove these instructions
| #
| snapshots 1d 60d
| prune 1d 5m
| rebalance 1d 5m
| dedup 1d 5m
| reblock 1d 5m
| recopy 30d 10m
`----

uname -a says:
,----
| muni# uname -a
| DragonFly muni.brainbot.com 2.11-DEVELOPMENT DragonFly
v2.11.0.737.g5232a-DEVELOPMENT #0: Mon Aug 22 13:36:08 CEST 2011
:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64
`----

History

#1 Updated by dillon over 3 years ago

:
:I'm trying to use dragonflybsd as a backup server. I'm rsyncing from
:multiple remote systems to a dragonfly master. Running ls -la in one
:of the synced directories in a snapshot gives me "No such file or
:directory errors":
:
:,----
:| muni# pwd
:|
:/backup/sync/@@0x000000010fae19c0/wodka-v/pediapress.com/rootfs/usr/share/zoneinfo/posix/Etc
:| muni# ls -la
:| ls: GMT: No such file or directory
:| ls: GMT+0: No such file or directory
:| ls: GMT-0: No such file or directory

Where are you getting the transaction id (the @@ id) from? Is it
coming from an official snapshot softlink generated with one of
the hammer snapshot commands?

The fine-grained history can catch inodes and file data out of sync,
but the official snapshots shouldn't.

'hammer snapls <fs>' will list all official snapshots. They should
coinside with the snapshot softlinks placed in /var/hammer (by default).

-Matt
Matthew Dillon
<>

#2 Updated by schmir over 3 years ago

"Matthew Dillon (via DragonFly issue tracker)"
<> writes:

> Matthew Dillon <> added the comment:
> :
> :,----
> :| muni# pwd
> :|
> :/backup/sync/@@0x000000010fae19c0/wodka-v/pediapress.com/rootfs/usr/share/zoneinfo/posix/Etc
> :| muni# ls -la
> :| ls: GMT: No such file or directory
> :| ls: GMT+0: No such file or directory
> :| ls: GMT-0: No such file or directory
>
> Where are you getting the transaction id (the @@ id) from? Is it
> coming from an official snapshot softlink generated with one of
> the hammer snapshot commands?
>
> The fine-grained history can catch inodes and file data out of sync,
> but the official snapshots shouldn't.
>
> 'hammer snapls <fs>' will list all official snapshots. They should
> coinside with the snapshot softlinks placed in /var/hammer (by default).

The snapshot in question has been created via the daily periodic
scripts. It's symlinked in /var/hammer/backup/sync and appears in the
hammer snapls output:

`----

#3 Updated by dillon over 3 years ago

:
:The snapshot in question has been created via the daily periodic
:scripts. It's symlinked in /var/hammer/backup/sync and appears in the
:hammer snapls output:

It could have caught the rsync in the middle, though it isn't supposed
to break inode<->file linkages. rsync might be re-creating the hardlinks
on every pass.

Are the other snapshots ok?

The most reliable way is to create a manual snapshot as part of your
rsync script so the snapshot occurs on a quiescent filesystem. Something
like this:

do rsync stuff
sync; sleep 5; sync; sleep 5; sync; sleep 5
hammer snapshot /backup/sync /var/hammer/backup/sync "rsync snapshot"

-Matt
Matthew Dillon
<>

#4 Updated by schmir over 3 years ago

Matthew Dillon <> writes:

>
> It could have caught the rsync in the middle, though it isn't supposed
> to break inode<->file linkages. rsync might be re-creating the hardlinks
> on every pass.

rsync is running right before the snapshots are being created, so rsync
itself had already exited (however it didn't sync or sleep in betweeen)

>
> Are the other snapshots ok?

they are fine.

>
> The most reliable way is to create a manual snapshot as part of your
> rsync script so the snapshot occurs on a quiescent filesystem. Something
> like this:
>
> do rsync stuff
> sync; sleep 5; sync; sleep 5; sync; sleep 5
> hammer snapshot /backup/sync /var/hammer/backup/sync "rsync snapshot"

I do that now and will watch for similar problems.

Thanks for your help.

Also available in: Atom PDF