Project

General

Profile

Actions
Copy link

Bug #2123

open

hammer is losing files

Added by schmir about 13 years ago. Updated about 13 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
6.4
Start date:
Due date:
% Done:

0%

Estimated time:

Description

I'm trying to use dragonflybsd as a backup server. I'm rsyncing from
multiple remote systems to a dragonfly master. Running ls -la in one
of the synced directories in a snapshot gives me "No such file or
directory errors":

,---- | muni# pwd |
/backup/sync/@@0x000000010fae19c0/wodka-v/pediapress.com/rootfs/usr/share/zoneinfo/posix/Etc | muni# ls -la | ls: GMT: No such file or directory | ls: GMT+0: No such file or directory | ls: GMT-0: No such file or directory | ls: GMT0: No such file or directory | ls: Greenwich: No such file or directory | total 0 | drwxr-xr-x 1 root wheel 0 Aug 23 03:31 . | drwxr-xr-x 1 root wheel 0 Aug 23 03:31 .. | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+1 | rw-r--r- 1 root wheel 130 Aug 23 03:31 GMT+10 | rw-r--r- 1 root wheel 130 Aug 23 03:31 GMT+11 | rw-r--r- 1 root wheel 130 Aug 23 03:31 GMT+12 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+2 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+3 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+4 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+5 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+6 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+7 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+8 | rw-r--r- 1 root wheel 126 Aug 23 03:31 GMT+9 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-1 | rw-r--r- 1 root wheel 131 Aug 23 03:31 GMT-10 | rw-r--r- 1 root wheel 131 Aug 23 03:31 GMT-11 | rw-r--r- 1 root wheel 131 Aug 23 03:31 GMT-12 | rw-r--r- 1 root wheel 131 Aug 23 03:31 GMT-13 | rw-r--r- 1 root wheel 131 Aug 23 03:31 GMT-14 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-2 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-3 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-4 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-5 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-6 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-7 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-8 | rw-r--r- 1 root wheel 127 Aug 23 03:31 GMT-9 | rw-r--r- 2 root wheel 118 Aug 23 03:31 UCT | rw-r--r- 6 root wheel 118 Aug 23 03:31 UTC | rw-r--r- 6 root wheel 118 Aug 23 03:31 Universal | rw-r--r- 6 root wheel 118 Aug 23 03:31 Zulu
`----

The missing files are all hard links to the same file, I guess they
haven't changed during backups.

dmesg contains missing inode warnings:

,---- | muni# dmesg |tail | HAMMER: WARNING: Missing inode for dirent "GMT0" | obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000 | HAMMER: WARNING: Missing inode for dirent "Greenwich" | obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000 | HAMMER: WARNING: Missing inode for dirent "Greenwich" | obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000 | HAMMER: WARNING: Missing inode for dirent "GMT+0" | obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000 | HAMMER: WARNING: Missing inode for dirent "GMT+0" | obj_id = 000000010c351d62, asof=000000010fae19c0, lo=00010000
`----

hammer config looks like:
,---- | muni# hammer config /backup/sync/ | # No configuration present, here are some defaults | # you can uncomment. Also remove these instructions | # | snapshots 1d 60d | prune 1d 5m | rebalance 1d 5m | dedup 1d 5m | reblock 1d 5m | recopy 30d 10m
`----

uname a says:
,--- | muni# uname -a | DragonFly muni.brainbot.com 2.11-DEVELOPMENT DragonFly
v2.11.0.737.g5232a-DEVELOPMENT #0: Mon Aug 22 13:36:08 CEST 2011
:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64
`----

Actions
Copy link
 #1

Updated by dillon about 13 years ago

:
:I'm trying to use dragonflybsd as a backup server. I'm rsyncing from
:multiple remote systems to a dragonfly master. Running ls la in one
:of the synced directories in a snapshot gives me "No such file or
:directory errors":
:
:,---
:| muni# pwd
:|
:/backup/sync/@@0x000000010fae19c0/wodka-v/pediapress.com/rootfs/usr/share/zoneinfo/posix/Etc
:| muni# ls -la
:| ls: GMT: No such file or directory
:| ls: GMT+0: No such file or directory
:| ls: GMT-0: No such file or directory

Where are you getting the transaction id (the @@ id) from?  Is it
    coming from an official snapshot softlink generated with one of
    the hammer snapshot commands?
The fine-grained history can catch inodes and file data out of sync,
    but the official snapshots shouldn't.
'hammer snapls <fs>' will list all official snapshots.  They should
    coinside with the snapshot softlinks placed in /var/hammer (by default).
-Matt
                    Matthew Dillon 
                    <>
Actions
Copy link
 #2

Updated by schmir about 13 years ago

"Matthew Dillon (via DragonFly issue tracker)"
<> writes:

Matthew Dillon <> added the comment:
:
:,----
:| muni# pwd
:|
:/backup/sync/@@0x000000010fae19c0/wodka-v/pediapress.com/rootfs/usr/share/zoneinfo/posix/Etc
:| muni# ls -la
:| ls: GMT: No such file or directory
:| ls: GMT+0: No such file or directory
:| ls: GMT-0: No such file or directory

Where are you getting the transaction id (the @@ id) from? Is it
coming from an official snapshot softlink generated with one of
the hammer snapshot commands?

The fine-grained history can catch inodes and file data out of sync,
but the official snapshots shouldn't.

'hammer snapls <fs>' will list all official snapshots. They should
coinside with the snapshot softlinks placed in /var/hammer (by default).

The snapshot in question has been created via the daily periodic
scripts. It's symlinked in /var/hammer/backup/sync and appears in the
hammer snapls output:

`----

Actions
Copy link
 #3

Updated by dillon about 13 years ago

:
:The snapshot in question has been created via the daily periodic
:scripts. It's symlinked in /var/hammer/backup/sync and appears in the
:hammer snapls output:

It could have caught the rsync in the middle, though it isn't supposed
    to break inode<->file linkages.  rsync might be re-creating the hardlinks
    on every pass.
Are the other snapshots ok?
The most reliable way is to create a manual snapshot as part of your
    rsync script so the snapshot occurs on a quiescent filesystem.  Something
    like this:
do rsync stuff
    sync; sleep 5; sync; sleep 5; sync; sleep 5
    hammer snapshot /backup/sync /var/hammer/backup/sync "rsync snapshot" 
-Matt
                    Matthew Dillon 
                    &lt;&gt;
Actions
Copy link
 #4

Updated by schmir about 13 years ago

Matthew Dillon <> writes:

It could have caught the rsync in the middle, though it isn't supposed
to break inode<->file linkages. rsync might be re-creating the hardlinks
on every pass.

rsync is running right before the snapshots are being created, so rsync
itself had already exited (however it didn't sync or sleep in betweeen)

Are the other snapshots ok?

they are fine.

The most reliable way is to create a manual snapshot as part of your
rsync script so the snapshot occurs on a quiescent filesystem. Something
like this:

do rsync stuff
sync; sleep 5; sync; sleep 5; sync; sleep 5
hammer snapshot /backup/sync /var/hammer/backup/sync "rsync snapshot"

I do that now and will watch for similar problems.

Thanks for your help.

Actions
Copy link

Also available in: Atom PDF