https://bugs.dragonflybsd.org/https://bugs.dragonflybsd.org/favicon.ico?16293952082016-10-30T01:19:26ZDragonFlyBSD bugtrackerDragonFlyBSD - Bug #2962: Hammer PFS Slave has broken symbolic link, recreating it doesn't workhttps://bugs.dragonflybsd.org/issues/2962?journal_id=130252016-10-30T01:19:26ZAnonymous
<ul></ul><p>After discussing with Matt Dillon on the IRC channel, I've decided to copy the master's files to an adjacent drive, then dump the master/slave metadatas and finally destroy/recreate both master and slave HAMMER filesystems.</p>
<p>Results of the metadata will be posted here when my backup is complete.</p> DragonFlyBSD - Bug #2962: Hammer PFS Slave has broken symbolic link, recreating it doesn't workhttps://bugs.dragonflybsd.org/issues/2962?journal_id=130292016-10-31T04:32:39ZAnonymous
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul><p>After the IRC discussion, I noticed that the session for recreating the PFS did not complete as I expected. When it did, it had recreated the faulted PFS correctly.</p>
<p>Therefore I consider this issue closed given that:</p>
<p>1) A Faulted PFS may be deleted using pfs-destroy<br />2) It may be created using mirror-copy.</p>
<p>While that operation was pending, I made another backup of the master PFS to another drive entirely, also running HAMMER.</p>
<p>To my dismay, attempts to delete the backup directory (2.1TiB) caused a denial of service on the system. A hard reset only caused HAMMER to attempt to rollback the transaction log and caused yet another denial of service.</p>
<p>After giving it 6 hours to attempt to mount via `mount_hammer` (Control-T revealed it was in `nbufs`), I gave up, booted into single user mode, and disabled the master/slave drives for that HAMMER filesystem.</p>
<p>Any attempts to mount the DoS'ing HAMMER filesystem were ineffective due to it's insistence on reverting to a prior transaction using it's undo log.</p>
<p>I've since switched to the backup (slave) and made it the master and mounted it in the correct place.</p>
<p>The now-faulted HAMMER master (with 2.1TiB pending deletions of history) remains unmounted.</p>
<p>If there's any advice or interest in figuring out why a `sudo rm -rf /Archive1/backup` caused a denial of service, I'm happy to conduct any activities on it for one week from this update (ending on 11/5/2016).</p>
<p>After that, I will reformat the faulted master and set it up as a new mirrored-slave of the recently promoted slave-turned-master.</p>
<p>No data has been lost due to the master-slave mirroring of HAMMER, however this experience would have been catastrophic if I had conducted a very large deletion sweep on a HAMMER partition (no explicit PFS used to hold the errant data).</p>
<p>Lessons learned:</p>
<p>1. If you have a faulted PFS, destroy it and recreate it. Wait for any mirroring to be complete as the symlink to it will remain invalid until the mirroring is complete.</p>
<p>2. If you make a multi Terabyte copy of a HAMMER master to another HAMMER master with default snapshot/history configuration, do not attempt to delete it all at once. Suggest deleting in sweeps of a few gigabytes and increasing until system latency is noticeably implacted.</p>
<p>3. If you are storing vast quantities of data on HAMMER, ensure your snapshot/history configuration is sensible. Mine was using the defaults, which now seems questionable.</p>
<p>4. Mirror-mirror-mirror your data.</p>
<p>5. If you find yourself unable to boot due to HAMMER redo on a NON-ROOT HAMMER filesystem, use the boot menu to launch single user mode, mount /var, remount root as read-write and use `vi` or any other editor to comment out the guilty filesystem so you may get a working environment.</p>
<p>6. Do not take this as an indictment of HAMMER but a "stupid user" story wherein I provoked a pathological case while incorrectly assuming my earlier efforts to remirror the PFS were ineffective.</p>
<p>Ironically, the original offender (Archive2) and it's backup (Archive2Backup) were restored completely and have no issues, while my worst-case guess of backing up the master and deleting an now-no-longer-needed copy of the file hierarchy ended up causing an even bigger problem.</p> DragonFlyBSD - Bug #2962: Hammer PFS Slave has broken symbolic link, recreating it doesn't workhttps://bugs.dragonflybsd.org/issues/2962?journal_id=130302016-10-31T05:26:57ZAnonymous
<ul></ul><p>Apparently I forgot the rest of the sentence:</p>
<p>"No data has been lost due to the master-slave mirroring of HAMMER, however this experience would have been catastrophic if I had conducted a very large deletion sweep on a HAMMER partition (no explicit PFS used to hold the errant data)."</p>
<p>should be completed with:</p>
<p>"on a HAMMER partition without a mirror (i.e. the root)".</p> DragonFlyBSD - Bug #2962: Hammer PFS Slave has broken symbolic link, recreating it doesn't workhttps://bugs.dragonflybsd.org/issues/2962?journal_id=130312016-10-31T06:11:02Ztkusumikusumi.tomohiro@gmail.com
<ul></ul><p>What version (uname -r) is this ?</p> DragonFlyBSD - Bug #2962: Hammer PFS Slave has broken symbolic link, recreating it doesn't workhttps://bugs.dragonflybsd.org/issues/2962?journal_id=130322016-10-31T20:06:25ZAnonymous
<ul></ul><p>[ben@nyx ~]$ uname -r<br />4.4-RELEASE</p>
<p>I was in the process of upgrading to 4.6 on the first stage (make buildworld) but stopped when I discovered the problems.</p>
<p>Quite happy to resume upgrading (up to and including master) if that is adviseable.</p>