DragonFlyBSD

After the IRC discussion, I noticed that the session for recreating the PFS did not complete as I expected. When it did, it had recreated the faulted PFS correctly.

Therefore I consider this issue closed given that:

1) A Faulted PFS may be deleted using pfs-destroy
2) It may be created using mirror-copy.

While that operation was pending, I made another backup of the master PFS to another drive entirely, also running HAMMER.

To my dismay, attempts to delete the backup directory (2.1TiB) caused a denial of service on the system. A hard reset only caused HAMMER to attempt to rollback the transaction log and caused yet another denial of service.

After giving it 6 hours to attempt to mount via `mount_hammer` (Control-T revealed it was in `nbufs`), I gave up, booted into single user mode, and disabled the master/slave drives for that HAMMER filesystem.

Any attempts to mount the DoS'ing HAMMER filesystem were ineffective due to it's insistence on reverting to a prior transaction using it's undo log.

I've since switched to the backup (slave) and made it the master and mounted it in the correct place.

The now-faulted HAMMER master (with 2.1TiB pending deletions of history) remains unmounted.

If there's any advice or interest in figuring out why a `sudo rm -rf /Archive1/backup` caused a denial of service, I'm happy to conduct any activities on it for one week from this update (ending on 11/5/2016).

After that, I will reformat the faulted master and set it up as a new mirrored-slave of the recently promoted slave-turned-master.

No data has been lost due to the master-slave mirroring of HAMMER, however this experience would have been catastrophic if I had conducted a very large deletion sweep on a HAMMER partition (no explicit PFS used to hold the errant data).

Lessons learned:

1. If you have a faulted PFS, destroy it and recreate it. Wait for any mirroring to be complete as the symlink to it will remain invalid until the mirroring is complete.

2. If you make a multi Terabyte copy of a HAMMER master to another HAMMER master with default snapshot/history configuration, do not attempt to delete it all at once. Suggest deleting in sweeps of a few gigabytes and increasing until system latency is noticeably implacted.

3. If you are storing vast quantities of data on HAMMER, ensure your snapshot/history configuration is sensible. Mine was using the defaults, which now seems questionable.

4. Mirror-mirror-mirror your data.

5. If you find yourself unable to boot due to HAMMER redo on a NON-ROOT HAMMER filesystem, use the boot menu to launch single user mode, mount /var, remount root as read-write and use `vi` or any other editor to comment out the guilty filesystem so you may get a working environment.

6. Do not take this as an indictment of HAMMER but a "stupid user" story wherein I provoked a pathological case while incorrectly assuming my earlier efforts to remirror the PFS were ineffective.

Ironically, the original offender (Archive2) and it's backup (Archive2Backup) were restored completely and have no issues, while my worst-case guess of backing up the master and deleting an now-no-longer-needed copy of the file hierarchy ended up causing an even bigger problem.