RunBSD

← Minecraft Backups

ZFS-to-Tape Backup

15 July 2026 (updated 7 August 2026)

From gmktec (FreeBSD) to seneca (Linux): the migration story

This system started on gmktec, a FreeBSD 15.0-RELEASE box, backing up a Minecraft world to an HP DAT72 tape drive over USB. FreeBSD's USB/CAM stack never got along with the drive's USB bridge controller — mt commands intermittently failed, and a failed write could leave the sa(4) driver in a "frozen" state requiring a rewind to clear. A Raspberry Pi running Linux was tested as an interim replacement and worked well, confirming the problem was FreeBSD-specific rather than the drive or cabling.

As of 5 August 2026, the permanent replacement host is seneca — an industrial fanless mini-PC running Debian 13.6.0 "trixie" (kernel 6.12.100+deb13-amd64). The same DAT72 drive (model C7438A per dmesg) is attached, still over USB — confirmed via scsi host2: usb-storage 1-1:1.0 and cat /sys/class/scsi_host/host2/proc_name returning usb-storage. On Linux, the drive uses the st driver instead of FreeBSD's sa(4): /dev/st0 (rewind) and /dev/nst0 (no-rewind), in place of /dev/sa0 / /dev/nsa0. Everything below describes the current, Linux-based setup; where something carried over unchanged from the FreeBSD era, or was dropped/added during the port, that's called out explicitly.

ZFS-to-Tape Backup System — Setup Summary

Host: seneca (Debian 13.6.0 "trixie"). Tape drive: HP DAT72 USB (/dev/st0 / /dev/nst0, SCSI-over-USB via usb-storage). Purpose: daily incremental backup of a Minecraft world to tape, on top of the existing tarsnap cloud backup of the same directory (unchanged from the FreeBSD setup).

What's backed up

Scripts

Two scripts, deployed to /usr/local/sbin/ on seneca, root-owned, mode 755:

Ported from FreeBSD — what changed:

  1. Tape device naming — /dev/nsa0 (FreeBSD sa(4)) → /dev/nst0 (Linux st driver).
  2. Shebang — #!/bin/sh → #!/bin/bash. set -o pipefail isn't POSIX and isn't supported by dash, which is /bin/sh on Debian. Bash ships by default on Debian but isn't /bin/sh there, hence the explicit bash shebang.
  3. Date arithmetic — BSD date -j -f '%Y-%m-%d' "$DATE" '+%s' → GNU date -d "$DATE" '+%s'.
  4. State/lock paths — /var/db/zfs-tape-backup → /var/lib/zfs-tape-backup (Linux FHS convention; FreeBSD uses /var/db for this kind of application state, Linux uses /var/lib).
  5. dd byte-count parsing — GNU dd's default summary line format differs from FreeBSD's ("N bytes (X GB, Y GiB) copied, ..." vs "N bytes transferred in ... secs (...)"). The script tries the FreeBSD-style pattern first, then falls back to a GNU-style pattern. This fallback logic also surfaced a separate, more serious bug — see "Key technical issues" below.
  6. Dropped FreeBSD-specific "frozen tape" explanation — the rewind-then-reposition step (rewind to BOT, then mt fsf N forward to the last known-good file, before every run) was partly there to recover from a documented sa(4) driver bug where a failed write leaves the tape "frozen." That bug is FreeBSD/sa(4)-specific; the comments referencing it were removed from the script. The reposition step itself was kept, since resetting tape position explicitly at the start of each run is reasonable practice independent of that bug — it just hasn't been exercised against a real fault on Linux yet, so treat it as unverified-but-retained rather than confirmed-necessary here.
  7. Restore script's no-fsf-between-reads behavior — unchanged. This relies on standard SCSI sequential-access semantics (consuming the filemark's terminating zero-length read leaves a no-rewind device positioned at the start of the next file), which holds the same way on FreeBSD's sa(4) and Linux's st driver — not a FreeBSD peculiarity, so no change was needed.

New on Linux, not ported from FreeBSD: GFS-style local snapshot pruning (see its own section below) — the FreeBSD version never had this and would have grown its snapshot list unbounded.

zfs-tape-backup.sh

#!/bin/bash
#
# zfs-tape-backup.sh (Linux port)
#
# Daily snapshot + incremental zfs send to a no-rewind tape device.
# Writes a FULL send on the first run (or when the full-backup interval
# has elapsed) and an INCREMENTAL send (relative to the previous
# snapshot) on every other run. Each send is appended to the tape as a
# new tape file, since /dev/nst0 does not rewind after writing.
#
# Ported from the FreeBSD version (see ../freebsd/zfs-tape-backup.sh).
# Differences from the FreeBSD original:
#   - Shebang changed from /bin/sh to bash: `set -o pipefail` is not
#     POSIX and is not supported by dash, which is /bin/sh on Debian
#     and Debian-derived distros. Assumed here since that's what the
#     prior Raspberry Pi test host ran - confirm /bin/sh on "seneca"
#     if you're not sure, since bash needs to actually be installed
#     for this shebang to work.
#   - Tape device path uses Linux st(4) naming (/dev/nst0) instead of
#     FreeBSD sa(4) naming (/dev/nsa0).
#   - Date arithmetic uses GNU date syntax instead of BSD date syntax.
#   - State/lock directories use Linux FHS conventions.
#   - Dropped the FreeBSD-specific commentary about the sa(4) driver's
#     "frozen" state, which was the reason this host was originally on
#     FreeBSD before migrating to Linux (see zfs-tape-backup-summary.md).
#     The rewind-then-reposition step is kept anyway, since explicitly
#     resetting tape position at the start of every run is still good
#     practice regardless of platform.
#
# Requires (Debian/Debian-derivative package names - adjust for
# whatever distro "seneca" actually runs):
#   - mt-st       (provides /usr/bin/mt for the Linux st driver)
#   - mailutils or bsd-mailx  (provides the `mail` command used below;
#     you'll also need an MTA - e.g. postfix or msmtp - configured to
#     actually deliver, similar to how DMA was configured on FreeBSD)
#   - zfsutils-linux / OpenZFS - `zfs` CLI is identical to FreeBSD's
#
# Edit the CONFIG section below for your setup, then run this from
# cron (see the example at the bottom of this file).

set -eu -o pipefail

# Block size for writes hitting the tape device. On seneca, `dmesg`
# reports the st driver's block limits as 1 - 16777215 bytes (~16MiB),
# well above 64k - unlike the FreeBSD host's USB bridge, which capped
# single writes at 65536 bytes and required this to stay small. 64k is
# kept here anyway as a known-safe, already-tested value; it could be
# raised for throughput, but hasn't been benchmarked at a larger size.
# zfs send's own output buffer size can exceed this regardless, so we
# still pipe through dd to re-chunk writes to a safe size.
TAPE_BS="64k"

# Capacity tracking. DAT72 is rated at 36GB native (uncompressed) capacity
# (decimal GB, i.e. 36,000,000,000 bytes) - this is the manufacturer spec,
# not a measured value, so treat it as approximate. We track cumulative
# bytes actually written (post-compression, since that's what dd reports)
# and warn once you cross CAPACITY_WARN_PERCENT of that figure.
TAPE_NATIVE_CAPACITY_BYTES=36000000000
CAPACITY_WARN_PERCENT=80

# Email alerting. Uses the system mail(1) command. On Linux this routes
# through whatever MTA you've configured (postfix, msmtp, etc. - unlike
# FreeBSD there's no DMA-in-base default, so this needs to be set up
# explicitly). Set to "" to disable email alerts entirely.
ALERT_EMAIL="me@example.com"

# ---------------------------------------------------------------------
# CONFIG - edit these for your environment
# ---------------------------------------------------------------------
DATASET="tank/minecraft-backup/JJAC_Survival_World-vanilla"   # the ZFS dataset holding the world dir
TAPE="/dev/nst0"                  # no-rewind tape device (Linux st driver)
STATE_DIR="/var/lib/zfs-tape-backup"
STATE_FILE="$STATE_DIR/state"     # tracks last snapshot + tape position
LOG_FILE="/var/log/zfs-tape-backup.log"

# Local snapshot retention (grandfather-father-son / GFS-style). Tape is
# the long-term archive; once a day's snapshot has been safely written
# to tape, it doesn't need to stay on the live pool indefinitely. Only
# ever touches snapshots matching our own naming convention
# ($DATASET@YYYY-MM-DD), so it won't touch snapshots created by
# anything else. Set any tier to 0 to disable it. The just-created
# snapshot is always kept regardless of these settings, since it's the
# base the next day's incremental send is relative to.
KEEP_DAILY=7       # keep this many most-recent daily snapshots
KEEP_WEEKLY=4      # plus one snapshot per ISO week, for this many weeks before that
KEEP_MONTHLY=12    # plus one snapshot per calendar month, for this many months before that
KEEP_YEARLY=0      # plus one snapshot per year, for this many years before that (0 = disabled)
FULL_INTERVAL_DAYS=30             # take a fresh full backup this often
LOCK_FILE="/var/run/zfs-tape-backup.lock"

# ---------------------------------------------------------------------
# Setup
# ---------------------------------------------------------------------
mkdir -p "$STATE_DIR"
touch "$LOG_FILE"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') $*" >> "$LOG_FILE"
}

mail_alert() {
    # $1 = subject, $2 = body
    if [ -z "$ALERT_EMAIL" ]; then
        return 0
    fi
    if echo "$2" | mail -s "$1" "$ALERT_EMAIL"; then
        log "Sent alert email: $1"
    else
        log "WARNING: failed to send alert email: $1"
    fi
}

# Prevent overlapping runs
if [ -e "$LOCK_FILE" ]; then
    log "ERROR: lock file $LOCK_FILE exists, another run may be in progress. Aborting."
    mail_alert "ZFS tape backup did not run on $(hostname)" \
        "Lock file $LOCK_FILE already exists - a previous run may be stuck or crashed. Backup was skipped. Check $LOG_FILE."
    exit 1
fi
trap 'rm -f "$LOCK_FILE"' EXIT
touch "$LOCK_FILE"

TODAY=$(date '+%Y-%m-%d')
SNAP_NAME="${DATASET}@${TODAY}"

# ---------------------------------------------------------------------
# Read previous state, if any
# ---------------------------------------------------------------------
LAST_SNAP=""
LAST_FULL_DATE=""
TAPE_FILE_POS=0
TOTAL_BYTES_WRITTEN=0

if [ -f "$STATE_FILE" ]; then
    # shellcheck disable=SC1090
    . "$STATE_FILE"
fi

# Guard against running twice in one day
if [ "$SNAP_NAME" = "$LAST_SNAP" ]; then
    log "Snapshot $SNAP_NAME already exists / already backed up today. Nothing to do."
    exit 0
fi

# ---------------------------------------------------------------------
# Decide full vs incremental
# ---------------------------------------------------------------------
DO_FULL=0
if [ -z "$LAST_SNAP" ]; then
    DO_FULL=1
    log "No previous state found - performing initial FULL backup."
elif [ -z "$LAST_FULL_DATE" ]; then
    DO_FULL=1
else
    # GNU date accepts "YYYY-MM-DD" directly with -d; no -j/-f needed
    # like on BSD date.
    DAYS_SINCE_FULL=$(( ( $(date -d "$TODAY" '+%s') - $(date -d "$LAST_FULL_DATE" '+%s') ) / 86400 ))
    if [ "$DAYS_SINCE_FULL" -ge "$FULL_INTERVAL_DAYS" ]; then
        DO_FULL=1
        log "Full-backup interval reached ($DAYS_SINCE_FULL days since last full) - performing FULL backup."
    fi
fi

# ---------------------------------------------------------------------
# Reposition the tape to the correct append point
# ---------------------------------------------------------------------
# We never trust that the tape is already sitting wherever the last
# successful run left it. Instead, every run explicitly rewinds to the
# beginning and forward-spaces past exactly the number of known-good
# files ($TAPE_FILE_POS). This makes positioning self-correcting
# regardless of *why* the tape might be in the wrong place, and is
# kept from the FreeBSD original as good practice even though the
# specific sa(4) "frozen state" bug that motivated it there doesn't
# apply to Linux's st driver.
#
# Runs BEFORE any ZFS snapshot is taken, and if it fails, we abort
# loudly rather than risk writing (and overwriting existing backups)
# at the wrong tape position.
log "Repositioning tape: rewind, then forward-space $TAPE_FILE_POS file(s)"
if ! mt -f "$TAPE" rewind >>"$LOG_FILE" 2>&1; then
    log "ERROR: mt rewind failed while repositioning tape. Aborting before touching ZFS."
    mail_alert "ZFS tape backup FAILED on $(hostname)" \
        "mt rewind failed while repositioning the tape before today's backup. No snapshot was taken and nothing was written. Check $LOG_FILE and consider checking the drive/cabling in person."
    exit 1
fi
if [ "$TAPE_FILE_POS" -gt 0 ]; then
    if ! mt -f "$TAPE" fsf "$TAPE_FILE_POS" >>"$LOG_FILE" 2>&1; then
        log "ERROR: mt fsf $TAPE_FILE_POS failed while repositioning tape. Aborting before touching ZFS."
        mail_alert "ZFS tape backup FAILED on $(hostname)" \
            "mt fsf $TAPE_FILE_POS failed while repositioning the tape before today's backup. No snapshot was taken and nothing was written. This could mean the tape doesn't contain as much data as expected (wrong/blank tape loaded?) - check $LOG_FILE and verify the correct tape is in the drive before retrying."
        exit 1
    fi
fi
log "Tape repositioned successfully to file #$TAPE_FILE_POS"

# ---------------------------------------------------------------------
# Take today's snapshot
# ---------------------------------------------------------------------
if ! zfs snapshot "$SNAP_NAME"; then
    log "ERROR: zfs snapshot $SNAP_NAME failed. Aborting."
    mail_alert "ZFS tape backup FAILED on $(hostname)" \
        "zfs snapshot $SNAP_NAME failed - backup did not run. Check $LOG_FILE."
    exit 1
fi
log "Created snapshot $SNAP_NAME"

# ---------------------------------------------------------------------
# Send to tape
# ---------------------------------------------------------------------
DD_STATS=$(mktemp)

if [ "$DO_FULL" -eq 1 ]; then
    log "Sending FULL stream of $SNAP_NAME to $TAPE (tape file #$TAPE_FILE_POS, bs=$TAPE_BS)"
    if ! zfs send "$SNAP_NAME" | dd of="$TAPE" bs="$TAPE_BS" 2>"$DD_STATS"; then
        cat "$DD_STATS" >> "$LOG_FILE"
        log "ERROR: zfs send (full) to $TAPE failed."
        mail_alert "ZFS tape backup FAILED on $(hostname)" \
            "FULL backup of $SNAP_NAME to $TAPE failed. See $LOG_FILE for details.

$(cat "$DD_STATS")"
        rm -f "$DD_STATS"
        zfs destroy "$SNAP_NAME" 2>/dev/null || true
        exit 1
    fi
    SEND_TYPE="FULL"
    LAST_FULL_DATE="$TODAY"
else
    log "Sending INCREMENTAL stream ${LAST_SNAP} -> ${SNAP_NAME} to $TAPE (tape file #$TAPE_FILE_POS, bs=$TAPE_BS)"
    if ! zfs send -i "$LAST_SNAP" "$SNAP_NAME" | dd of="$TAPE" bs="$TAPE_BS" 2>"$DD_STATS"; then
        cat "$DD_STATS" >> "$LOG_FILE"
        log "ERROR: zfs send -i $LAST_SNAP $SNAP_NAME to $TAPE failed."
        mail_alert "ZFS tape backup FAILED on $(hostname)" \
            "INCREMENTAL backup ($LAST_SNAP -> $SNAP_NAME) to $TAPE failed. See $LOG_FILE for details.

$(cat "$DD_STATS")"
        rm -f "$DD_STATS"
        zfs destroy "$SNAP_NAME" 2>/dev/null || true
        exit 1
    fi
    SEND_TYPE="INCREMENTAL"
fi

cat "$DD_STATS" >> "$LOG_FILE"

# Parse the byte count out of dd's summary line, e.g.:
#   "1140916072 bytes transferred in 335.187011 secs (3403819 bytes/sec)"
# (GNU dd's default output format differs slightly: "1140916072 bytes
# (1.1 GB, 1.1 GiB) copied, 335.187 s, 3.4 MB/s" - confirmed on seneca's
# GNU dd via a real run. Both patterns are tried below.
#
# IMPORTANT: each grep|awk here is wrapped in `|| true`. Without it, a
# grep that matches nothing exits 1, and under `pipefail` that makes
# the whole pipeline's exit status 1 even though awk itself exits 0 -
# which under `set -e` kills the script right here, silently, with no
# error logged and no mail_alert (this isn't inside an `if`-guarded
# block). That's exactly what happened tracking this down on seneca:
# the FreeBSD-style pattern below never matches GNU dd's output, so
# without `|| true` this line always aborted the script immediately
# after a successful tape write, before state/tape-index.log/pruning
# ever ran.
BYTES_THIS_RUN=$(grep -oE '^[0-9]+ bytes transferred' "$DD_STATS" | awk '{print $1}') || true
if [ -z "${BYTES_THIS_RUN:-}" ]; then
    # Fall back to GNU dd's default summary format.
    BYTES_THIS_RUN=$(grep -oE '^[0-9]+ bytes' "$DD_STATS" | awk '{print $1}') || true
fi
rm -f "$DD_STATS"
if [ -z "${BYTES_THIS_RUN:-}" ]; then
    log "WARNING: could not parse bytes-written from dd output; capacity tracking for this run skipped."
    BYTES_THIS_RUN=0
fi

TOTAL_BYTES_WRITTEN=$((TOTAL_BYTES_WRITTEN + BYTES_THIS_RUN))

log "Wrote $SEND_TYPE stream for $SNAP_NAME as tape file #$TAPE_FILE_POS ($BYTES_THIS_RUN bytes)"

# ---------------------------------------------------------------------
# Capacity check
# ---------------------------------------------------------------------
CAPACITY_PERCENT=$((TOTAL_BYTES_WRITTEN * 100 / TAPE_NATIVE_CAPACITY_BYTES))
log "Cumulative bytes written to this tape: $TOTAL_BYTES_WRITTEN / $TAPE_NATIVE_CAPACITY_BYTES (${CAPACITY_PERCENT}% of rated native capacity)"
if [ "$CAPACITY_PERCENT" -ge "$CAPACITY_WARN_PERCENT" ]; then
    log "WARNING: tape is at ${CAPACITY_PERCENT}% of its rated native capacity (threshold: ${CAPACITY_WARN_PERCENT}%). Consider rotating in a fresh tape soon."
    mail_alert "ZFS tape backup: tape nearing capacity on $(hostname)" \
        "Tape is at ${CAPACITY_PERCENT}% of its rated native capacity (${TOTAL_BYTES_WRITTEN} / ${TAPE_NATIVE_CAPACITY_BYTES} bytes).
Warning threshold: ${CAPACITY_WARN_PERCENT}%.
Consider rotating in a fresh tape soon. See $STATE_DIR/tape-index.log for the full history on this tape."
fi

# ---------------------------------------------------------------------
# Update state
# ---------------------------------------------------------------------
NEW_TAPE_FILE_POS=$((TAPE_FILE_POS + 1))
cat > "$STATE_FILE" <> "$STATE_DIR/tape-index.log"

log "Backup complete. State updated: LAST_SNAP=$SNAP_NAME TAPE_FILE_POS=$NEW_TAPE_FILE_POS"

# ---------------------------------------------------------------------
# Prune old local snapshots (GFS-style retention)
# ---------------------------------------------------------------------
# Runs after a successful backup only - if the send to tape failed we
# already exited above, so we never prune a snapshot whose tape copy
# doesn't exist yet. See KEEP_DAILY/KEEP_WEEKLY/KEEP_MONTHLY/KEEP_YEARLY
# in the CONFIG section for the retention tiers.
prune_snapshots() {
    local snaps
    snaps=$(zfs list -t snapshot -o name -H -r "$DATASET" 2>/dev/null \
        | grep -E "^${DATASET}@[0-9]{4}-[0-9]{2}-[0-9]{2}$" \
        | sort -t@ -k2 -r)

    if [ -z "$snaps" ]; then
        log "Prune: no matching snapshots found for $DATASET - nothing to do."
        return 0
    fi

    # Each tier is evaluated independently (does this snapshot represent
    # the newest one seen so far in its ISO week / calendar month / year,
    # and is that tier's quota not yet used up?) and the keep decision is
    # the union of all tiers, same approach tools like sanoid use. This
    # deliberately allows tiers to overlap - e.g. if a whole ISO week
    # falls inside the daily retention window, that week's "slot" in the
    # weekly quota is still consumed by a snapshot that was going to be
    # kept anyway. That means actual retention depth from the weekly/
    # monthly/yearly tiers can end up a bit shallower than the raw
    # KEEP_* numbers might suggest, depending on how period boundaries
    # line up with the daily window - but it never prunes something it
    # shouldn't, so it's a safe tradeoff for the simplicity.
    local daily_count=0 weekly_count=0 monthly_count=0 yearly_count=0
    local -A seen_weeks=() seen_months=() seen_years=()
    local destroy_list=""

    while IFS= read -r snap; do
        local snap_date week_key month_key year_key
        local d_keep=0 w_keep=0 m_keep=0 y_keep=0
        snap_date="${snap#*@}"

        if [ "$KEEP_DAILY" -gt 0 ] && [ "$daily_count" -lt "$KEEP_DAILY" ]; then
            d_keep=1
            daily_count=$((daily_count + 1))
        fi

        if [ "$KEEP_WEEKLY" -gt 0 ] && [ "$weekly_count" -lt "$KEEP_WEEKLY" ]; then
            week_key=$(date -d "$snap_date" '+%G-W%V' 2>/dev/null)
            if [ -n "$week_key" ] && [ -z "${seen_weeks[$week_key]:-}" ]; then
                seen_weeks[$week_key]=1
                w_keep=1
                weekly_count=$((weekly_count + 1))
            fi
        fi

        if [ "$KEEP_MONTHLY" -gt 0 ] && [ "$monthly_count" -lt "$KEEP_MONTHLY" ]; then
            month_key=$(date -d "$snap_date" '+%Y-%m' 2>/dev/null)
            if [ -n "$month_key" ] && [ -z "${seen_months[$month_key]:-}" ]; then
                seen_months[$month_key]=1
                m_keep=1
                monthly_count=$((monthly_count + 1))
            fi
        fi

        if [ "$KEEP_YEARLY" -gt 0 ] && [ "$yearly_count" -lt "$KEEP_YEARLY" ]; then
            year_key=$(date -d "$snap_date" '+%Y' 2>/dev/null)
            if [ -n "$year_key" ] && [ -z "${seen_years[$year_key]:-}" ]; then
                seen_years[$year_key]=1
                y_keep=1
                yearly_count=$((yearly_count + 1))
            fi
        fi

        # The snapshot just sent to tape is always kept regardless of the
        # tiers above - it's the base the next day's incremental send
        # will be relative to.
        if [ "$snap" = "$SNAP_NAME" ] || [ "$d_keep" -eq 1 ] || [ "$w_keep" -eq 1 ] || [ "$m_keep" -eq 1 ] || [ "$y_keep" -eq 1 ]; then
            :
        else
            destroy_list="$destroy_list $snap"
        fi
    done <<< "$snaps"

    if [ -z "$destroy_list" ]; then
        log "Prune: all existing snapshots fall within retention policy (daily=$KEEP_DAILY weekly=$KEEP_WEEKLY monthly=$KEEP_MONTHLY yearly=$KEEP_YEARLY) - nothing pruned."
        return 0
    fi

    log "Prune: removing snapshots outside retention policy (daily=$KEEP_DAILY weekly=$KEEP_WEEKLY monthly=$KEEP_MONTHLY yearly=$KEEP_YEARLY):$destroy_list"
    for snap in $destroy_list; do
        if zfs destroy "$snap" 2>>"$LOG_FILE"; then
            log "Prune: destroyed $snap"
        else
            log "WARNING: Prune: failed to destroy $snap - left in place, will retry next run. Check $LOG_FILE for the zfs error."
        fi
    done
}

prune_snapshots

exit 0

# ---------------------------------------------------------------------
# Example cron entry (run as root, daily at 2am):
#   0 2 * * * /usr/local/sbin/zfs-tape-backup.sh
#
# When you swap in a fresh/blank tape, delete $STATE_FILE (but keep
# tape-index.log for your records, or archive it) so the next run
# starts over with a FULL backup at tape file #0 and resets the
# cumulative capacity counter (TOTAL_BYTES_WRITTEN) to 0.
# ---------------------------------------------------------------------

zfs-tape-restore.sh

#!/bin/bash
#
# zfs-tape-restore.sh (Linux port)
#
# Restores a dataset from a tape written by zfs-tape-backup.sh.
# Reads tape files in order starting from the beginning, receiving
# the full stream first and then each incremental in sequence.
#
# Ported from the FreeBSD version (see ../freebsd/zfs-tape-restore.sh).
# Differences from the FreeBSD original:
#   - Shebang changed from /bin/sh to bash, matching zfs-tape-backup.sh
#     (pipefail is not POSIX / not supported by dash).
#   - Tape device path uses Linux st(4) naming (/dev/nst0) instead of
#     FreeBSD sa(4) naming (/dev/nsa0).
#   - INDEX_LOG path matches the Linux STATE_DIR used in the ported
#     backup script (/var/lib/zfs-tape-backup instead of /var/db/...).
#   - The no-explicit-fsf-between-reads behavior is unchanged: this
#     relies on standard SCSI sequential-access semantics (once a read
#     consumes the terminating zero-length read at a filemark, a
#     no-rewind device is positioned at the start of the next file),
#     which holds for Linux's st driver the same as it did for
#     FreeBSD's sa(4) - not a FreeBSD-specific behavior.
#
# Usage: zfs-tape-restore.sh  [num-files]
#
#   target-dataset   e.g. zroot/home/dhw/jjac-survival-world-restore
#                    (must not already exist)
#   num-files        how many tape files to restore (default: all files
#                     recorded in the tape-index.log)
#
# Check /var/lib/zfs-tape-backup/tape-index.log first to see how many
# files are on the tape and what each one is (FULL or INCREMENTAL).
#
# Example:
#   zfs-tape-restore.sh zroot/home/dhw/jjac-survival-world-restore

set -eu -o pipefail

TAPE="/dev/nst0"
TAPE_BS="64k"   # must match the block size used when writing (see zfs-tape-backup.sh)
INDEX_LOG="/var/lib/zfs-tape-backup/tape-index.log"

TARGET="${1:-}"
if [ -z "$TARGET" ]; then
    echo "Usage: $0  [num-files]" >&2
    exit 1
fi

if [ -n "${2:-}" ]; then
    NUM_FILES="$2"
elif [ -f "$INDEX_LOG" ]; then
    NUM_FILES=$(wc -l < "$INDEX_LOG" | tr -d ' ')
else
    echo "No tape-index.log found and no file count given - specify num-files explicitly." >&2
    exit 1
fi

echo "Restoring $NUM_FILES tape file(s) into dataset: $TARGET"
echo "Rewinding tape..."
mt -f "$TAPE" rewind

i=0
while [ "$i" -lt "$NUM_FILES" ]; do
    if [ "$i" -eq 0 ]; then
        echo "Receiving tape file #0 (expected FULL stream)..."
        dd if="$TAPE" bs="$TAPE_BS" | zfs receive "$TARGET"
    else
        echo "Receiving tape file #$i (expected INCREMENTAL stream)..."
        dd if="$TAPE" bs="$TAPE_BS" | zfs receive -F "$TARGET"
        # -F rolls the target back to match the incoming incremental's
        # origin snapshot if needed; safe here since we're restoring
        # in strict order into a dedicated restore target.
    fi

    i=$((i + 1))
    # No explicit fsf needed here: per standard SCSI sequential-access
    # semantics (st(4) on Linux, same as sa(4) on FreeBSD), once a read
    # consumes the terminating zero-length read at a filemark, the
    # no-rewind device is already positioned at the start of the next
    # tape file.
done

echo "Restore complete. Snapshots now present on $TARGET:"
zfs list -t snapshot -r "$TARGET"

Key technical issues hit and fixed

  1. Block size mismatch (si_iosize_max) — FreeBSD-era, historical. The USB SCSI bridge in the DAT72 adapter capped single writes at 65536 bytes (64KB) under FreeBSD's USB/CAM stack. zfs send's own write buffer exceeds this, causing cannot split request errors. Fix: always pipe zfs send through dd bs=64k to re-chunk writes to a safe size. On seneca, dmesg reports the Linux st driver's block limits as 1–16,777,215 bytes (~16MiB) — well above the 64k cap that motivated this fix — so there's headroom to raise TAPE_BS for throughput if desired, though this hasn't been benchmarked at a larger size. TAPE_BS=64k was kept as a known-safe value either way.
  2. Frozen tape state — FreeBSD-era, historical, doesn't apply to Linux. An aborted write (triggered by the block-size issue above) left FreeBSD's sa(4) driver in a "frozen" state (visible in dmesg, not in mt status), refusing further I/O until explicitly cleared via mt rewind. This specific driver bug doesn't exist in Linux's st driver, and was in fact the underlying reason this host migrated off FreeBSD. The rewind-then-reposition step the FreeBSD fix introduced (mt rewind, then mt fsf $TAPE_FILE_POS, at the start of every run) was kept in the Linux port anyway, since explicitly resetting tape position each run is reasonable practice independent of the platform — it just hasn't been exercised against a real fault on seneca yet.
  3. Restore positioning bug — applies to both platforms. The original restore script called mt fsf 1 between reads to advance to the next tape file. This was wrong: once a sequential read consumes the terminating zero-length read at a filemark, the no-rewind device is already positioned at the start of the next file — on both FreeBSD's sa(4) and Linux's st driver, since this is standard SCSI sequential-access behavior, not platform-specific. The extra fsf skipped past the real next file onto nonexistent data, causing an I/O error. Fix: removed the explicit skip; reads now walk file-to-file on their own.
  4. Validating restores against a moving target — applies to both platforms. Diffing a restored copy against the live source directory is unreliable if rsync runs in between, since it compares two different points in time. Fix: diff against the ZFS snapshot's frozen state instead, via the hidden .zfs/snapshot/<name> path. Both full and full+incremental restore chains have been validated this way on both hosts with a clean (no-diff) result.
  5. New, Linux-specific: silent script death after a successful tape write. The first FULL backup on seneca (7.3GB, ~35 min at ~3.5 MB/s) wrote successfully to tape, but the script then died silently immediately afterward, before updating state, tape-index.log, or running the new prune step. Root cause: the dd byte-count parser, grep -oE '^[0-9]+ bytes transferred' "$DD_STATS" | awk '{print $1}', expected FreeBSD-style dd output. GNU dd's actual summary line ("7318552928 bytes (7.3 GB, 6.8 GiB) copied, 2115.92 s, 3.5 MB/s") doesn't contain the word "transferred", so grep matches nothing and exits 1. Under pipefail, a pipeline's exit status is the rightmost non-zero exit code in the chain — here that's grep's 1, even though the following awk exits 0. Since this assignment wasn't wrapped in an if, set -e killed the script on the spot, silently, with no log line and no email alert. Fix: append || true to both parser attempts (FreeBSD-style, then a GNU-style fallback), so a non-matching grep no longer aborts the script — the existing fallback/WARNING logic further down now gets the chance to run and handles a genuinely-unparseable case correctly. Confirmed the fix in isolation with a standalone set -e -o pipefail reproduction before deploying. Because the crash happened after the tape write but before state was persisted, /var/lib/zfs-tape-backup/state had to be reconstructed by hand from the log's actual numbers afterward, so the next run did an INCREMENTAL instead of overwriting tape file #0 with a redundant FULL — verified afterward against zfs list -t snapshot and the reconstructed state.
  6. New, Linux-specific: DKMS module rebuild failure after a kernel update. A routine Debian security update on 6 August 2026 bumped the kernel from 6.12.100+deb13-amd64 to 6.12.101+deb13-amd64 and required a reboot. After rebooting, /tank was empty and the 3:00 AM backup failed cleanly (ERROR: zfs snapshot ... failed, alert email sent as designed — no tape write was attempted, no corruption). See "DKMS / kernel update incident" below for the full root cause and fix.

DKMS / kernel update incident (7 August 2026)

OpenZFS on Debian ships as a DKMS (out-of-tree, built-on-install) kernel module rather than a precompiled one. Root cause: linux-headers-6.12.101+deb13-amd64 wasn't installed — the kernel image package updated ahead of its matching headers package, since the linux-headers-amd64 tracking meta-package (which pulls matching headers automatically for whatever kernel is current) wasn't installed. Without matching headers, DKMS had nothing to build the module against for the new kernel, so zfs.ko only existed for the old 6.12.100 kernel (confirmed via find /lib/modules -iname "zfs.ko*"), and modprobe zfs failed outright on the running 6.12.101 kernel. The tape drive itself was unaffected throughout (st/usb-storage are separate kernel modules, unrelated to ZFS).

Fix:

sudo apt update
sudo apt install linux-headers-$(uname -r)

installing the matching headers triggered zfs-dkms's postinst hook, which rebuilt and installed zfs.ko for the running kernel automatically (Autoinstall of module zfs/2.3.2 ... succeeded). Then:

sudo modprobe zfs
sudo zpool import -a

brought the pool back online with no errors and no data loss — zpool status showed ONLINE, and both existing snapshots (@2026-08-05, @2026-08-06) were intact.

Knock-on effect: the Shockbyte world-data rsync (2:00 AM, upstream of the ZFS snapshot) also failed that morning, since JJAC_Survival_World-vanilla didn't exist as a real mountpoint while the pool was unimported — it had nowhere to sync to. It was re-run manually once the pool was back, before re-running the backup script, so that day's snapshot reflected current data rather than stale pre-outage state.

Prevention: linux-headers-amd64 was confirmed not installed — only the version-pinned linux-headers-6.12.101+deb13-amd64 package was present, which doesn't auto-track future kernels. Installed the meta-package (resolved to version 6.12.101-1, matching the running kernel). Going forward, any apt upgrade that pulls a new kernel image should pull its matching headers in the same transaction, letting DKMS rebuild zfs.ko automatically without manual intervention.

Incremental backup sizes and Shockbyte hibernation (confirmed 6–7 August 2026)

Shockbyte's "hibernation" feature is enabled on this Minecraft server — a documented Shockbyte feature that puts the server to sleep when no players are online, resuming with a short delay when someone connects (source). This explains the very small INCREMENTAL sizes seen so far: 80,248 bytes for 2026-08-06 and 176,128 bytes for 2026-08-07, against a 7.3GB FULL baseline. With the server mostly asleep, there's very little world-state change for ZFS to diff day to day. This also confirms the open question noted under "Not yet re-verified" at setup time — now that the world directory is its own dataset (see "Dataset carve-out" above), incremental sends scale with actual daily churn rather than the whole parent dataset, and at these sizes both the 43.2GB pool and the 36GB tape have a very long runway.

Local snapshot retention (GFS pruning) — added 2026-08-05, Linux-only

The backup script only ever created local ZFS snapshots and never destroyed them, so the live pool's snapshot list would grow unbounded. Since tape is the long-term archive, a snapshot doesn't need to stay on the live pool once it's safely on tape. A prune_snapshots step now runs after every successful backup, using a grandfather-father-son retention scheme (KEEP_DAILY=7, KEEP_WEEKLY=4, KEEP_MONTHLY=12, KEEP_YEARLY=0 by default — tune in the CONFIG section).

Capacity tracking

Email alerting

Debian ships no MTA and no traditional syslog by default, so both had to be set up from scratch on seneca — unlike FreeBSD, where DMA is preconfigured in base. Postfix 3.10.12 is installed and configured as a smarthost relay through Hover (mail.hover.com:465, implicit TLS), replacing DMA. Test send via echo "test" | mail -s "..." me@example.com confirmed end-to-end delivery.

/etc/postfix/main.cf:

relayhost = [mail.hover.com]:465
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_generic_maps = hash:/etc/postfix/generic

/etc/postfix/sasl_passwd holds the Hover credentials ([mail.hover.com]:465 username:password, postmap'd, mode 600).

Hit the same issue the FreeBSD/DMA setup had: the first test bounced with 550 5.7.1 Message Rejected — Hover rejects mail where the envelope sender doesn't match the authenticated account. On FreeBSD this was solved with DMA's MASQUERADE directive; the Postfix equivalent is smtp_generic_maps, which rewrites the envelope/header sender on outbound mail. /etc/postfix/generic:

dhw@seneca.internal me@example.com
@seneca.internal me@example.com

(postmap'd after editing.) Once that was in place, the retry sent cleanly.

Logging: Debian dropped rsyslog from the default install starting with Bookworm (12), carried into Trixie (13), so there's no traditional /var/log/mail.log. Postfix logs go to journald instead: journalctl -u postfix.service (not -u postfix — the systemd unit is postfix.service, and child processes like pickup/qmgr/smtp/bounce are visible under it since they share its cgroup). Note journalctl -g <pattern> only searches the message body, not the postfix/smtp[pid]: prefix, so grepping for postfix that way misses these entries — filter by -u postfix.service and --since/--until instead. Install rsyslog if you want /var/log/mail.log back.

The backup script emails on: send failures (full or incremental), snapshot creation failure, a stuck lock file (possible crashed prior run), a tape repositioning failure, and the capacity warning threshold. Recipient set via ALERT_EMAIL in the script (set to "" to disable).

Security notes (2026-08-05)

Cron

0 3 * * * /usr/local/sbin/zfs-tape-backup.sh

Changed from the FreeBSD script's 0 2 * * * to 0 3 * * * on seneca. The Shockbyte world-data rsync (the job that populates JJAC_Survival_World-vanilla from the game host) also runs at 2:00 AM and takes ~15 minutes to finish. Running the ZFS snapshot at 2:00 AM alongside it risked snapshotting a partially-synced world mid-rsync. 3:00 AM leaves a ~45-minute buffer past the rsync's typical finish time.

Add via sudo crontab -e for root, or /etc/crontab with an explicit username field: 0 3 * * * root /usr/local/sbin/zfs-tape-backup.sh.

Verified on seneca

Not yet re-verified on this host

Open items / possible follow-ups