RunBSD

← Minecraft Backups

Tarsnap Backup (gmktec)

2 August 2026

What this does

Every night at 3AM, gmktec creates a tarsnap archive of the same trimmed Minecraft world directory that feeds the ZFS-to-Tape Backup — a fully independent, encrypted offsite copy in addition to the local disk and tape copies. At 3:30AM, a retention script prunes old archives down to a grandfather-father-son schedule so the archive count doesn't grow forever.

Cron (/etc/crontab on gmktec)

# 3 AM Daily Shockbyte tarsnap backup
0 3 * * * root ( echo "===== $(date) =====" ; tarsnap -c -f shockbyte-minecraft-`date +\%Y\%m\%d-\%H-\%M` /home/dhw/tarsnap-backups/shockbyte/minecraft/JJAC_Survival_World-vanilla ; echo ) >> /var/log/tarsnap-cron.log 2>&1
#
# 3:30 AM Daily Tarsnap archive rotation
30 3 * * * root /usr/local/sbin/tarsnap-rotate.sh --commit >> /var/log/tarsnap-rotate.log 2>&1

Archives are named shockbyte-minecraft-YYYYMMDD-HH-MM — the -HH-MM suffix matters, since tarsnap-rotate.sh (below) parses the date back out of the archive name and needs a delimiter right after the 8-digit date to do it.

Gotcha: tarsnap wouldn't fire from /etc/crontab

Looked like it ran fine — showed up in the cron log every night — but no archive ever appeared. Turned out nothing was actually wrong with the job: /etc/crontab entries mail their output locally by default, and with no MTA configured on that box, any real error just vanished into the void, silently, forever.

Fix: redirect output explicitly so failures are actually visible (now baked into the cron line above via >> /var/log/tarsnap-cron.log 2>&1).

Archive retention: why it was needed

Archives had been deleted manually/ad hoc with no systematic rotation. Besides being a general backup-hygiene gap, this was actively distorting tarsnap --print-stats readings — see Finding 1 below for how an unrelated investigation surfaced this.

tarsnap-rotate.sh

A grandfather-father-son (GFS) retention script, deployed to /usr/local/sbin/tarsnap-rotate.sh.

Retention policy (configurable at the top of the script):

KEEP_DAILY=14      # every archive kept for 14 days
KEEP_WEEKLY=8       # then one per ISO week for 8 more weeks
KEEP_MONTHLY=12     # then one per calendar month for 12 more months
KEEP_YEARLY=2       # then one per calendar year, for 2 more years (0 = no expiration)

How it works:

Deployment:

sudo cp tarsnap-rotate.sh /usr/local/sbin/
sudo chmod +x /usr/local/sbin/tarsnap-rotate.sh
sudo /usr/local/sbin/tarsnap-rotate.sh              # dry run first, always
sudo /usr/local/sbin/tarsnap-rotate.sh --commit      # actually delete

Known limitation: retention windows (not the year/week/month bucketing itself) use fixed approximations (30 days/month, 365 days/year), which can cause an archive right at a boundary to be pruned a day or two earlier than a precise calendar-month/year calculation would suggest. Not considered a practical problem, but worth knowing if exact retention counts ever matter.

Investigation: does MCA Selector's chunk-pruning bloat tarsnap the way it bloats ZFS/tape?

This came up as follow-on work from the ZFS-to-Tape Backup project, after discovering that MCA Selector (run in backup.sh's de-chunking step) was causing massive daily ZFS-incremental churn — nearly as large as a full backup, out of an ~800MB total dataset. That raised the obvious question: is the same thing quietly inflating tarsnap's storage too?

Finding 1 — confirmed root cause on the ZFS side. MCA Selector prunes chunks by filtering on Minecraft's own InhabitedTime NBT field — it reads that value, it doesn't create or modify it. But deleting any chunk from a region file triggers a deFragment() step (confirmed via GitHub issue stack traces referencing MCAFile.java) that repacks remaining chunks to reclaim freed space. Kept chunks' raw byte data is copied verbatim (dos.write(data), confirmed via a GitHub PR snippet) — their actual NBT content is untouched — but the whole region file gets rebuilt as a new file, written to a temp file and atomically swapped in via Files.move. Net effect: any single deleted chunk anywhere in a region file forces the entire file to be rewritten from scratch, dragging every other unrelated chunk in that file along for the ride from any block-level backup system's perspective — even though the actual chunk content didn't change. Confirmed directly against real data: a zfs diff between two snapshots showed literally every changed file (584 of 584) as a matched delete+recreate pair, not an in-place modification.

Finding 2 — tarsnap is not affected the same way. Tarsnap uses content-defined chunking (a rolling-hash approach, confirmed via an academic paper analyzing its algorithm) rather than ZFS's fixed-offset block-level copy-on-write. That means tarsnap can recognize duplicate byte sequences even when they've been relocated within a file — exactly what MCA Selector's rewrite does — where ZFS cannot. Confirmed empirically with tarsnap --print-stats -f <archive> against real archives, all created from MCA-Selector-pruned data:

Conclusion: no changes needed to the tarsnap branch of the backup workflow. MCA Selector still runs before every tarsnap upload as before — it still achieves its storage-cost-reduction goal on this side, since tarsnap's dedup already absorbs the "file looks entirely new" problem that hurt ZFS/tape. (The decision to skip MCA Selector was made only for the ZFS-based tape pipeline on separate hardware, where the block-level rewrite penalty is real — see the ZFS-to-Tape Backup page.)

tarsnap-rotate.sh

#!/bin/sh
#
# tarsnap-rotate.sh
#
# Grandfather-father-son retention for tarsnap archives named like:
#   shockbyte-minecraft-YYYYMMDD-HH-MM
#
# Keeps:
#   - every archive within KEEP_DAILY days (the "sons")
#   - the earliest archive of each ISO week within KEEP_WEEKLY weeks (the "fathers")
#   - the earliest archive of each calendar month within KEEP_MONTHLY months (the "grandfathers")
#   - the earliest archive of each calendar year within KEEP_YEARLY years, or forever if KEEP_YEARLY=0
#
# SAFE BY DEFAULT: runs in dry-run mode (just prints what it would do) unless
# called with --commit. Always review a dry-run before committing, especially
# the first time or after changing the config below.
#
# Usage:
#   ./tarsnap-rotate.sh            # dry run - prints keep/delete decisions, deletes nothing
#   ./tarsnap-rotate.sh --commit   # actually deletes archives marked for deletion

set -eu

# ---------------------------------------------------------------------
# CONFIG - edit these for your retention policy
# ---------------------------------------------------------------------
ARCHIVE_PREFIX="shockbyte-minecraft-"
KEEP_DAILY=14      # days: keep every archive newer than this
KEEP_WEEKLY=8       # weeks: keep one archive/week (oldest in each ISO week) newer than this
KEEP_MONTHLY=12     # months: keep one archive/month (oldest in each calendar month) newer than this
KEEP_YEARLY=2       # years: keep one archive/year (oldest in each calendar year) newer than this; 0 = keep forever
LOG_FILE="/var/log/tarsnap-rotate.log"

MODE="dryrun"
if [ "${1:-}" = "--commit" ]; then
    MODE="commit"
fi

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') $*" >> "$LOG_FILE"
}

TODAY_EPOCH=$(date '+%s')
WORKFILE=$(mktemp)
trap 'rm -f "$WORKFILE"' EXIT

log "=== tarsnap-rotate.sh starting (mode=$MODE) ==="

# ---------------------------------------------------------------------
# Build a table: archive_name epoch age_days year_key week_key month_key
# ---------------------------------------------------------------------
tarsnap --list-archives | grep "^${ARCHIVE_PREFIX}" | while read -r ARCHIVE; do
    DATEPART=$(echo "$ARCHIVE" | sed -E "s/^${ARCHIVE_PREFIX}([0-9]{8})-.*/\1/")

    # Skip anything that doesn't parse as an 8-digit date - don't guess, don't touch it
    case "$DATEPART" in
        [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]) ;;
        *)
            log "WARNING: could not parse date from archive name '$ARCHIVE' - skipping (will not be deleted)"
            continue
            ;;
    esac

    EPOCH=$(date -j -f '%Y%m%d' "$DATEPART" '+%s')
    AGE_DAYS=$(( (TODAY_EPOCH - EPOCH) / 86400 ))
    YEAR_KEY=$(date -j -f '%Y%m%d' "$DATEPART" '+%Y')
    WEEK_KEY=$(date -j -f '%Y%m%d' "$DATEPART" '+%Y-%V')
    MONTH_KEY=$(date -j -f '%Y%m%d' "$DATEPART" '+%Y-%m')

    echo "$ARCHIVE $EPOCH $AGE_DAYS $YEAR_KEY $WEEK_KEY $MONTH_KEY" >> "$WORKFILE"
done

if [ ! -s "$WORKFILE" ]; then
    log "No archives matched prefix '$ARCHIVE_PREFIX' - nothing to do."
    exit 0
fi

# ---------------------------------------------------------------------
# Classify each archive as KEEP or DELETE using awk (grouping/min logic,
# no date parsing needed here - that was already done above)
# ---------------------------------------------------------------------
DECISIONS=$(awk -v keep_daily="$KEEP_DAILY" \
                 -v keep_weekly="$KEEP_WEEKLY" \
                 -v keep_monthly="$KEEP_MONTHLY" \
                 -v keep_yearly="$KEEP_YEARLY" '
{
    name[NR] = $1; epoch[NR] = $2; age[NR] = $3
    ykey[NR] = $4; wkey[NR] = $5; mkey[NR] = $6
    n = NR

    # Track the earliest (min epoch) archive seen so far for each bucket
    if (!(ykey[NR] in y_min_epoch) || epoch[NR] < y_min_epoch[ykey[NR]]) {
        y_min_epoch[ykey[NR]] = epoch[NR]; y_min_idx[ykey[NR]] = NR
    }
    if (!(wkey[NR] in w_min_epoch) || epoch[NR] < w_min_epoch[wkey[NR]]) {
        w_min_epoch[wkey[NR]] = epoch[NR]; w_min_idx[wkey[NR]] = NR
    }
    if (!(mkey[NR] in m_min_epoch) || epoch[NR] < m_min_epoch[mkey[NR]]) {
        m_min_epoch[mkey[NR]] = epoch[NR]; m_min_idx[mkey[NR]] = NR
    }
}
END {
    for (i = 1; i <= n; i++) {
        keep = 0; reason = ""

        if (age[i] <= keep_daily) { keep = 1; reason = "daily" }
        else if (age[i] <= (keep_weekly * 7) && w_min_idx[wkey[i]] == i) { keep = 1; reason = "weekly" }
        else if (age[i] <= (keep_monthly * 30) && m_min_idx[mkey[i]] == i) { keep = 1; reason = "monthly" }
        else if ((keep_yearly == 0 || age[i] <= (keep_yearly * 365)) && y_min_idx[ykey[i]] == i) { keep = 1; reason = "yearly" }

        if (keep) {
            print "KEEP " name[i] " (" reason ", age=" age[i] "d)"
        } else {
            print "DELETE " name[i] " (age=" age[i] "d, no bucket match)"
        }
    }
}
' "$WORKFILE")

echo "$DECISIONS"
log "$(echo "$DECISIONS" | grep -c '^KEEP') archives to keep, $(echo "$DECISIONS" | grep -c '^DELETE') to delete"

if [ "$MODE" = "dryrun" ]; then
    echo ""
    echo "(dry run - nothing deleted. Re-run with --commit to actually delete the above.)"
    log "Dry run complete. Re-run with --commit to apply."
    exit 0
fi

# ---------------------------------------------------------------------
# Commit mode: actually delete
# ---------------------------------------------------------------------
echo "$DECISIONS" | grep '^DELETE' | awk '{print $2}' | while read -r ARCHIVE; do
    log "Deleting archive: $ARCHIVE"
    if tarsnap -d -f "$ARCHIVE" >>"$LOG_FILE" 2>&1; then
        log "Successfully deleted: $ARCHIVE"
    else
        log "ERROR: failed to delete archive: $ARCHIVE"
    fi
done

log "=== tarsnap-rotate.sh finished (mode=$MODE) ==="

# ---------------------------------------------------------------------
# Example cron entry (run as root, daily at 3:30am - after the 3am backup):
#   30 3 * * * /usr/local/sbin/tarsnap-rotate.sh --commit >> /var/log/tarsnap-rotate.log 2>&1
# ---------------------------------------------------------------------

Open items / next steps