RunBSD

← Minecraft Backups

backup.sh & DVD Archiving

15 July 2026 (updated 8 August 2026)

What this does

Once shockbyte_download_backup.py (see the Shockbyte Backup Download page) finishes pulling down the nightly instance.zip, it automatically runs backup.sh (via /bin/zsh) on the same Mac mini — its output lands in the same stdout.log/stderr.log the LaunchAgent already writes. To skip this step entirely, set POST_PROCESS_SCRIPT = None near the top of shockbyte_download_backup.py.

Step by step, as of the 8 August 2026 update:

  1. Unzips instance.zip into a scratch folder, then deletes the zip.
  2. Runs the world integrity checks (size/region-count trend and the level.dat Data.Time check — see below) against the raw, just-unzipped world, before anything below has a chance to shrink it.
  3. Strips out files that don't belong in a world backup: .cache, .shockbyte, crash-reports, debug, libraries, logs, versions, server.jar, server-icon.png, world/paper-world.yml, world/session.lock, world/DIM-1, and world/datapacks.
  4. Re-zips the cleaned (but no longer chunk-pruned) world as jswv.zip. This is what iCloud gets — gmktec and seneca read the unzipped $EXTRACT_DIR directly, not the zip. Chunk pruning via mcaselector now only happens on a separate clone used for the DVD burn — see "DVD burn cadence & DVD-only pruning" below.
  5. Runs an rsync dry-run delete-count check, then the real rsync --delete, to gmktec, authenticating with a dedicated, passphrase-less SSH key (see "SSH authentication" below). On success, SSHes into gmktec to trigger forward-to-batcave.sh, which forwards gmktec's own already-current raw copy on to a ZFS dataset on batcave — see "batcave: from an SMB mount to a gmktec-forwarded ZFS mirror" below for why this replaced a direct Mac→batcave step entirely.
  6. Same dry-run check, then a real rsync, to seneca (the tape-backup host — see the ZFS-to-Tape Backup page). Each host gets its own dry-run check and its own delete-count history file.
  7. On DVD burn days only (weekly, not nightly — see below): clones the cleaned world, chunk-prunes the clone via mcaselector, builds an ISO from it, and burns it if all pre-checks pass.
  8. Copies jswv.zip into a dated folder (YYYY-MM-DD-HH-MM) in iCloud Drive, then runs GFS rotation to prune old dated folders there.
  9. On restore-check days only (weekly): runs the lightweight restore verification against the latest iCloud copy.

There is no same-machine local dated-folder copy, and no direct Mac→batcave step of any kind — see "What changed" below for both.

If backup.sh fails, shockbyte_download_backup.py exits with code 4 (rather than the 0 it'd otherwise return) so a failure here is distinguishable in logs/monitoring from a successful download — see the Shockbyte Backup Download page for the full exit-code list.

What changed (8 August 2026)

Two structural changes since the 5 August rewrite, both aimed at removing fragility rather than adding new checks:

Both changes were prompted by real friction with the previous approach, not a scheduled review pass this time. The 5 August rewrite (alerting, the three integrity checks, weekly-not-nightly DVD burns, removing the local-disk copy) is still the larger piece of history here — see the "Fan-out destinations" and other sections below for how that combines with this update.

One loose end not addressed in this pass: mail_alert()'s ALERT_EMAIL in backup.sh and the equivalent constants in forward-to-batcave.sh and zfs-snapshot-prune.sh (the two new companion scripts) are set independently, and — before redaction for this site — pointed at two different-looking addresses on the same underlying account. Worth double-checking they're intentionally distinct or should be unified, same open question the original TODO in backup.sh already flagged for zfs-tape-backup.sh's separate ALERT_EMAIL.

Also worth a note, not a fix: a couple of the alert message bodies embedded in the script below (check_world_integrity's and check_level_dat_time's anomaly alerts, and the unzip-failure alert) still say "all seven copies"/"none of the seven copies," left over from before the local-disk copy was removed on 5 August — while prepare_dvd_clone's own alert already says "six." Harmless (it's just message text, not logic), but a stray reminder that the copy-count prose drifted in a couple of spots when destinations were removed.

Fan-out destinations

As of this update, a given night's world data can end up in up to six places, most tracing back through this one script:

(A sixth branch, tarsnap on gmktec, is downstream of the gmktec copy above rather than a destination backup.sh writes to directly — see the Tarsnap Backup page.)

SSH authentication

Replaced 8 August 2026. Every rsync/SSH call to gmktec or seneca now authenticates with a dedicated, passphrase-less key ($SSH_KEY, currently ~/.ssh/backup_automation), passed explicitly (-e "ssh -i $SSH_KEY" for rsync, -i "$SSH_KEY" for the plain SSH trigger call to forward-to-batcave.sh) — there's no ssh-agent involved in this pipeline at all anymore, and ensure_ssh_agent_key() has been removed from the script entirely.

This deliberately isn't David's general-purpose personal key: a narrower-scoped, automation-only key limits the blast radius of this particular Mac being compromised (it's also used for regular web browsing, making it the most exposed machine in the pipeline), and being passphrase-less rather than agent-loaded sidesteps the previous failure mode outright rather than self-healing around it.

History, for context: the previous approach used an agent-loaded, Keychain-backed personal key (ssh-add --apple-use-keychain), which decrypts a key into whatever agent process happens to be running and keeps it there only in that process's memory — a reboot (or logout/login, or the agent crashing) starts a new, empty agent with nothing to re-add the key automatically. This was confirmed as the cause of a real failure: the 2026-08-04 02:03 run failed its rsync with Permission denied (publickey,keyboard-interactive) after the Mac had been rebooted earlier that evening. The 5 August fix was ensure_ssh_agent_key(), a pre-flight self-healing check. The 8 August fix replaced the whole mechanism instead — a passphrase-less key has nothing to "lose" on reboot, so there's no longer a failure mode to check for or self-heal from here.

Alerting

backup.sh now has its own mail_alert() function, but it doesn't reuse zfs-tape-backup.sh's approach: that script's mail_alert() relies on FreeBSD's DMA (or, on seneca, Postfix — see the ZFS-to-Tape Backup page), neither of which exists on this Mac mini. Instead, since the LaunchAgent already runs inside the GUI session, backup.sh drives the already-signed-in Mail.app account via AppleScript rather than standing up a new SMTP relay on the Mac.

The subject and body are both escaped (backslashes and double quotes) before being interpolated into the AppleScript string literals, so a failure message containing either character can't break out of the script. If osascript itself fails, that failure is logged (not alerted — there'd be nowhere left to send it). The message body also now appends the hostname, timestamp, and log path automatically (mail_alert()'s full_body), so every alert is self-describing without each call site needing to repeat that context.

Alerts fire on: unzip failure, zip failure, rsync failure (gmktec and seneca independently), the forward-to-batcave.sh SSH trigger failing (only checked if the gmktec rsync itself succeeded — a failed gmktec rsync skips the trigger rather than risk forwarding stale data to batcave), DVD clone failure, DVD burn failure, and iCloud copy failure — plus the "possible anomaly" alerts from the integrity checks below, which are advisory rather than failure alerts. There's no longer a ssh-agent reload alert, since there's no agent step left to fail (see "SSH authentication" above), and no batcave mount/copy alerts, since the Mac no longer touches batcave directly (a forward-to-batcave.sh failure alerts instead — see above).

World integrity checks

Three complementary checks, all measuring the raw unzipped world before file-stripping (step 2 above runs right after unzip, before step 3) so pruning further down can't swamp the signal. None of these block the backup — they're anomaly detectors, not classifiers, since none of them can perfectly distinguish "a bad plugin nuked the world" from "a player legitimately terraforming a huge area." The right behavior is to alert loudly and let a human glance at it.

  1. Size / region-file-count trend. Measures world/'s total size (du -sk) and .mca region-file count, logs both to a rolling history CSV, and flags (with an email alert) if today's figure is more than INTEGRITY_DROP_PCT (currently 15%, a placeholder) below the trailing INTEGRITY_WINDOW-night (14) average, once at least INTEGRITY_MIN_HISTORY (5) nights of history exist.
  2. rsync dry-run delete-count. Since backup.sh already rsync --deletes to both gmktec and seneca, a rsync -an --delete -i dry-run first is a near-free way to see "tonight wants to delete N files that existed yesterday" — a direct signature of a mass deletion. Verified directly against this Mac's rsync (openrsync, protocol 29): --stats has no delete counter at all, but -i (itemize-changes) prefixes each would-be deletion with a clean, greppable *deleting line regardless of file vs. now-empty directory. Tracked as a trend (flags a jump above the trailing average, not a flat threshold) with separate history files per host, and only flags if the count is both a big jump (RSYNC_DELETE_JUMP_PCT, 50%, placeholder) and at least RSYNC_DELETE_MIN_ABS (10) files, to avoid noise at tiny counts.
  3. level.dat Data.Time check. The most precise of the three. Data.Time is a Minecraft NBT tick counter documented to always increase and never reset during normal play (Java Edition level format – Minecraft Wiki). Unlike the two trend-based checks above, this isn't a trend — a rollback or fresh-world swap shows up as tonight's value simply failing to be greater than last night's, full stop. Parsed via a small LevelDatTime.java helper that lives alongside backup.sh and needs no separate compile step on JDK 11+ (single-file source-code execution) — java is already a hard dependency here via mcaselector. On an older JDK, compile it once with javac LevelDatTime.java and the script will still find the resulting .class file in the same directory.

All three checks' thresholds (INTEGRITY_DROP_PCT, RSYNC_DELETE_JUMP_PCT, RSYNC_DELETE_MIN_ABS) are explicitly placeholders in the script, pending a few weeks of real history to tune against.

batcave: from an SMB mount to a gmktec-forwarded ZFS mirror

Replaced 8 August 2026. Previously, backup.sh mounted a batcave SMB share directly from the Mac (retrying up to 5 times, 10s apart, with network diagnostics logged before each attempt) using a hardcoded username/password in the script, then copied jswv.zip into a dated folder there and GFS-rotated it the same way as iCloud. That whole path is gone.

New architecture: gmktec already has its own current raw copy of the world (from backup.sh's rsync in step 5 above) sitting on a much faster gmktec↔batcave network segment than the Mac has. Instead of the Mac pushing a zip over SMB, backup.sh SSHes into gmktec (using the dedicated key from "SSH authentication" above) right after that rsync succeeds, to trigger a new companion script:

Both companion scripts are triggered directly by the step that produces fresh data (forward-to-batcave.sh by backup.sh's gmktec rsync succeeding; zfs-snapshot-prune.sh by forward-to-batcave.sh's own rsync succeeding) rather than run from their own cron jobs on a fixed schedule — the same reasoning already used for why seneca's tape-backup cron moved off 2am: don't race a job you don't control the finish time of. If the gmktec rsync fails, the trigger to forward-to-batcave.sh is skipped entirely, so a bad night's data on the Mac can't also propagate to batcave.

This removes the SMB mount, the hardcoded plaintext NAS credentials that came with it, and a whole class of macOS TCC/Full-Disk-Access fragility around SMB network-volume access that a plain outbound SSH connection doesn't run into at all. batcave's retention also moves from GFS-rotated dated folders (shared logic with iCloud) to native ZFS snapshot pruning — a more natural fit for a dataset that's meant to exist as a versioned, snapshotted mirror rather than a pile of zips.

GFS rotation for dated backup folders

Mirrors the retention policy already deployed for tarsnap (tarsnap-rotate.sh: KEEP_DAILY=14, KEEP_WEEKLY=8, KEEP_MONTHLY=12, KEEP_YEARLY=2 — see the Tarsnap Backup page), applied here to plain dated folders (YYYY-MM-DD-HH-MM). As of 8 August 2026, this only applies to iCloud — batcave moved to native ZFS snapshot pruning instead (see above), so rotate_dated_dirs() is now only ever called once, against $ICLOUD_DIR.

Known issue, still being investigated: a production run on 2026-08-03 (launchd-triggered via launchctl kickstart) saw the rotation function find zero entries immediately after the preceding mkdir+cp into the same folder had already succeeded with no error — confirmed afterward, by hand, that the folders genuinely existed. Leading theory is a same-run visibility lag (e.g. iCloud Drive's FileProvider layer not yet reflecting a just-written directory to a readdir/glob call), not a permissions issue. A temporary diagnostic block still logs what [ -d ], a null-glob-qualified glob, and ls -la each see on the run's own just-written folder specifically, retrying a few times with a short sleep before the real rotation logic runs — this is meant to confirm or rule out the theory, and is safe to remove once the cause is confirmed either way. Now that batcave no longer runs through this same function, this diagnostic only ever fires against iCloud, which should narrow down whether the FileProvider theory holds up once more data comes in.

Weekly restore verification

The "0" in 3-2-1-1-0 — confidence that a backup you'd actually reach for in an emergency still works, not just that it exists. Runs weekly (RESTORE_CHECK_DAY, currently Sunday), not nightly — this is meant to catch slow corruption, not repeat the same check every night — against the most recent iCloud copy:

  1. unzip -t — catches truncated/corrupted archives.
  2. Extracts the zip and confirms world/level.dat exists.
  3. Parses level.dat as valid NBT (reusing the same LevelDatTime.java helper from the integrity checks above) — catches garbled world data that a structurally-valid zip could still contain.

Non-fatal — alerts on any failure but never blocks the backup — and always cleans up its scratch extraction directory whether it passes or fails. If this "lightweight" approach ever proves insufficient, the "heavyweight" alternative discussed is spinning up an actual Minecraft server against the extracted world and watching for errors.

DVD burn cadence & DVD-only pruning

Two related changes, both carried over unchanged from the 5 August rewrite:

On burn days, the same three pre-checks as before still gate the actual burn:

  1. Size check — the ISO must be under MAX_SIZE (currently 4,500,000,000 bytes, sized for a standard 4.7GB DVD; bump this if burning to a DVD-DL, Blu-ray, or M-Disc).
  2. Burner check — hdiutil burn -list must show an actual optical drive attached.
  3. Media check — drutil status must not report "No Media Inserted".

Any of those failing just skips the burn with a log line — not treated as an error. A successful burn deletes the ISO afterward; a failed burn call keeps the ISO around so it can be retried by hand.

Safety guards

Two safety guards remain unchanged from earlier versions of backup.sh: it hard-stops before running any cleanup if unzip fails, and it guards against an unset EXTRACT_DIR (: "${EXTRACT_DIR:?...}") before any rm -rf runs against it, to avoid deleting the wrong directory. The final rm -rf on the extract directory retries up to 3 times (2s apart) to absorb an intermittent "directory not empty" race with Spotlight/Finder regenerating .DS_Store files. The old "log and exit on persistent failure" behavior for a stuck SMB mount no longer applies, since there's no SMB mount left on the Mac's side at all (see "batcave" above) — a batcave failure now surfaces as a forward-to-batcave.sh alert instead, without blocking or retrying anything on the Mac itself.

Dependencies

backup.sh

#!/bin/zsh

ROOT_PATH="/Users/david/scripts/shockbyte_dl"
DL_PATH="/Users/david/scripts/shockbyte_dl/incoming"
ZIP_FILE="${DL_PATH}/instance.zip"
NEW_ZIP="${DL_PATH}/jswv.zip"
EXTRACT_DIR="${DL_PATH}/jswv"
ISO_FILE="${ROOT_PATH}/jswv.iso"
MAX_SIZE=4500000000
OK_TO_BURN=1
GMKTEC_HOST="dhw@gmktec"
GMKTEC_PATH="/home/dhw/tarsnap-backups/shockbyte/minecraft/JJAC_Survival_World-vanilla"
# Shared by gmktec and seneca -- same keypair, added to both hosts'
# authorized_keys. Dedicated, passphrase-less key used only for this
# automation (not David's general-purpose personal key) -- deliberately
# narrower in scope than that key, since this Mac is the machine most
# likely to be compromised (it's also used for regular web browsing).
# Passphrase-less rather than agent-loaded on purpose: this Mac's own
# ssh-agent previously lost its loaded identity on every reboot (Keychain
# only re-decrypts a key into whatever agent process happens to be
# running at the time), which is exactly the failure mode this sidesteps
# -- no agent involved at all, so there's nothing to lose on reboot.
SSH_KEY="/Users/david/.ssh/backup_automation"
# seneca is the new Linux host for the tape drive (the FreeBSD box it used
# to run on had USB/CAM stack problems with the tape drive's host
# controller). backup.sh only keeps this rsynced mirror current -- the
# actual tape writes happen via a separate script running on seneca
# itself, same division of labor as tarsnap-rotate.sh on gmktec.
SENECA_HOST="dhw@seneca"
SENECA_PATH="/tank/minecraft-backup/JJAC_Survival_World-vanilla"
# batcave no longer gets an SMB mount + cp from the Mac at all. gmktec
# already has its own up-to-date raw copy of the world (from the rsync
# leg above) sitting on a much faster gmktec<->batcave network segment
# (both 2.5GbE) than the Mac's own 1GbE port -- so instead of the Mac
# pushing a zip over SMB, gmktec forwards its already-local raw copy to
# a ZFS dataset on batcave directly, triggered over SSH right after the
# gmktec rsync above succeeds. This also removes the SMB mount entirely,
# along with the macOS TCC/Full-Disk-Access fragility that came with it.
GMKTEC_FORWARD_SCRIPT="/usr/local/sbin/forward-to-batcave.sh"
ICLOUD_DIR="/Users/david/Library/Mobile Documents/com~apple~CloudDocs/minecraft"
LOGFILE="/Users/david/scripts/shockbyte_dl/logs/backup_script.log"
# Captured once so every dated folder from this run (local, iCloud, batcave)
# gets the same name, and so a matching mkdir/cp pair can't land in different
# minutes and miss each other.
RUN_TS="$(date +"%Y-%m-%d-%H-%M")"

log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $1" >> "$LOGFILE" }

# TODO: confirm this matches (or should match) the address zfs-tape-backup.sh
# alerts to, for consistency across the pipeline.
ALERT_EMAIL="me@example.com"

# zfs-tape-backup.sh's mail_alert() relies on FreeBSD's dma, which doesn't
# exist here. This Mac mini's LaunchAgent runs in the GUI session (per the
# SSH_AUTH_SOCK note in the README), so we can drive the already-signed-in
# Mail.app account via AppleScript instead of standing up a new SMTP relay.
mail_alert() {
    local subject="$1"
    local body="$2"
    local full_body
    full_body=$(printf '%s\n\nHost: %s\nTime: %s\nLog:  %s' \
        "$body" "$(hostname)" "$(date '+%Y-%m-%d %H:%M:%S')" "$LOGFILE")

    # Escape backslashes and double quotes so $subject/$full_body can't break
    # out of the AppleScript string literals below.
    local as_subject="${subject//\\/\\\\}"
    as_subject="${as_subject//\"/\\\"}"
    local as_body="${full_body//\\/\\\\}"
    as_body="${as_body//\"/\\\"}"

    if ! osascript <<APPLESCRIPT
tell application "Mail"
    set newMessage to make new outgoing message with properties {subject:"backup.sh: ${as_subject}", content:"${as_body}", visible:false}
    tell newMessage
        make new to recipient at end of to recipients with properties {address:"${ALERT_EMAIL}"}
        send
    end tell
end tell
APPLESCRIPT
    then
        log "WARNING: mail_alert failed to send via Mail.app (subject: $subject)"
    fi
}

# ---- World integrity check (size / region-file-count trend) ----
# A live world essentially never shrinks on its own -- a big drop signals a
# rollback, a bad plugin, or a partial wipe upstream, before it gets copied
# to all seven destinations. This only measures world/, not the whole
# extract dir, so libraries/logs/versions/etc. (removed a few lines down)
# can't add noise. It's an anomaly detector, not a classifier: it alerts
# loudly once enough history exists but never blocks the run, since it
# can't tell "world got nuked" from "player terraformed a huge area."
INTEGRITY_HISTORY_FILE="/Users/david/scripts/shockbyte_dl/logs/world_size_history.csv"
INTEGRITY_MIN_HISTORY=5      # nights of history needed before flagging anomalies
INTEGRITY_WINDOW=14          # trailing nights averaged for comparison
# TODO: placeholder until a few weeks of real history exist to tune this against.
INTEGRITY_DROP_PCT=15        # flag if today is more than this % below the trailing average

check_world_integrity() {
    local size_bytes region_count
    size_bytes=$(du -sk "$EXTRACT_DIR"/world 2>/dev/null | awk '{print $1 * 1024}')
    region_count=$(find "$EXTRACT_DIR"/world/dimensions/minecraft -type f -name "*.mca" 2>/dev/null | wc -l | tr -d ' ')

    if [ -z "$size_bytes" ] || [ -z "$region_count" ] || [ "$region_count" -eq 0 ]; then
        log "WARNING: could not measure world size/region count for integrity check (size=$size_bytes, regions=$region_count). Skipping."
        return
    fi

    log "world integrity: size=${size_bytes} bytes, region files=${region_count}"

    touch "$INTEGRITY_HISTORY_FILE"
    local history_count
    history_count=$(wc -l < "$INTEGRITY_HISTORY_FILE" | tr -d ' ')

    if [ "$history_count" -ge "$INTEGRITY_MIN_HISTORY" ]; then
        local avg_size avg_count size_drop_pct count_drop_pct size_flagged count_flagged
        avg_size=$(tail -n "$INTEGRITY_WINDOW" "$INTEGRITY_HISTORY_FILE" | awk -F',' '{sum+=$2; n++} END {if (n>0) printf "%.0f", sum/n; else print 0}')
        avg_count=$(tail -n "$INTEGRITY_WINDOW" "$INTEGRITY_HISTORY_FILE" | awk -F',' '{sum+=$3; n++} END {if (n>0) printf "%.0f", sum/n; else print 0}')

        size_drop_pct=$(awk -v cur="$size_bytes" -v avg="$avg_size" 'BEGIN { if (avg > 0) printf "%.1f", (avg - cur) / avg * 100; else print 0 }')
        count_drop_pct=$(awk -v cur="$region_count" -v avg="$avg_count" 'BEGIN { if (avg > 0) printf "%.1f", (avg - cur) / avg * 100; else print 0 }')
        size_flagged=$(awk -v d="$size_drop_pct" -v t="$INTEGRITY_DROP_PCT" 'BEGIN { print (d >= t) ? 1 : 0 }')
        count_flagged=$(awk -v d="$count_drop_pct" -v t="$INTEGRITY_DROP_PCT" 'BEGIN { print (d >= t) ? 1 : 0 }')

        if [ "$size_flagged" -eq 1 ] || [ "$count_flagged" -eq 1 ]; then
            log "WARNING: possible world anomaly -- size ${size_drop_pct}% below trailing ${INTEGRITY_WINDOW}-night average (${avg_size} bytes), region count ${count_drop_pct}% below average (${avg_count} files)."
            mail_alert "possible world anomaly" "Tonight's extracted world looks smaller than usual, which can mean a rollback, a bad plugin, or a partial wipe upstream (before Shockbyte even made its own backup).

Size: ${size_bytes} bytes (${size_drop_pct}% below the trailing ${INTEGRITY_WINDOW}-night average of ${avg_size} bytes)
Region files: ${region_count} (${count_drop_pct}% below the trailing average of ${avg_count} files)

This is just an anomaly detector -- it can't tell a corrupted world from a player legitimately terraforming a huge area. The backup ran normally and all seven copies will still be made from tonight's data; take a look when you get a chance."
        fi
    else
        log "world integrity: only ${history_count} night(s) of history so far, need ${INTEGRITY_MIN_HISTORY} before flagging anomalies."
    fi

    echo "$(date '+%Y-%m-%d'),${size_bytes},${region_count}" >> "$INTEGRITY_HISTORY_FILE"
}

# ---- SSH agent pre-flight check (gmktec + seneca) ----
# Removed 2026-08-08. $SSH_KEY is now a dedicated, passphrase-less
# key used only for this automation, so there's no ssh-agent/Keychain
# step in the loop at all anymore -- every rsync/ssh call below passes
# `-i "$SSH_KEY"` explicitly and authenticates directly. This was
# the whole point: the old approach (an agent-loaded, Keychain-backed
# identity) lost its loaded key on every reboot, since Keychain only
# re-decrypts a key into whatever agent process is running at the time --
# confirmed as the cause of the 2026-08-04 02:03 run's rsync failure. A
# passphrase-less key sidesteps that failure mode entirely rather than
# self-healing around it.

# ---- rsync dry-run delete-count check ----
# Both gmktec and seneca now sync with real --delete, so this dry-run is a
# genuine preview of what's about to happen a few lines later, not just a
# hypothetical -- it tracks the count as a trend and flags a spike above
# the trailing average rather than a flat threshold, same reasoning as the
# world-size check (mcaselector-driven pruning only ever touches the DVD
# clone, so day-to-day deletions here should normally be small/upstream-
# driven, not routine).
#
# Verified directly against this Mac's rsync (openrsync, protocol 29):
# `--stats` has no delete counter at all ("Number of files" is just the
# source file count, unrelated to deletions), but `-i` (itemize-changes)
# prefixes each would-be deletion with a clean, greppable "*deleting "
# regardless of whether it's a file or now-empty directory.
#
# Parameterized so gmktec and seneca can share the same logic with
# separate history files -- call once per host.
RSYNC_DELETE_HISTORY_FILE="/Users/david/scripts/shockbyte_dl/logs/gmktec_delete_history.csv"
SENECA_DELETE_HISTORY_FILE="/Users/david/scripts/shockbyte_dl/logs/seneca_delete_history.csv"
RSYNC_DELETE_MIN_HISTORY=5     # nights of history needed before flagging spikes
RSYNC_DELETE_WINDOW=14         # trailing nights averaged for comparison
# TODO: both placeholders until real history exists to tune against.
RSYNC_DELETE_JUMP_PCT=50       # flag if tonight's count is this much higher than the trailing average...
RSYNC_DELETE_MIN_ABS=10        # ...but only if it's also at least this many files (avoids noise at tiny counts)

check_rsync_delete_count() {
    local host="$1"
    local remote_path="$2"
    local history_file="$3"
    local label="$4"

    local dry_run_output rsync_exit delete_count
    dry_run_output=$(rsync -an --delete -i -e "ssh -i $SSH_KEY" "$EXTRACT_DIR"/ "${host}:${remote_path}/" 2>&1)
    rsync_exit=$?
    if [ "$rsync_exit" -ne 0 ]; then
        log "WARNING: rsync dry-run failed (exit ${rsync_exit}) for $label, skipping delete-count check."
        return
    fi

    delete_count=$(echo "$dry_run_output" | grep -c '^\*deleting ')

    log "rsync dry-run ($label): would delete ${delete_count} file(s)/folder(s)."

    touch "$history_file"
    local history_count
    history_count=$(wc -l < "$history_file" | tr -d ' ')

    if [ "$history_count" -ge "$RSYNC_DELETE_MIN_HISTORY" ]; then
        local avg_count jump_pct jump_flagged
        avg_count=$(tail -n "$RSYNC_DELETE_WINDOW" "$history_file" | awk -F',' '{sum+=$2; n++} END {if (n>0) printf "%.0f", sum/n; else print 0}')
        jump_pct=$(awk -v cur="$delete_count" -v avg="$avg_count" 'BEGIN { if (avg > 0) printf "%.1f", (cur - avg) / avg * 100; else if (cur > 0) print 100; else print 0 }')
        jump_flagged=$(awk -v d="$jump_pct" -v t="$RSYNC_DELETE_JUMP_PCT" -v c="$delete_count" -v m="$RSYNC_DELETE_MIN_ABS" 'BEGIN { print (d >= t && c >= m) ? 1 : 0 }')

        if [ "$jump_flagged" -eq 1 ]; then
            log "WARNING: $label delete-count jumped ${jump_pct}% above trailing ${RSYNC_DELETE_WINDOW}-night average (${avg_count} -> ${delete_count})."
            mail_alert "$label dry-run delete count spiked" "A dry-run rsync -i --delete against $label reports ${delete_count} file(s)/folder(s) would be deleted tonight -- ${jump_pct}% above the trailing ${RSYNC_DELETE_WINDOW}-night average of ${avg_count}.

The real rsync a few lines below also uses --delete now, so this isn't just hypothetical -- whatever this dry-run reports is about to actually happen. Some growth night-to-night is normal, but a jump this size can mean a rollback, a bad plugin, or a partial wipe upstream. This check does not block the sync, it only warns -- worth a look."
        fi
    else
        log "$label delete-count check: only ${history_count} night(s) of history so far, need ${RSYNC_DELETE_MIN_HISTORY} before flagging spikes."
    fi

    echo "$(date '+%Y-%m-%d'),${delete_count}" >> "$history_file"
}

# ---- level.dat Data.Time check ----
# Data.Time is a Minecraft NBT tick counter documented to always increase
# and never reset during normal play (Minecraft Wiki: Java Edition level
# format -- https://minecraft.wiki/w/Java_Edition_level_format -- worth
# re-confirming against the actual level.dat before relying on this long
# term). Unlike the two trend-based checks above, this isn't a trend: a
# rollback or fresh-world swap shows up as tonight's value failing to be
# *greater* than last night's, full stop.
#
# LevelDatTime.java lives alongside this script and needs no separate
# compile step on JDK 11+ (single-file source-code execution) -- java is
# already a hard dependency here via mcaselector. If the Mac's JDK predates
# 11, compile it once with `javac LevelDatTime.java` and this will still
# find LevelDatTime.class in the same directory.
LEVEL_DAT_TIME_HELPER="/Users/david/scripts/shockbyte_dl/LevelDatTime.java"
LEVEL_DAT_TIME_HISTORY_FILE="/Users/david/scripts/shockbyte_dl/logs/level_dat_time_history.csv"

check_level_dat_time() {
    local level_dat="$EXTRACT_DIR/world/level.dat"
    if [ ! -f "$level_dat" ]; then
        log "WARNING: $level_dat not found, skipping Data.Time check."
        return
    fi

    local current_time
    current_time=$(java "$LEVEL_DAT_TIME_HELPER" "$level_dat" 2>>"$LOGFILE")
    case "$current_time" in
        ''|*[!0-9]*)
            log "WARNING: could not read Data.Time from $level_dat (helper said: '$current_time'). Skipping."
            return
            ;;
    esac

    log "level.dat Data.Time: ${current_time}"

    touch "$LEVEL_DAT_TIME_HISTORY_FILE"
    local last_line last_time
    last_line=$(tail -n 1 "$LEVEL_DAT_TIME_HISTORY_FILE")
    if [ -n "$last_line" ]; then
        last_time=$(echo "$last_line" | cut -d',' -f2)
        if [ -n "$last_time" ] && [ "$current_time" -le "$last_time" ]; then
            log "WARNING: Data.Time did not increase (last: ${last_time}, tonight: ${current_time})."
            mail_alert "level.dat Data.Time did not increase" "Tonight's world/level.dat Data.Time (${current_time}) is not greater than last night's recorded value (${last_time}).

Data.Time is a tick counter documented to always increase and never reset during normal play, so this is the most direct available signal of a world rollback or a fresh-world swap upstream, before Shockbyte even made its own backup.

This check ran normally and did not block the backup -- all seven copies were still made from tonight's data. Worth a look when you get a chance."
        fi
    else
        log "level.dat Data.Time check: no prior recorded value yet, nothing to compare against."
    fi

    echo "$(date '+%Y-%m-%d'),${current_time}" >> "$LEVEL_DAT_TIME_HISTORY_FILE"
}

# ---- Grandfather-Father-Son (GFS) rotation for dated backup folders ----
# Mirrors the retention policy already deployed for tarsnap
# (tarsnap-rotate.sh: KEEP_DAILY=14, KEEP_WEEKLY=8, KEEP_MONTHLY=12,
# KEEP_YEARLY=2), applied here to plain dated folders (YYYY-MM-DD-HH-MM)
# instead of tarsnap archive names -- local, iCloud, and batcave never had
# any rotation before now; they just accumulated forever.
#
# Defaults to dry-run (nothing deleted, decisions only logged) until
# ROTATE_COMMIT is set to 1 below -- same rollout caution as
# tarsnap-rotate.sh: review a dry-run pass first, then flip it on. Never
# deletes a folder whose name doesn't parse as a date; logs it as a
# warning and leaves it alone instead, as a safety net.
#
# The bucketing logic itself was tested against 1300 nights of synthetic
# daily data (plus malformed folder names) before being wired in here --
# dry-run and --commit-equivalent runs produced identical survivor sets.
# What ISN'T tested for real: this Mac's BSD `date -j -f` syntax below
# (I only have GNU date in my own sandbox to test against). Worth
# confirming once with:
#   date -j -f "%Y-%m-%d" "2026-08-03" "+%G-W%V"
# which should print something like "2026-W32". If that errors instead,
# let me know and I'll adjust the format string.
KEEP_DAILY=14
KEEP_WEEKLY=8
KEEP_MONTHLY=12
KEEP_YEARLY=2        # 0 = keep yearly folders forever
ROTATE_COMMIT=${ROTATE_COMMIT:-1}

epoch_of() { date -j -f "%Y-%m-%d" "$1" "+%s" 2>/dev/null; }
isoweek_of() { date -j -f "%Y-%m-%d" "$1" "+%G-W%V" 2>/dev/null; }
month_of() { date -j -f "%Y-%m-%d" "$1" "+%Y-%m" 2>/dev/null; }

rotate_dated_dirs() {
    local root_path="$1"
    local label="$2"

    if [ ! -d "$root_path" ]; then
        log "WARNING: rotate_dated_dirs ($label): $root_path does not exist, skipping."
        return
    fi

    # ---- TEMP diagnostic: settle-time check on the just-written RUN_TS folder ----
    # Investigating a production run (2026-08-03, launchd-triggered via
    # `launchctl kickstart`) where this function's glob found zero entries
    # immediately after the preceding mkdir+cp into this same root_path
    # succeeded with no error -- confirmed afterward, by hand, that the
    # folders genuinely existed. Leading theory: a same-run visibility lag
    # (e.g. iCloud Drive's FileProvider layer not yet reflecting a just-
    # written directory to a readdir/glob call), not a permissions issue.
    # This logs what `-d`, a null-glob-qualified glob, and `ls -la` each see
    # on this run's own RUN_TS folder specifically, retrying a few times
    # with a short sleep so the real glob below runs against fresher state.
    # If the theory's right, later attempts should start seeing it; if it's
    # never visible after these retries, that rules the theory out instead.
    # Safe to remove entirely once the cause is confirmed either way.
    local expected_dir="${root_path}/${RUN_TS}"
    local diag_attempt diag_glob diag_ls
    for diag_attempt in 0 1 2 3 4; do
        if [ -d "$expected_dir" ]; then
            log "rotate_dated_dirs ($label) diag: attempt $diag_attempt (t+${diag_attempt}x2s) -- [ -d ] sees $expected_dir"
        else
            log "rotate_dated_dirs ($label) diag: attempt $diag_attempt (t+${diag_attempt}x2s) -- [ -d ] does NOT see $expected_dir"
        fi
        diag_glob=("$root_path"/*/(N))
        log "rotate_dated_dirs ($label) diag: attempt $diag_attempt -- glob sees ${#diag_glob[@]} entrie(s): ${diag_glob[*]}"
        diag_ls=$(ls -la "$root_path" 2>&1)
        log "rotate_dated_dirs ($label) diag: attempt $diag_attempt -- ls -la $root_path:"
        log "$diag_ls"
        if [ -d "$expected_dir" ] && [ "${#diag_glob[@]}" -gt 0 ]; then
            log "rotate_dated_dirs ($label) diag: became visible on attempt $diag_attempt, proceeding to real rotation."
            break
        fi
        sleep 2
    done
    # ---- end TEMP diagnostic ----

    local today_epoch
    today_epoch=$(date "+%s")

    # name/datepart/epoch declared ONCE here, outside the loop below --
    # zsh prints "name=value" instead of silently re-declaring if `local`
    # is issued again on a variable that already has a value, which bit
    # me during testing when these lived inside the loop instead.
    local name datepart epoch
    local entries
    entries=$(
        for dir in "$root_path"/*/; do
            [ -d "$dir" ] || continue
            name=$(basename "$dir")
            case "$name" in
                [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9])
                    datepart="${name:0:10}"
                    epoch=$(epoch_of "$datepart")
                    if [ -z "$epoch" ]; then
                        log "WARNING: rotate_dated_dirs ($label): could not parse date from '$name', skipping -- never deleting unparseable names."
                        continue
                    fi
                    echo "${epoch}|${name}|${datepart}"
                    ;;
                *)
                    log "WARNING: rotate_dated_dirs ($label): '$name' doesn't match expected YYYY-MM-DD-HH-MM pattern, skipping -- never deleting unparseable names."
                    ;;
            esac
        done | sort -t'|' -k1,1n
    )

    if [ -z "$entries" ]; then
        log "rotate_dated_dirs ($label): no dated folders found under $root_path."
        return
    fi

    local weekly_boundary=$(( KEEP_DAILY + KEEP_WEEKLY * 7 ))
    local monthly_boundary=$(( weekly_boundary + KEEP_MONTHLY * 30 ))

    local seen_weeks="" seen_months="" seen_years=""
    local kept=0 deleted=0
    # Same hoist-above-the-loop rule as above.
    local e_epoch e_name e_datepart age_days decision reason wk mo yr

    while IFS='|' read -r e_epoch e_name e_datepart; do
        age_days=$(( (today_epoch - e_epoch) / 86400 ))
        decision=""; reason=""

        if [ "$age_days" -lt "$KEEP_DAILY" ]; then
            decision="KEEP"; reason="within ${KEEP_DAILY}-day daily window"
        elif [ "$age_days" -lt "$weekly_boundary" ]; then
            wk=$(isoweek_of "$e_datepart")
            case " $seen_weeks " in
                *" $wk "*) decision="DELETE"; reason="weekly bucket $wk already has a survivor" ;;
                *) decision="KEEP"; reason="earliest in weekly bucket $wk"; seen_weeks="$seen_weeks $wk" ;;
            esac
        elif [ "$age_days" -lt "$monthly_boundary" ]; then
            mo=$(month_of "$e_datepart")
            case " $seen_months " in
                *" $mo "*) decision="DELETE"; reason="monthly bucket $mo already has a survivor" ;;
                *) decision="KEEP"; reason="earliest in monthly bucket $mo"; seen_months="$seen_months $mo" ;;
            esac
        elif [ "$KEEP_YEARLY" -eq 0 ] || [ "$age_days" -lt $(( monthly_boundary + KEEP_YEARLY * 365 )) ]; then
            yr="${e_datepart:0:4}"
            case " $seen_years " in
                *" $yr "*) decision="DELETE"; reason="yearly bucket $yr already has a survivor" ;;
                *) decision="KEEP"; reason="earliest in yearly bucket $yr"; seen_years="$seen_years $yr" ;;
            esac
        else
            decision="DELETE"; reason="older than the ${KEEP_YEARLY}-year retention window"
        fi

        if [ "$decision" = "DELETE" ]; then
            deleted=$((deleted+1))
            if [ "$ROTATE_COMMIT" -eq 1 ]; then
                rm -rf "${root_path}/${e_name}"
                log "rotate_dated_dirs ($label): DELETED $e_name ($reason)"
            else
                log "rotate_dated_dirs ($label): [dry-run] would DELETE $e_name ($reason)"
            fi
        else
            kept=$((kept+1))
            log "rotate_dated_dirs ($label): keeping $e_name ($reason)"
        fi
    done < <(echo "$entries")

    log "rotate_dated_dirs ($label): kept=$kept deleted=$deleted (commit=${ROTATE_COMMIT})"
}

# ---- Lightweight restore verification ----
# The "0" in 3-2-1-1-0 -- confidence that a backup you'd actually reach for
# in an emergency still works, not just that it exists. Runs weekly (not
# nightly; this is meant to catch slow corruption, not repeat the same
# check every night) against the most recent iCloud backup: does the zip
# itself extract cleanly (unzip -t, catches truncated/corrupted archives),
# and does world/level.dat parse as valid NBT (reusing LevelDatTime.java,
# catches garbled world data that a structurally-valid zip could still
# contain). Non-fatal, alerts on failure, always cleans up its scratch
# extraction whether it passes or fails. If this "Lightweight" restore
# verification does not sit well, the "Heavyweight" alternative would be
# to spool up a Minecraft server, load the world, then monitor for errors.
RESTORE_CHECK_DAY=7   # 1=Monday ... 7=Sunday (matches `date +%u`)
RESTORE_CHECK_SCRATCH="/Users/david/scripts/shockbyte_dl/restore_check_scratch"

check_restore_integrity() {
    local today_dow
    today_dow=$(date "+%u")
    if [ "$today_dow" -ne "$RESTORE_CHECK_DAY" ]; then
        log "restore check: not scheduled today (runs on day $RESTORE_CHECK_DAY of the week), skipping."
        return
    fi

    local latest_dir latest_zip
    latest_dir=$(ls -1 "$ICLOUD_DIR" 2>/dev/null | grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}-[0-9]{2}$' | sort | tail -n 1)
    if [ -z "$latest_dir" ]; then
        log "WARNING: restore check: no dated backups found under $ICLOUD_DIR, skipping."
        return
    fi
    latest_zip="${ICLOUD_DIR}/${latest_dir}/jswv.zip"
    if [ ! -f "$latest_zip" ]; then
        log "WARNING: restore check: expected zip not found at $latest_zip, skipping."
        return
    fi

    log "restore check: testing $latest_zip..."

    rm -rf "$RESTORE_CHECK_SCRATCH"
    mkdir -p "$RESTORE_CHECK_SCRATCH"

    if ! unzip -tq "$latest_zip" > /dev/null 2>&1; then
        log "WARNING: restore check FAILED: unzip -t reported errors in $latest_zip."
        mail_alert "restore check failed: zip integrity" "unzip -t failed against $latest_zip (from $latest_dir) -- the archive itself appears corrupted. Caught by the weekly restore-verification check; nothing about tonight's run was blocked."
        rm -rf "$RESTORE_CHECK_SCRATCH"
        return
    fi

    unzip -q "$latest_zip" -d "$RESTORE_CHECK_SCRATCH"
    if [ $? -ne 0 ]; then
        log "WARNING: restore check FAILED: unzip extraction failed for $latest_zip despite passing -t."
        mail_alert "restore check failed: extraction" "unzip of $latest_zip (from $latest_dir) failed during extraction despite passing the -t integrity test. Worth a look."
        rm -rf "$RESTORE_CHECK_SCRATCH"
        return
    fi

    local level_dat_path
    level_dat_path=$(find "$RESTORE_CHECK_SCRATCH" -type f -name "level.dat" -path "*/world/level.dat" 2>/dev/null | head -n 1)
    if [ -z "$level_dat_path" ]; then
        log "WARNING: restore check FAILED: no world/level.dat found after extracting $latest_zip."
        mail_alert "restore check failed: missing level.dat" "Extracted $latest_zip (from $latest_dir) successfully, but no world/level.dat was found inside it. Worth a look."
        rm -rf "$RESTORE_CHECK_SCRATCH"
        return
    fi

    local restore_time
    restore_time=$(java "$LEVEL_DAT_TIME_HELPER" "$level_dat_path" 2>&1)
    case "$restore_time" in
        ''|*[!0-9]*)
            log "WARNING: restore check FAILED: could not parse Data.Time from $level_dat_path (helper said: '$restore_time')."
            mail_alert "restore check failed: level.dat parse error" "Extracted $latest_zip (from $latest_dir), but level.dat failed to parse as valid NBT (helper output: '$restore_time'). This suggests the world data itself may be corrupted, even though the zip extracted fine."
            rm -rf "$RESTORE_CHECK_SCRATCH"
            return
            ;;
    esac

    log "restore check PASSED: $latest_dir extracts cleanly and level.dat parses (Data.Time=${restore_time})."
    rm -rf "$RESTORE_CHECK_SCRATCH"
}

# ---- DVD burn cadence + DVD-only pruning ----
# Pruning is now DVD-only -- gmktec, iCloud, and batcave all get the
# stripped-but-unpruned world going forward. Only the DVD's ISO is built
# from a pruned copy, since the ~5.7GB -> ~800MB reduction pruning buys is
# only needed to fit a single-layer disc in the first place.
#
# DVD burns also move from nightly to weekly. DVD is the only air-gapped
# copy in the scheme right now, but nightly media-swapping was the
# highest-friction manual step in the whole pipeline for a benefit that's
# mostly about worst-case recovery point, not freshness -- going from
# "lose under a day" to "lose under a week" in the worst case seemed like
# a reasonable trade for 7x fewer swaps.
DVD_BURN_DAY=7   # 1=Monday ... 7=Sunday (matches `date +%u`); trivially adjustable
DVD_EXTRACT_DIR="${DL_PATH}/jswv_dvd"

is_dvd_burn_day() {
    [ "$(date +%u)" -eq "$DVD_BURN_DAY" ]
}

# Clones EXTRACT_DIR (stripped, unpruned) into DVD_EXTRACT_DIR and prunes
# ONLY the clone via mcaselector; the caller builds the ISO from
# DVD_EXTRACT_DIR afterward. Tries `cp -c` (APFS copy-on-write clone --
# near-instant, and near-free in disk space until mcaselector's rewrite
# diverges the pruned files from the original) first, but explicitly falls
# back to a plain recursive copy on failure rather than trusting cp's own
# fallback behavior, which I haven't been able to verify from this sandbox
# (no APFS or BSD cp available to test against).
prepare_dvd_clone() {
    rm -rf "$DVD_EXTRACT_DIR"
    log "cloning $EXTRACT_DIR to $DVD_EXTRACT_DIR for DVD-only pruning..."
    if ! cp -c -R "$EXTRACT_DIR" "$DVD_EXTRACT_DIR" 2>>"$LOGFILE"; then
        log "WARNING: cp -c (APFS clone) failed or unsupported -- falling back to a plain copy (slower, and uses real disk space up front instead of a near-free clone)."
        rm -rf "$DVD_EXTRACT_DIR"
        cp -R "$EXTRACT_DIR" "$DVD_EXTRACT_DIR"
        if [ $? -ne 0 ]; then
            log "ERROR: fallback copy to $DVD_EXTRACT_DIR also failed. Skipping DVD burn tonight."
            mail_alert "DVD clone failed" "Both cp -c and a plain fallback copy failed to create $DVD_EXTRACT_DIR from $EXTRACT_DIR. DVD burn skipped tonight; the other six copies are unaffected."
            return 1
        fi
    fi

    log "de-chunking DVD copy (overworld)..."
    java --enable-final-field-mutation=ALL-UNNAMED -jar /Users/david/mcaselector/mcaselector-2.8.jar --mode delete --world "$DVD_EXTRACT_DIR"/world --region "$DVD_EXTRACT_DIR"/world/dimensions/minecraft/overworld/region --query "InhabitedTime < 1minute"

    log "de-chunking DVD copy (the_end)..."
    java --enable-final-field-mutation=ALL-UNNAMED -jar /Users/david/mcaselector/mcaselector-2.8.jar --mode delete --world "$DVD_EXTRACT_DIR"/world --region "$DVD_EXTRACT_DIR"/world/dimensions/minecraft/the_end/region --query "InhabitedTime < 1minute"

    log "de-chunking DVD copy (the_nether)..."
    java --enable-final-field-mutation=ALL-UNNAMED -jar /Users/david/mcaselector/mcaselector-2.8.jar --mode delete --world "$DVD_EXTRACT_DIR"/world --region "$DVD_EXTRACT_DIR"/world/dimensions/minecraft/the_nether/region --query "InhabitedTime < 1minute"

    find "$DVD_EXTRACT_DIR"/ -name ".DS_Store" -delete
    return 0
}

if [ ! -f "$ZIP_FILE" ]; then
    log "ERROR: $ZIP_FILE not found. Aborting unzip step."
    exit 1
fi

mkdir -p "$EXTRACT_DIR"

# Unzip the downloaded instance.zip, and if successful, remove.
log "unzipping file..."
unzip -q "$ZIP_FILE" -d "$EXTRACT_DIR"
if [ $? -ne 0 ]; then
    log "ERROR: unzip failed."
    mail_alert "unzip failed" "unzip of $ZIP_FILE into $EXTRACT_DIR failed. Aborting before any cleanup or fan-out -- none of the seven copies ran tonight."
    exit 1
else
    rm "$ZIP_FILE"
fi

# Make sure the extract directory exists, or 'rm -rf' would be dangerous
: "${EXTRACT_DIR:?EXTRACT_DIR is not set}"

# Measure the raw, unpruned world before anything below touches it --
# mcaselector's chunk-pruning further down is *supposed* to shrink the
# world every night, so measuring after that would swamp the signal.
check_world_integrity
check_level_dat_time

# Remove unwanted files and folders from Minecraft world
log "removing unwanted files and directories..."
rm -rf "$EXTRACT_DIR"/.cache
rm -rf "$EXTRACT_DIR"/.shockbyte
rm -rf "$EXTRACT_DIR"/crash-reports
rm -rf "$EXTRACT_DIR"/debug
rm -rf "$EXTRACT_DIR"/libraries
rm -rf "$EXTRACT_DIR"/logs
rm -rf "$EXTRACT_DIR"/versions
rm -rf "$EXTRACT_DIR"/server.jar
rm -rf "$EXTRACT_DIR"/server-icon.png
rm -rf "$EXTRACT_DIR"/world/paper-world.yml
rm -rf "$EXTRACT_DIR"/world/session.lock
rm -rf "$EXTRACT_DIR"/world/DIM-1
rm -rf "$EXTRACT_DIR"/world/datapacks

# Need to find and remove ".DS_Store" files here. No longer needs to run
# after mcaselector specifically -- pruning happens on a separate DVD-only
# clone now (see prepare_dvd_clone below), not on EXTRACT_DIR.
find "$EXTRACT_DIR"/ -name ".DS_Store" -delete

# Re-zip the cleaned, UNPRUNED folder -- this is what iCloud and batcave
# get now. gmktec's rsync below reads $EXTRACT_DIR directly, same unpruned
# data. Only the DVD branch further down prunes, via its own clone.
log "rezipping cleaned folder into new jswv.zip file..."
pushd "$DL_PATH"
zip -qr jswv ./jswv
if [ $? -ne 0 ]; then
    log "ERROR: zip of $EXTRACT_DIR into $NEW_ZIP failed."
    mail_alert "zip failed" "Re-zipping $EXTRACT_DIR into $NEW_ZIP failed. The iCloud and batcave copies (which cp $NEW_ZIP) will be missing or stale tonight. rsync to gmktec/seneca and the DVD burn are unaffected since they read from $EXTRACT_DIR (or a clone of it), not the zip."
fi
popd

# .DS_Store creeps back in almost immediately even after the pass above --
# macOS regenerates it fast enough that a second pass right before rsync
# catches the ones that snuck back in during the zip step.
find "$EXTRACT_DIR"/ -name ".DS_Store" -delete

log "checking rsync dry-run delete count against gmktec..."
check_rsync_delete_count "$GMKTEC_HOST" "$GMKTEC_PATH" "$RSYNC_DELETE_HISTORY_FILE" "gmktec"

# Rsync to gmktec. --delete keeps gmktec an exact mirror of tonight's
# EXTRACT_DIR -- tarsnap-rotate.sh (which reads from here) still keeps its
# own full history independently via tarsnap's own archive retention, so
# this doesn't cost any tarsnap history.
log "rsyncing to gmktec..."
rsync -a --delete -e "ssh -i $SSH_KEY" "$EXTRACT_DIR"/ "${GMKTEC_HOST}:${GMKTEC_PATH}/"
if [ $? -ne 0 ]; then
    log "ERROR: rsync to gmktec failed."
    mail_alert "rsync to gmktec failed" "rsync of $EXTRACT_DIR to gmktec failed. The other backup destinations are unaffected -- this only costs tonight's gmktec/tarsnap copy. Since gmktec's own copy is now stale/incomplete, batcave's forward step was skipped too rather than risk pushing inconsistent data."
else
    # gmktec's own copy is only trustworthy to forward on if the rsync
    # above actually succeeded -- gated here rather than running
    # unconditionally, so a bad gmktec rsync can't also poison batcave.
    log "triggering forward-to-batcave on gmktec..."
    if ! ssh -i "$SSH_KEY" "${GMKTEC_HOST}" "${GMKTEC_FORWARD_SCRIPT}"; then
        log "ERROR: triggering forward-to-batcave.sh on gmktec failed."
        mail_alert "batcave forward failed" "SSHing into gmktec to run ${GMKTEC_FORWARD_SCRIPT} (which forwards tonight's world data to batcave and triggers its ZFS snapshot) failed. The other backup destinations are unaffected -- this only costs tonight's batcave copy. Check gmktec's own forward-to-batcave.log as well as this run's log."
    fi
fi

log "checking rsync dry-run delete count against seneca..."
check_rsync_delete_count "$SENECA_HOST" "$SENECA_PATH" "$SENECA_DELETE_HISTORY_FILE" "seneca"

# Rsync to seneca (tape host). Same --delete/mirror approach as gmktec --
# the separate tape-write script on seneca reads whatever's currently
# here, same division of labor as tarsnap-rotate.sh does on gmktec.
log "rsyncing to seneca..."
rsync -a --delete -e "ssh -i $SSH_KEY" "$EXTRACT_DIR"/ "${SENECA_HOST}:${SENECA_PATH}/"
if [ $? -ne 0 ]; then
    log "ERROR: rsync to seneca failed."
    mail_alert "rsync to seneca failed" "rsync of $EXTRACT_DIR to seneca (tape host) failed. The other backup destinations are unaffected -- this only costs tonight's tape-write source copy on seneca."
fi

if is_dvd_burn_day; then
    log "today is a DVD burn day (day $DVD_BURN_DAY of the week) -- preparing pruned clone for the ISO..."
    if prepare_dvd_clone; then
        log "making iso file for burning..."
        hdiutil makehybrid -udf -o "$ISO_FILE" "$DVD_EXTRACT_DIR"
    fi
    rm -rf "$DVD_EXTRACT_DIR"

    # DVD burning pre-checks -- only meaningful on a burn day, so these (and
    # the actual burn attempt below) live entirely inside this branch now
    # instead of running, and logging, every night regardless.
    # Change MAX_SIZE if other media is used, such as a DVD-DL, or Blu-Ray, or M-Disc

    # Check 1: does the ISO exist and is it a reasonable size? (won't exist
    # if prepare_dvd_clone or hdiutil makehybrid failed above)
    if [ ! -f "$ISO_FILE" ]; then
        log "ISO file not found at $ISO_FILE. Skipping burn."
        OK_TO_BURN=0
    else
        SIZE=$(stat -f %z "$ISO_FILE")
        if [ "$SIZE" -ge "$MAX_SIZE" ]; then
            log "ISO file is too large to burn ($SIZE bytes)."
            OK_TO_BURN=0
        fi
    fi

    # Check 2: is a burner attached?
    if [ "$OK_TO_BURN" -eq 1 ]; then
        BURNER_INFO=$(hdiutil burn -list 2>&1)
        if echo "$BURNER_INFO" | grep -q "no CD/DVD drives present"; then
            log "No optical burner detected. Skipping burn."
            OK_TO_BURN=0
        fi
    fi

    # Check 3: is media loaded?
    if [ "$OK_TO_BURN" -eq 1 ]; then
        DRIVE_STATUS=$(drutil status 2>&1)
        if echo "$DRIVE_STATUS" | grep -q "No Media Inserted"; then
            log "Drive present, but no media inserted. Skipping burn."
            OK_TO_BURN=0
        fi
    fi

    # Final decision
    if [ "$OK_TO_BURN" -eq 1 ]; then
        log "All checks passed. Proceeding with burn..."
        hdiutil burn "$ISO_FILE"
        if [ $? -eq 0 ]; then
            log "Burn completed successfully."
            log "Removing ISO file..."
            rm "$ISO_FILE"
            if [ $? -ne 0 ]; then
                log "WARNING: Failed to remove ISO file at $ISO_FILE."
            fi
        else
            log "ERROR: hdiutil burn failed. Keeping ISO file for retry."
            mail_alert "DVD burn failed" "hdiutil burn failed for $ISO_FILE. The ISO was kept at that path so the burn can be retried by hand."
        fi
    else
        log "Burn step skipped due to failed pre-check(s)."
    fi
else
    log "not a DVD burn day (runs on day $DVD_BURN_DAY of the week) -- skipping ISO build and burn."
fi

# Sometimes this fails with an error that the directory is not empty, so try a few times.
log "removing unzipped instance.zip..."
for attempt in 1 2 3; do
  rm -rf "$EXTRACT_DIR"
  if [ ! -d "$EXTRACT_DIR" ]; then
    break
  fi
  log "rm -rf instance didn't fully clear it (attempt $attempt) -- retrying..."
  sleep 2
done
if [ -d "$EXTRACT_DIR" ]; then
  log "WARNING: instance/ still exists after 3 cleanup attempts. Leaving it"
  log "in place rather than looping forever -- check it by hand."
fi

# No local (ROOT_PATH) copy anymore -- it was never a real backup leg, just
# a same-machine, same-disk convenience for glancing at what's going on,
# which meant it never protected against anything the processing machine's
# own disk failure wouldn't already destroy. ROOT_PATH is still used below
# for the DVD's staging ISO_FILE, just not for a retained dated archive.
log "making iCloud directory for current date..."
mkdir -p "$ICLOUD_DIR"/"$RUN_TS"

log "copying jswv.zip to iCloud..."
cp "$NEW_ZIP" "$ICLOUD_DIR"/"$RUN_TS"
if [ $? -ne 0 ]; then
    log "ERROR: copy of $NEW_ZIP to $ICLOUD_DIR failed."
    mail_alert "iCloud copy failed" "cp of $NEW_ZIP to $ICLOUD_DIR failed."
fi

log "rotating old iCloud backups (GFS: ${KEEP_DAILY}/${KEEP_WEEKLY}/${KEEP_MONTHLY}/${KEEP_YEARLY})..."
rotate_dated_dirs "$ICLOUD_DIR" "iCloud"

check_restore_integrity

# batcave no longer gets touched from here at all -- see the
# GMKTEC_FORWARD_SCRIPT trigger right after the gmktec rsync above.
# batcave's own retention (ZFS snapshot + GFS prune) also now happens
# entirely on batcave/gmktec's side, not via rotate_dated_dirs here.

log "removing jswv.zip..."
rm "$NEW_ZIP"

log "all finished!"