RunBSD

← Minecraft Backups

Daily Pipeline-Health Digest

8 August 2026

What this does

Closes the last open item from the Minecraft Backups page's "Needed improvements" list: confirming a clean night used to mean manually checking logs across four separate machines (Mac mini, gmktec, seneca, and batcave). daily-digest.sh runs once a night, pulls today's relevant log lines from all four, and emails one combined summary via Mail.app/AppleScript — the same alerting mechanism backup.sh already uses.

Scheduled via a LaunchAgent (com.david.daily-digest.plist) at 5:00 AM — late enough that Shockbyte's download (2AM), backup.sh (immediately after), seneca's tape backup (3AM), and gmktec's tarsnap job have all had time to finish for the night.

What it pulls, and from where

Each remote log is checked for existence before being grepped (ssh host "[ -f /path/to/log ]", with output silenced via >/dev/null 2>&1 so a routine "file doesn't exist yet" check doesn't print SSH connection noise) — a missing log just gets logged as "... log is missing" and the script moves on to the next one, instead of dying on the first host that's ever offline or hasn't produced a log file yet.

Authentication

Every SSH call uses the same dedicated, passphrase-less key backup.sh uses for gmktec and seneca as of 8 August 2026 (~/.ssh/backup_automation) — not David's general-purpose personal key, and no ssh-agent/Keychain involved at all. See the backup.sh & DVD Archiving page's "SSH authentication" section for the full reasoning (narrower blast radius on this particular Mac, and no reboot-loses-the-loaded-key failure mode to self-heal around, since there's no agent in the loop to lose anything). The nested hop to batcave reuses gmktec's own separate batcave_key rather than adding a second key to this script.

Alerting

mail_digest() mirrors backup.sh's mail_alert() pattern — AppleScript driving the already-signed-in Mail.app account, since this Mac's LaunchAgent runs inside the GUI session and there's no MTA installed here. A few bugs were caught and fixed while building this:

The digest log (pipeline-digest.log) is only deleted after a successful send — a failed send leaves it in place so a night's content isn't lost.

LaunchAgent

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.david.daily-digest</string>

    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/caffeinate</string>
        <string>-i</string>
        <string>/Users/david/scripts/shockbyte_dl/daily-digest.sh</string>
    </array>

    <key>WorkingDirectory</key>
    <string>/Users/david/scripts/shockbyte_dl</string>

    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>5</integer>
        <key>Minute</key>
        <integer>0</integer>
    </dict>

    <key>RunAtLoad</key>
    <false/>

    <key>StandardOutPath</key>
    <string>/Users/david/scripts/shockbyte_dl/logs/stdout.log</string>

    <key>StandardErrorPath</key>
    <string>/Users/david/scripts/shockbyte_dl/logs/stderr.log</string>

    <key>ProcessType</key>
    <string>Interactive</string>
</dict>
</plist>

caffeinate -i keeps the Mac from idle-sleeping mid-run, same reasoning as backup.sh's own LaunchAgent — this script needs network reachability to four hosts. ProcessType: Interactive matches the GUI-session requirement for the Mail.app-driven alerting. The script needs its executable bit set (chmod +x daily-digest.sh), since caffeinate execs it directly rather than through an interpreter.

daily-digest.sh

#!/bin/zsh

# Daily digest notification email script. This script attempts to
# aggregate various log files from the different machines involved
# in the separate branches of the backup scheme, then email them
# as a daily digest of the Minecraft backup process.

LOGFILE="/Users/david/scripts/shockbyte_dl/logs/pipeline-digest.log"
log() { echo "$1" >> "$LOGFILE"; }
DIGEST_EMAIL="me@example.com"

# Same dedicated, passphrase-less key backup.sh uses for gmktec/seneca as
# of 2026-08-08 -- not David's general-purpose personal key, and no
# ssh-agent/Keychain involved at all. See backup.sh's own comment above
# its SSH_KEY for the full reasoning (narrower blast radius on this Mac
# specifically, and sidesteps the reboot-loses-the-loaded-key failure
# mode entirely rather than self-healing around it).
SSH_KEY="/Users/david/.ssh/backup_automation"

# This Mac mini's LaunchAgent runs in the GUI session (per the
# SSH_AUTH_SOCK note in the README), so we can drive the already-signed-in
# Mail.app account via AppleScript.
mail_digest() {
    local subject
    subject="Daily Backup Digest"
    local full_body
    full_body=$(cat "$LOGFILE")

    local as_subject="${subject//\\/\\\\}"
    as_subject="${as_subject//\"/\\\"}"
    local as_body="${full_body//\\/\\\\}"
    as_body="${as_body//\"/\\\"}"

    local osascript_output
    osascript_output=$(osascript <<APPLESCRIPT 2>&1
tell application "Mail"
    set newMessage to make new outgoing message with properties {subject:"backup-digest: ${as_subject}", content:"${as_body}", visible:false}
    tell newMessage
        make new to recipient at end of to recipients with properties {address:"${DIGEST_EMAIL}"}
        send
    end tell
end tell
APPLESCRIPT
)
    if [ $? -ne 0 ]; then
        log "WARNING: mail_digest failed to send via Mail.app (subject: $subject) -- osascript output: $osascript_output"
        return 1
    fi
}

# Get seneca's tape backup log
log "seneca's tape backup log:"
if ssh -i "$SSH_KEY" dhw@seneca "[ -f /var/log/zfs-tape-backup.log ]" >/dev/null 2>&1; then
    log "$(ssh -i "$SSH_KEY" dhw@seneca grep "$(date -I)" /var/log/zfs-tape-backup.log)"
else
    log "seneca log is missing (/var/log/zfs-tape-backup.log not found on seneca)"
fi
log ""

# Get gmktec's tarsnap logs
log "gmktec's tarsnap cron log:"
if ssh -i "$SSH_KEY" dhw@gmktec "[ -f /var/log/tarsnap-cron.log ]" >/dev/null 2>&1; then
    log "$(ssh -i "$SSH_KEY" dhw@gmktec tail /var/log/tarsnap-cron.log)"
else
    log "gmktec's tarsnap cron log is missing (/var/log/tarsnap-cron.log not found on gmktec)"
fi
log ""
log "gmktec's tarsnap rotate log:"
if ssh -i "$SSH_KEY" dhw@gmktec "[ -f /var/log/tarsnap-rotate.log ]" >/dev/null 2>&1; then
    log "$(ssh -i "$SSH_KEY" dhw@gmktec grep "$(date -I)" /var/log/tarsnap-rotate.log)"
else
    log "gmktec's tarsnap rotate log is missing (/var/log/tarsnap-rotate.log not found on gmktec)"
fi
log ""

# Get gmktec's forward-to-batcave log -- added 2026-08-08, same day the
# SMB-mount-to-batcave leg was replaced with gmktec forwarding its own
# rsync copy on to batcave over SSH + ZFS. Runs as unprivileged dhw (not
# root cron), so its log lives under dhw's own home dir, not /var/log --
# see forward-to-batcave.sh's own STATE_DIR comment for why.
log "gmktec's forward-to-batcave log:"
if ssh -i "$SSH_KEY" dhw@gmktec "[ -f /home/dhw/forward-to-batcave/forward-to-batcave.log ]" >/dev/null 2>&1; then
    log "$(ssh -i "$SSH_KEY" dhw@gmktec grep "$(date -I)" /home/dhw/forward-to-batcave/forward-to-batcave.log)"
else
    log "gmktec's forward-to-batcave log is missing (/home/dhw/forward-to-batcave/forward-to-batcave.log not found on gmktec)"
fi
log ""

# Get batcave's ZFS snapshot+prune log -- same 2026-08-08 addition. The
# Mac has no direct SSH access to batcave at all in this architecture
# (deliberately -- the whole point of the gmktec-forwards-to-batcave
# design was to keep the Mac off that link entirely). So this hops
# through gmktec instead, reusing gmktec's own existing batcave key
# rather than requiring a second key to be authorized on batcave.
log "batcave's zfs-snapshot-prune log:"
if ssh -i "$SSH_KEY" dhw@gmktec "ssh -i /home/dhw/.ssh/batcave_key dhw@batcave [ -f /home/dhw/zfs-snapshot-prune/zfs-snapshot-prune.log ]" >/dev/null 2>&1; then
    log "$(ssh -i "$SSH_KEY" dhw@gmktec "ssh -i /home/dhw/.ssh/batcave_key dhw@batcave grep $(date -I) /home/dhw/zfs-snapshot-prune/zfs-snapshot-prune.log")"
else
    log "batcave's zfs-snapshot-prune log is missing or unreachable via gmktec (/home/dhw/zfs-snapshot-prune/zfs-snapshot-prune.log)"
fi
log ""

# Get Mac's backup log.
if [ -f "/Users/david/scripts/shockbyte_dl/logs/backup_script.log" ]; then
    log "$(grep "$(date -I)" /Users/david/scripts/shockbyte_dl/logs/backup_script.log)"
else
    log "Mac's backup_script log is missing (/Users/david/scripts/shockbyte_dl/logs/backup_script.log not found on David's MacBook Air)"
fi

# Mail the daily-digest
mail_digest
mail_digest_exit=$?

# Only remove the digest log if it actually sent -- keep it around on
# failure so tonight's content isn't lost.
if [ "$mail_digest_exit" -eq 0 ]; then
    rm "$LOGFILE"
fi

Status

Added and debugged 8 August 2026; not yet run across multiple nights in production. Once it's proven stable, this closes item 4 ("One daily pipeline-health digest") on the Minecraft Backups page's "Needed improvements" list.

Open items / possible follow-ups