# Mack Hankins — Full Content

## GitLab as a Business Asset: Inventory, Valuation, and Disaster Recovery

URL: https://mackhankins.com/blog/gitlab-as-a-business-asset-inventory-valuation-and-disaster-recovery
Published: 2026-05-07
Reading time: 7 min

When people talk about disaster recovery for development systems, they usually focus on infrastructure: servers, storage, backups, and restore procedures. Those things matter, but they are only part of the picture. A self-hosted GitLab instance also contains years of source code, issue history, merge request discussions, deployment pipelines, snippets, and contributor knowledge. In other words, it is not just a server workload. It is a business asset.

That distinction matters because disaster recovery is not only about whether a server can be restored. It is also about whether the organization can recover the work, decisions, and operational knowledge embedded in the platform. If a GitLab instance disappeared tomorrow, the replacement problem would not be limited to reinstalling software and restoring storage. It would include recreating repositories, rebuilding deployment logic, recovering issue history, and replacing the institutional context captured over years of development.

## Why GitLab should be treated as an asset

It is easy to think of GitLab as a place where code lives. In practice, it often holds far more than source files.

A mature instance may include:

- application source code
- issue and ticket history
- merge requests and code review discussions
- CI/CD pipeline definitions
- snippets and one-off automation
- contributor activity and authorship history
- language and repository statistics
- evidence of long-term system evolution

Taken together, those records represent labor, decisions, and operational knowledge. They also represent replacement cost. Even if an organization has backups, it is still useful to understand the value of what is being protected.

That value matters for several reasons:

- disaster recovery planning
- insurance and risk discussions
- prioritizing backup and restore testing
- identifying critical projects
- justifying investment in platform maintenance
- documenting the scale of institutional knowledge tied to the system

## The point is not the script

The script itself is only a means to an end. The more important idea is that APIs make software assets measurable.

A self-hosted GitLab instance exposes enough metadata through the GitLab API to build a useful inventory of what exists in the platform. That inventory can then support a broader conversation about replacement cost and disaster recovery.

Instead of asking only, “Do we have backups?” an organization can ask better questions:

- How many projects are we protecting?
- How much development history is represented there?
- How many issues and merge requests capture operational decisions?
- How much deployment knowledge exists in CI/CD configurations?
- How many contributors have shaped the platform over time?
- How much of our institutional memory is embedded in GitLab?

Those are the kinds of questions that move disaster recovery from infrastructure thinking to business continuity thinking.

## What the API can tell you

GitLab’s API makes it possible to gather a practical inventory of a self-hosted instance. Depending on permissions and configuration, you can retrieve information such as:

- project counts
- repository and storage size
- commit history
- issue counts
- merge request counts
- contributor lists
- language breakdowns
- snippet counts
- CI/CD configuration presence
- oldest commit dates and project age

That information is useful on its own. Even without assigning a dollar value, it helps answer a basic governance question: what exactly lives in this platform?

A small example looks like this:

```python
projects = api_get("/projects", {"statistics": "true"})
issue_count = api_get_count(f"/projects/{project_id}/issues")
contributors = api_get(f"/projects/{project_id}/repository/contributors")
```

The point of calls like these is not technical novelty. The point is that they let you turn a development platform into something you can inventory, summarize, and explain.

## From inventory to valuation

Once you have an inventory, the next step is estimation.

No automated model can perfectly capture the value of a long-lived software environment. But a rough estimate is still better than treating the instance as if it were only a virtual machine with a disk attached.

One practical approach is to estimate replacement cost using several categories of measurable work:

- commits as a rough proxy for implementation effort
- issues as a proxy for captured requirements and problem-solving
- merge requests as a proxy for review and coordination effort
- snippets as a proxy for small but useful automation artifacts
- CI/CD configurations as a proxy for deployment and operational engineering work

For example, a script can apply assumptions such as hourly labor cost and estimated effort per commit, issue, merge request, or pipeline configuration. That produces a conservative estimate based only on what GitLab can directly measure.

A broader model can go further by considering:

- total years of development represented by the platform
- average staffing over that history
- loaded salary or labor cost
- the percentage of work actually tracked in version control
- domain expertise that would be expensive to replace

This is especially relevant in specialized environments where the platform reflects compliance knowledge, legacy integrations, or workflows that are not easy for a new team to absorb quickly.

## Why this matters for disaster recovery

The disaster recovery value of this approach is straightforward.

If you can measure what is in GitLab, you can make better decisions about how aggressively to protect it.

That helps organizations:

- prioritize backup strategies
- justify offsite replication or secondary infrastructure
- decide which projects require the fastest recovery
- understand the difference between restoring data and restoring capability
- explain risk in terms leadership can understand

A server can be rebuilt. A repository can be restored. But the real challenge is recovering the accumulated work and context that make those systems useful.

That is why valuation matters. It gives decision-makers a way to understand that the platform is not just hardware, storage, or a software license. It is a container for years of labor and institutional memory.

## A note on assumptions and limits

Any valuation model based on API data has limits.

Commit counts are imperfect proxies. Not every meaningful task produces a commit. Some work happens outside GitLab. Some repositories are more valuable than others regardless of size or activity. And any labor-rate assumption will vary by organization.

That is fine, as long as the estimate is presented honestly.

The goal is not to produce a perfect accounting number. The goal is to create a defensible estimate that supports planning, prioritization, and risk discussions.

It is also worth noting that any published examples should redact sensitive implementation details. Internal hostnames, tokens, private URLs, and environment-specific identifiers should be replaced with generic placeholders in code samples and screenshots.

For example:

```python
GITLAB_URL = os.getenv("GITLAB_URL", "https://gitlab.example.com")
GITLAB_TOKEN = os.getenv("GITLAB_TOKEN", "YOUR_TOKEN")
```

That keeps the focus where it belongs: on the method and the purpose, not on internal details.

## Conclusion

A self-hosted GitLab instance should be treated as more than infrastructure. It is a software asset with measurable replacement cost, operational significance, and disaster recovery implications.

Using the GitLab API to inventory projects, history, contributors, and pipeline configuration is a practical way to make that asset visible. Once visible, it becomes easier to discuss value, justify protection, and plan recovery around what actually matters.

Backups are essential. But understanding what you are backing up is just as important.

---

## ZFS Snapshot Replication Between TrueNAS Systems

URL: https://mackhankins.com/blog/zfs-snapshot-replication-between-truenas-systems
Published: 2026-04-30
Reading time: 7 min

If you manage any data you care about, a single copy on a single system isn't enough. ZFS makes it remarkably easy to replicate datasets between systems using `zfs send` and `zfs receive` over SSH — and TrueNAS wraps this in a decent GUI.

This post walks through setting up nightly incremental replication from one TrueNAS box to another, with **asymmetric retention**: the source keeps snapshots for just 3 days (saving disk space), while the destination holds them for 120 days (giving you months of recovery points).

After the initial full transfer, each nightly run only ships the blocks that changed since the last snapshot. For write-once workloads like media archives or backups, daily incrementals are tiny.

---

## Prerequisites

- **Source**: TrueNAS CORE 12.x+ (the system pushing snapshots)
- **Destination**: TrueNAS or FreeNAS 11.x+ (the system receiving snapshots)
- SSH enabled on the destination
- Network connectivity between the two boxes
- A destination dataset with `readonly=on`

---

## Step 1: Prepare the Destination

### Enable SSH

In the destination's web GUI, go to **Services** and enable **SSH**. Verify from a shell:

```bash
sockstat -l4 | grep :22
```

You should see `sshd` listening on port 22.

### Create the Receiving Dataset

```bash
zfs create tank/replication
zfs set readonly=on tank/replication
```

The `readonly=on` property prevents accidental writes to your replica. ZFS replication bypasses this flag internally, so snapshots will still be received without issue.

### Verify Pool Properties

```bash
zfs get compression,atime,readonly tank tank/replication
```

I'd recommend `compression=lz4` and `atime=off` on the parent pool if you haven't already.

---

## Step 2: Create an SSH Connection on the Source

### Generate an SSH Keypair

On the source, go to **System > SSH Connections > ADD**.

| Setting | Value |
|---------|-------|
| Name | Something recognizable (e.g., the destination hostname) |
| Setup Method | Manual |
| Host | Destination IP address |
| Port | 22 |
| Username | root |
| Private Key | Generate New |
| Cipher | Standard |

Save the connection.

### Install the Public Key on the Destination

Extract the public key from the source's keychain and install it on the destination. From the source's shell:

```bash
# Find the keypair ID
midclt call keychaincredential.query | python3 -c " # [tl! focus:start]
import sys, json
for c in json.load(sys.stdin):
    print(c['name'], 'ID:', c['id'], 'Type:', c['type'])
" # [tl! focus:end]

# Extract public key and push it to the destination
midclt call keychaincredential.query '[[\"id\",\"=\",KEY_ID]]' \
  | python3 -c "
import sys, json
print(json.load(sys.stdin)[0]['attributes']['public_key'])
" | ssh root@DEST_IP \
    'mkdir -p /root/.ssh && cat >> /root/.ssh/authorized_keys && chmod 600 /root/.ssh/authorized_keys'
```

Replace `KEY_ID` with the ID from the first command, and `DEST_IP` with your destination's IP. You'll be prompted for the root password once — after that, key-based auth takes over.

### Discover the Remote Host Key

Back in the source's web GUI, edit the SSH connection you just created and click **Discover Remote Host Key**. This populates the host key in the correct format.

> **Heads up**: The host key must be stored *without* the hostname prefix. The `Discover Remote Host Key` button handles this automatically. If you're setting the key via the API, store only the key type and data (e.g., `ecdsa-sha2-nistp256 AAAA...`), not `192.168.1.x ecdsa-sha2-nistp256 AAAA...`. The middleware prepends the hostname on its own.

### Verify Connectivity

```bash
ssh -v root@DEST_IP "zfs list"
```

This should connect without a password prompt and list the destination's datasets.

---

## Step 3: Create the Replication Task

On the source, go to **Tasks > Replication Tasks > ADD**.

### What and Where

| Setting | Value |
|---------|-------|
| Source Location | On this System |
| Source Dataset | Your dataset (e.g., `tank/data`) |
| Recursive | Checked |
| Destination Location | On a Different System |
| SSH Connection | The connection from Step 2 |
| Destination Dataset | `tank/replication` on the destination |
| SSH Transfer Security | Encryption (or No Encryption for LAN) |

### When

| Setting | Value |
|---------|-------|
| Replication Schedule | Run On a Schedule |
| Schedule | Daily at 00:00 |
| Destination Snapshot Lifetime | Same as Source (we'll change this next) |

Click **START REPLICATION**. This creates both the replication task and an associated periodic snapshot task on the source.

---

## Step 4: Adjust Retention Policies

The wizard defaults to 2-week retention everywhere. For our setup, we want minimal footprint on the source and long retention on the destination.

### Source: 3-Day Retention

```bash
# List snapshot tasks
midclt call pool.snapshottask.query | python3 -c "
import sys, json
for t in json.load(sys.stdin):
    print(f'ID:{t[\"id\"]} dataset:{t[\"dataset\"]} lifetime:{t[\"lifetime_value\"]} {t[\"lifetime_unit\"]}')
"

# Update to 3 days
midclt call pool.snapshottask.update TASK_ID \  # [tl! highlight]
  '{"lifetime_value": 3, "lifetime_unit": "DAY"}'  # [tl! highlight]
```

**Why 3 days?** Incremental replication needs at least one common snapshot between source and destination. Three days gives you a buffer — if replication fails one night, you still have two more days before the chain breaks. Bump this higher if your connectivity is unreliable.

### Destination: 120-Day Retention

```bash
# List replication tasks
midclt call replication.query | python3 -c "
import sys, json
for t in json.load(sys.stdin):
    print(f'ID:{t[\"id\"]} name:{t[\"name\"]} retention:{t[\"retention_policy\"]}')
"

# Update to 120-day custom retention
midclt call replication.update TASK_ID \  # [tl! highlight]
  '{"retention_policy": "CUSTOM", "lifetime_value": 120, "lifetime_unit": "DAY"}'  # [tl! highlight]
```

### Retention Summary

| System | Retention | Why |
|--------|-----------|-----|
| Source | 3 days | Minimize disk usage on the working system |
| Destination | 120 days | ~4 months of daily recovery points |

---

## Step 5: Run the Initial Sync

From **Tasks > Replication Tasks**, expand the task and click **RUN NOW**.

The first run is a full `zfs send` of the entire dataset. Depending on size and network speed, this could take a while. Monitor the State column for progress.

After the initial sync, every subsequent nightly run is incremental and only transfers changed blocks.

---

## Recovering Files from Snapshots

### Single File Recovery

ZFS snapshots are accessible as read-only directories. On the destination, browse into:

```
/mnt/tank/replication/.zfs/snapshot/
```

Each snapshot is a timestamped folder. Find the date before the unwanted change and copy the file out:

```bash
# List available snapshots
ls /mnt/tank/replication/.zfs/snapshot/

# Grab a file from a specific snapshot
cp /mnt/tank/replication/.zfs/snapshot/auto-2026-03-15_00-00/path/to/file.tif \
   /tmp/recovered_file.tif
```

### Full Disaster Recovery

The **RESTORE** button on the Replication Tasks page reverses the direction, pushing everything from the destination back to the source. Only use this if the source pool is completely lost and needs to be rebuilt.

---

## Troubleshooting

### "Invalid SSH host key" Error

The stored host key format is wrong. Edit the SSH connection and click **Discover Remote Host Key** again. Remember: the key must not include the hostname prefix — the middleware adds it automatically.

### Replication Stuck or Failed

Check the middleware log on the source:

```bash
grep -i "replication\|zettarepl" /var/log/middlewared.log | tail -30
```

### Broken Incremental Chain

If the source's snapshots were pruned before replication ran (e.g., the destination was offline longer than the retention period), incremental replication fails because there's no common snapshot. Your options:

1. Delete all snapshots on the destination dataset and re-run for a fresh full sync
2. Manually create a common base snapshot

### Testing SSH with the Keychain Key

```bash
# Extract private key to a temp file
midclt call keychaincredential.query '[[\"id\",\"=\",KEY_ID]]' \
  | python3 -c "
import sys, json
print(json.load(sys.stdin)[0]['attributes']['private_key'])
" > /tmp/test_key
chmod 600 /tmp/test_key

# Test the connection
ssh -i /tmp/test_key root@DEST_IP "zfs list -r tank"

# Clean up
rm /tmp/test_key
```

---

## Things Worth Knowing

- **`readonly=on` doesn't block replication** — ZFS handles this internally. Your destination stays protected from accidental writes while still receiving snapshots.
- **Semi-auto SSH setup won't work across versions** — If the destination runs FreeNAS 11.x and the source is TrueNAS CORE 12+, the API versions differ. Use the manual method above.
- **"No Encryption" is fast but unencrypted** — Skipping SSH encryption on the transfer can be significantly faster on a LAN. Only do this on a trusted network.
- **Snapshot overhead is proportional to change rate** — For write-once data, 120 days of daily snapshots adds minimal overhead beyond the base dataset size. For heavy-write workloads, monitor your snapshot space usage.


---

## Cleaning Up Old Kernels on Ubuntu

URL: https://mackhankins.com/blog/cleaning-up-old-kernels-on-ubuntu
Published: 2026-04-23
Reading time: 4 min

If you've been running an Ubuntu server for a while, you've probably noticed `/boot` filling up with old kernel images. Ubuntu installs new kernels with each update but doesn't always clean up the old ones.

## Why Not Just Use `apt autoremove`?

The official answer is `sudo apt autoremove --purge`. In practice, it often leaves old kernels behind. This happens because:

- **Manually installed kernels** — if a kernel was installed via `apt install linux-image-*`, apt considers it explicitly requested and won't auto-remove it.
- **Stale autoremoval config** — Ubuntu maintains a protection list in `/etc/apt/apt.conf.d/01autoremove-kernels` that can get out of sync after partial upgrades or version pinning.
- **Held or pinned packages** — anything marked `hold` or pinned in apt preferences is excluded from autoremoval, even if it's no longer needed.

When `autoremove` isn't cutting it, a targeted script can take care of the rest.

## The Script

```bash
#!/bin/bash
# Remove old kernels on Ubuntu/Debian systems
# Run without arguments for a dry run
# Run with "exec" argument to actually remove old kernels
# Example: sudo ./remove-old-kernels.sh exec

set -euo pipefail

# Ensure we're running on a Debian-based system
if ! command -v dpkg &>/dev/null; then
    echo "Error: This script requires dpkg (Debian/Ubuntu systems only)."
    exit 1
fi

CURRENT_KERNEL=$(uname -r)
CURRENT_VERSION=$(echo "$CURRENT_KERNEL" | sed 's/-[a-z].*$//')
echo "Currently running kernel: $CURRENT_KERNEL"

# Get only installed (ii) kernel-related packages, excluding the running kernel
# and meta-packages (linux-image-generic, linux-headers-generic, etc.)
OLD_KERNELS=$(
    dpkg --list |
        awk '/^ii\s+(linux-image|linux-headers|linux-modules)/ { print $2 }' |
        grep -v "$CURRENT_VERSION" |
        grep -E '[0-9]+\.[0-9]+\.[0-9]+' || true
)

if [ -z "$OLD_KERNELS" ]; then
    echo "No old kernels found to remove."
    exit 0
fi

echo ""
echo "Old kernels to be removed:"
echo "$OLD_KERNELS"
echo ""

if [ "${1:-}" == "exec" ]; then
    if [ "$EUID" -ne 0 ]; then
        echo "Error: Must run as root. Use: sudo $0 exec"
        exit 1
    fi
    apt purge -y $OLD_KERNELS
    echo ""
    echo "Done. Running 'update-grub' to refresh boot menu..."
    update-grub
    echo "Old kernels removed successfully."
else
    echo "Dry run complete. To remove these kernels, run:"
    echo "  sudo $0 exec"
fi
```

## How It Works

The script has two modes: a **dry run** (default) that shows what would be removed, and an **exec** mode that actually purges the packages.

**Finding old kernels:**

```bash
dpkg --list |
    awk '/^ii\s+(linux-image|linux-headers|linux-modules)/ { print $2 }' |
    grep -v "$CURRENT_VERSION" |
    grep -E '[0-9]+\.[0-9]+\.[0-9]+'
```

This pipeline does a few important things:

1. `awk '/^ii/'` — only matches **installed** packages. `dpkg --list` also shows removed and half-configured packages; we don't want those.
2. `grep -v "$CURRENT_VERSION"` — excludes anything matching the running kernel's base version (e.g., `6.8.0-106`). This protects both the flavor-specific packages like `linux-image-6.8.0-106-generic` and the common packages like `linux-headers-6.8.0-106` that the running kernel depends on.
3. `grep -E '[0-9]+\.[0-9]+\.[0-9]+'` — only matches versioned packages. This is the key safety measure: it excludes meta-packages like `linux-image-generic` and `linux-headers-generic`. Removing those would break future kernel updates via `apt upgrade`.

After purging, the script runs `update-grub` to refresh the boot menu so removed kernels no longer appear at startup.

## Usage

First, do a dry run to review what will be removed:

```bash
sudo bash remove-old-kernels.sh
```

Example output:

```
Currently running kernel: 6.8.0-106-generic

Old kernels to be removed:
linux-headers-6.8.0-101
linux-headers-6.8.0-101-generic
linux-image-6.8.0-101-generic
linux-modules-6.8.0-101-generic
linux-modules-extra-6.8.0-101-generic
```

If the list looks right, run it for real:

```bash
sudo bash remove-old-kernels.sh exec
```

## Things to Keep in Mind

- **Always do the dry run first.** Review the package list before removing anything.
- **This is for Ubuntu/Debian only.** Fedora, Arch, and others handle kernel packaging differently.
- **Try `apt autoremove --purge` first.** Use this script for what autoremove misses.
- Tested on **Ubuntu 24.04 LTS** with kernel `6.8.0-106-generic`.

---

## Cheap Cold Storage Backups with AWS Glacier Deep Archive and TrueNAS

URL: https://mackhankins.com/blog/cheap-cold-storage-backups-with-aws-glacier-deep-archive-and-truenas
Published: 2026-04-16
Reading time: 5 min

If you need an off-site backup for a large dataset but don't want to pay traditional cloud storage prices, Glacier Deep Archive is hard to beat at **~$1/TB/month**. The trade-off is retrieval time (12-48 hours), but for a true last-resort backup, that's fine.

TrueNAS has built-in Cloud Sync Tasks that use rclone under the hood, making it straightforward to push a dataset to S3 with the Glacier Deep Archive storage class — no lifecycle rules needed.

---

## AWS Setup

### Create the Bucket and IAM User

1. Create an S3 bucket in your preferred region
2. Create a dedicated IAM user for the backup (e.g., `truenas-backup`)
3. Attach the **AmazonS3FullAccess** managed policy (or scope it down to your specific bucket if you prefer least-privilege)
4. Generate an access key pair and store the secret key somewhere safe — you can't retrieve it after creation

### How the Storage Class Works

Objects are uploaded directly to Glacier Deep Archive using rclone's `--s3-storage-class DEEP_ARCHIVE` flag. TrueNAS handles this automatically when you select the storage tier in the Cloud Sync Task UI. No S3 lifecycle rules are required.

This is actually a nice side benefit — if your AWS account has restrictions on lifecycle rule creation (e.g., Organization-level Service Control Policies), the direct storage class approach bypasses that entirely.

---

## TrueNAS Configuration

### Step 1: Add Cloud Credentials

1. Navigate to **Credentials > Backup Credentials > Cloud Credentials**
2. Click **Add**
3. Set **Provider** to Amazon S3
4. Enter a name (e.g., `AWS Glacier Backup`)
5. Enter your Access Key ID and Secret Access Key
6. Click **Verify Credential** to confirm connectivity
7. Save

### Step 2: Create the Cloud Sync Task

Navigate to **Data Protection > Cloud Sync Tasks** and click **Add**.

| Setting | Value |
|---------|-------|
| Description | Monthly Off-site Backup |
| Direction | PUSH |
| Transfer Mode | SYNC |
| Credential | Your AWS credential from Step 1 |
| Bucket | Your S3 bucket name |
| Source Path | `/mnt/your-pool` |
| Storage Class | Glacier Deep Archive |
| Schedule | Monthly (1st of month, off-hours) |

### SYNC vs COPY: Which Transfer Mode?

**SYNC** mirrors the exact state of your pool each month — it adds new files, updates changed files, and removes files from the remote that no longer exist locally.

**COPY** only adds new files and never deletes from the remote. Use this if you want the remote to accumulate everything ever added to the pool, even files you later removed locally.

For most backup scenarios, SYNC is what you want. If you're paranoid about accidental local deletions propagating to your backup, use COPY — but know that your bucket will grow indefinitely.

---

## Excluding Directories

If you need to exclude certain directories from the backup, add exclusion patterns in the Cloud Sync Task's filter/exclude field:

```text
large-temp-files/**
scratch/**
```

Adjust paths based on your dataset structure.

---

## Performance Expectations

Based on real-world testing with a 1GB upload:

- **Upload speed**: ~97 Mbps (unthrottled, will vary by ISP and region)
- **Test duration**: ~1 minute 24 seconds for 1GB

### Estimated Times for a 25TB Dataset

| Throttle Setting | Estimated Duration |
|------------------|-------------------|
| 10 Mbps | ~5 days 12 hours |
| 25 Mbps | ~2 days 5 hours |
| 50 Mbps | ~1 day 3 hours |
| 75 Mbps | ~18 hours |
| Unthrottled (~97 Mbps) | ~9 hours 15 minutes |

TrueNAS supports bandwidth throttling in the Cloud Sync Task settings. **50 Mbps** is a reasonable middle ground — completes within a weekend while leaving bandwidth available for other operations.

The first sync transfers everything. Subsequent monthly syncs only transfer new or changed files, making them significantly faster.

---

## Things to Know About Glacier Deep Archive

### Retrieval Takes Time

This isn't a quick-access backup. Retrieving data takes **12-48 hours** depending on the retrieval tier (Standard or Bulk). Plan accordingly for disaster recovery — this is your cold copy, not your first line of defense.

### 180-Day Minimum Storage Charge

Objects in Glacier Deep Archive have a 180-day minimum storage charge. Delete something before 180 days and you're billed for the remaining time. At ~$1/TB/month this is negligible for small deletions, but worth knowing if you're doing large-scale removals.

### Strategies to Avoid Early Deletion Fees

For write-once datasets, this is a non-issue. If deletions are a concern:

- **Use COPY mode** instead of SYNC — never deletes from remote, so no deletion fees. Bucket grows over time.
- **Enable S3 versioning** — deletes add a marker rather than removing the object. Old versions age past 180 days naturally. (Requires lifecycle rules to clean up old versions eventually.)
- **Just be aware** — if you delete files locally and run SYNC, those objects get removed from the bucket on the next run, potentially triggering fees.

---

## Troubleshooting

### Viewing Task Logs

In the TrueNAS UI, go to **Data Protection > Cloud Sync Tasks** and look for a task report dropdown after a run completes. It shows start time, finish time, and success/failure.

```bash
# Or check the middleware log directly
grep -i "cloud_sync" /var/log/middlewared.log | tail -50
```

### Verifying Uploads

Log into the AWS Console, navigate to your S3 bucket, and confirm objects are listed with the **Glacier Deep Archive** storage class.

### Permission Errors

If you see errors related to lifecycle rules or other S3 operations, it's likely an Organization-level Service Control Policy. These can't be modified from a member account — talk to whoever manages your AWS Organization. The direct storage class approach in this guide avoids most of these issues.

---

## Cost Summary

For a 25TB dataset:

| Item | Monthly Cost |
|------|-------------|
| Glacier Deep Archive Storage | ~$25 |
| PUT requests (first sync, ~25M objects) | ~$1.25 |
| Subsequent syncs (incremental) | Negligible |
| **Total ongoing** | **~$25/month** |

That's a full off-site backup of 25TB for the price of a decent lunch. Hard to argue with that.

---

## A Simple Bash Script for Multi-Server MySQL Backups

URL: https://mackhankins.com/blog/a-simple-bash-script-for-multi-server-mysql-backups
Published: 2026-04-14
Reading time: 6 min

# MySQL Backup Script

A Bash script that automates MySQL database backups across multiple servers using `mysqldump`. It reads server credentials from a JSON config file, backs up all non-system databases, organizes backups into daily and monthly directories, and enforces a configurable retention policy.

## Requirements

- `jq` - JSON processor
- `mysql-client` - MySQL client tools (`mysql` and `mysqldump`)

## Setup

### 1. Configure the Script

Copy the example config and fill in your server details:

```bash
cp config.example.json config.json
```

Edit `config.json` with your servers:

```json
{
    "backup_dir": "/backups/mysql",
    "retention_days": 7,
    "servers": [
        {
            "host": "db1.example.internal",
            "login_path": "db1-backup"
        }
    ]
}
```

> Credentials are not stored in `config.json`. They live in an encrypted `~/.mylogin.cnf` file managed by `mysql_config_editor` (see step 2).

> `login_path` is the name of the credential entry in `.mylogin.cnf`. If omitted, the script falls back to using `host` as the login-path name.

### 2. The script
create `mysql_backup.sh`
```bash
#!/bin/bash

set -e

# Check if jq is installed
if ! command -v jq &> /dev/null; then
    echo "jq is required but not installed. Please install jq to use this script."
    exit 1
fi

# Check if mysqldump is installed
if ! command -v mysqldump &> /dev/null; then
    echo "mysqldump is required but not installed. Please install mysql-client to use this script."
    exit 1
fi

# Get the directory of the script
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# Set the config file path relative to the script directory
CONFIG_FILE="${SCRIPT_DIR}/config.json"

# Function to read configuration and check backup directory
read_config_and_check_dir() {
    if [ ! -f "$CONFIG_FILE" ]; then
        echo "Configuration file not found: $CONFIG_FILE"
        exit 1
    fi

    BACKUP_DIR=$(jq -r '.backup_dir' "$CONFIG_FILE")
    RETENTION_DAYS=$(jq -r '.retention_days' "$CONFIG_FILE")

    # Check if backup directory exists, create if it doesn't
    if [ ! -d "$BACKUP_DIR" ]; then
        echo "Backup directory does not exist. Creating: $BACKUP_DIR"
        mkdir -p "$BACKUP_DIR"
        if [ $? -ne 0 ]; then
            echo "Failed to create backup directory. Please check permissions and path."
            exit 1
        fi
    fi

    echo "Using backup directory: $BACKUP_DIR"
}

# Function to create backup
create_backup() {
    local HOST=$1
    local LOGIN_PATH=$2
    local DB=$3
    local BACKUP_PATH="$BACKUP_DIR/$HOST/$DB"

    mkdir -p "$BACKUP_PATH/daily"
    mkdir -p "$BACKUP_PATH/monthly"

    echo "Backing up database: $DB"
    mysqldump --login-path="$LOGIN_PATH" --single-transaction --routines --triggers "$DB" > "$BACKUP_PATH/daily/$DB-$DATE.sql"

    # Create monthly backup on the 1st of each month
    if [ "$(date +%d)" = "01" ]; then
        cp "$BACKUP_PATH/daily/$DB-$DATE.sql" "$BACKUP_PATH/monthly/$DB-$MONTH.sql"
    fi
}

# Function to remove old backups
cleanup_old_backups() {
    local HOST=$1
    local DB=$2
    local BACKUP_PATH="$BACKUP_DIR/$HOST/$DB/daily"

    find "$BACKUP_PATH" -name "*.sql" -type f -mtime +$RETENTION_DAYS -delete
}

# Function to list databases
list_databases() {
    local LOGIN_PATH=$1

    echo "Attempting to list databases using login-path $LOGIN_PATH" >&2
    mysql --login-path="$LOGIN_PATH" -N -e "SHOW DATABASES;" 2>/dev/null | grep -Ev "^(information_schema|performance_schema|mysql|sys)$"
}

# Main backup process
main() {
    read_config_and_check_dir

    DATE=$(date +"%Y-%m-%d")
    MONTH=$(date +"%Y-%m")

    echo "Starting backup process..."

    jq -c '.servers[]' "$CONFIG_FILE" | while read -r server; do
        HOST=$(echo $server | jq -r '.host')
        LOGIN_PATH=$(echo $server | jq -r '.login_path // .host')

        echo "Backing up server: $HOST (login-path: $LOGIN_PATH)"

        # Get list of databases
        echo "Listing databases..."
        DATABASES=$(list_databases "$LOGIN_PATH")

        if [ -z "$DATABASES" ]; then
            echo "No databases found or unable to list databases on $HOST. Check your login-path and permissions."
            continue
        fi

        echo "Databases found:"
        echo "$DATABASES"

        echo "$DATABASES" | while read -r DB; do
            create_backup "$HOST" "$LOGIN_PATH" "$DB"
            cleanup_old_backups "$HOST" "$DB"
        done
    done

    echo "Backup process completed."
}

main
```

### 3. Store Credentials with `mysql_config_editor`

Since the cron job runs as root, credentials must live in root's home (`/root/.mylogin.cnf`). For each server in `config.json`, create a matching login-path:

```bash
sudo mysql_config_editor set --login-path=db1-backup --host=db1.example.internal --user=backups --password
```

You'll be prompted for the password, which is then stored in `/root/.mylogin.cnf` (obfuscated, mode `0600`).

Verify entries:

```bash
sudo mysql_config_editor print --all
```

Remove an entry if needed:

```bash
sudo mysql_config_editor remove --login-path=db1-backup
```

Test a connection:

```bash
sudo mysql --login-path=db1-backup -e "SHOW DATABASES;"
```

### 4. Create a MySQL Backup User on the Remote Server

Connect to MySQL on the database server:

```bash
mysql -u root -p
```

Create the backup user. Replace `198.51.100.25` with the IP of the server running this script, or use `%` to allow connections from any host:

```sql
CREATE USER 'backups'@'198.51.100.25' IDENTIFIED BY 'strong_password_here';

GRANT SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER, PROCESS ON *.* TO 'backups'@'198.51.100.25';

FLUSH PRIVILEGES;
```

> `PROCESS` must be granted globally (`*.*`) and is required to avoid tablespace errors during `mysqldump`.

Verify the user was created:

```sql
SELECT user, host FROM mysql.user WHERE user = 'backups';
```

**If you need to fix the host IP after creation:**

```sql
RENAME USER 'backups'@'wrong_host' TO 'backups'@'198.51.100.25';
FLUSH PRIVILEGES;
```

### 5. Ensure MySQL Accepts Remote Connections

On the database server, check the bind address:

```bash
sudo grep -E "bind.address|bind-address" /etc/mysql/mysql.conf.d/mysqld.cnf
```

If it shows `127.0.0.1`, change it to listen on the network:

```ini
bind-address = 0.0.0.0
```

Restart MySQL:

```bash
sudo systemctl restart mysql
```

### 6. Open the Firewall

Allow the backup server to reach MySQL:

```bash
sudo ufw allow from 198.51.100.25 to any port 3306
```

> Avoid opening port 3306 to the world. Lock it to specific source IPs.

### 7. Make the Script Executable and Run

```bash
chmod +x mysql_backup.sh
sudo ./mysql_backup.sh
```

> Must run as root — `/root/.mylogin.cnf` is only readable by root, and cron runs backups as root too.

## Cron Job

Run backups daily at 2 AM:

```bash
sudo crontab -e
```

```text
0 2 * * * /path/to/mysqlbackup/mysql_backup.sh
```

## Backup Structure

```text
backup_dir/
└── hostname/
    └── database_name/
        ├── daily/
        │   └── database_name-YYYY-MM-DD.sql
        └── monthly/
            └── database_name-YYYY-MM.sql
```

- **Daily backups** are created on every run.
- **Monthly backups** are created on the 1st of each month.
- Daily backups older than the configured `retention_days` are automatically deleted.

## Excluded Databases

The following system databases are automatically excluded:

- `information_schema`
- `performance_schema`
- `mysql`
- `sys`

## Required MySQL Privileges

| Privilege | Scope | Purpose |
|-----------|-------|---------|
| `SELECT` | `*.*` | Read table data |
| `LOCK TABLES` | `*.*` | Consistent backups |
| `SHOW VIEW` | `*.*` | Back up views |
| `EVENT` | `*.*` | Back up events |
| `TRIGGER` | `*.*` | Back up triggers |
| `PROCESS` | `*.*` | Dump tablespace info |

## Restoring a Backup

To restore a dump using a configured login-path:

```bash
mysql --login-path=db1-backup < /path/to/backup.sql
```

Because the dump is created with `--databases`, it includes the `CREATE DATABASE` and `USE` statements needed to recreate and select the database during restore.

To restore with a different host or user:

```bash
mysql -h db1.example.internal -u your-user -p < /path/to/backup.sql
```

## Notes

A few practical details worth calling out:

- `--single-transaction` works best for InnoDB tables and avoids locking during most backups.
- This script writes plain `.sql` dumps, which keeps restores simple.
- If storage becomes a concern, piping the output through `gzip` is an easy next step.

It’s not fancy, but it’s easy to understand, easy to schedule, and easy to extend.

---

## Automated Multi-Server PostgreSQL Backups with a Simple Bash Script

URL: https://mackhankins.com/blog/automated-multi-server-postgresql-backups-with-a-simple-bash-script
Published: 2026-04-09
Reading time: 5 min

I needed a way to back up PostgreSQL databases across multiple servers without installing anything heavy. The requirements were simple:

- Back up every non-system database on each server
- Organize backups by server and database name
- Keep daily backups with configurable retention
- Create monthly snapshots on the 1st
- Drive the whole thing from a single JSON config file

Here's what I ended up with.

---

## Requirements

- `bash`
- `jq` — for parsing the JSON configuration
- `pg_dump` / `psql` — PostgreSQL client tools

---

## The Config File

Create a `config.json` in the same directory as the script. It defines where to store backups, how long to keep them, and which servers to hit:

```json
{
    "backup_dir": "/backups/postgres",
    "retention_days": 7,
    "servers": [
        {
            "host": "db-primary.local",
            "user": "backup_user"
        },
        {
            "host": "db-replica.local",
            "user": "backup_user"
        }
    ]
}
```

Add as many servers as you need. The script iterates through all of them. Passwords are not stored here — they're managed via `.pgpass` (more on that below).

---

## The Script

Create `postgres_backup.sh`:

```bash
#!/bin/bash

set -e

# Check for jq dependency
if ! command -v jq &> /dev/null; then
    echo "jq is required but not installed."
    exit 1
fi

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CONFIG_FILE="${SCRIPT_DIR}/config.json"

read_config_and_check_dir() {
    if [ ! -f "$CONFIG_FILE" ]; then
        echo "Configuration file not found: $CONFIG_FILE"
        exit 1
    fi

    BACKUP_DIR=$(jq -r '.backup_dir' "$CONFIG_FILE")
    RETENTION_DAYS=$(jq -r '.retention_days' "$CONFIG_FILE")

    if [ ! -d "$BACKUP_DIR" ]; then
        echo "Creating backup directory: $BACKUP_DIR"
        mkdir -p "$BACKUP_DIR"
        if [ $? -ne 0 ]; then
            echo "Failed to create backup directory."
            exit 1
        fi
    fi

    echo "Using backup directory: $BACKUP_DIR"
}

create_backup() {
    local HOST=$1
    local USER=$2
    local DB=$3
    local BACKUP_PATH="$BACKUP_DIR/$HOST/$DB"

    mkdir -p "$BACKUP_PATH/daily"
    mkdir -p "$BACKUP_PATH/monthly"

    echo "Backing up database: $DB"
    pg_dump -h $HOST -U $USER \
        -d "$DB" -F c -f "$BACKUP_PATH/daily/$DB-$DATE.dump"

    # Monthly snapshot on the 1st
    if [ "$(date +%d)" = "01" ]; then
        cp "$BACKUP_PATH/daily/$DB-$DATE.dump" \
           "$BACKUP_PATH/monthly/$DB-$MONTH.dump"
    fi
}

cleanup_old_backups() {
    local HOST=$1
    local DB=$2
    local BACKUP_PATH="$BACKUP_DIR/$HOST/$DB/daily"

    find "$BACKUP_PATH" -name "*.dump" -type f \
        -mtime +$RETENTION_DAYS -delete
}

list_databases() {
    local HOST=$1
    local USER=$2

    psql -h "$HOST" -U "$USER" -d "postgres" \
        -t -c "SELECT datname FROM pg_database
                WHERE datistemplate = false
                AND datname != 'postgres';" \
        | grep -v '^$' | sed 's/^ *//g'
}

main() {
    read_config_and_check_dir

    DATE=$(date +"%Y-%m-%d")
    MONTH=$(date +"%Y-%m")

    echo "Starting backup process..."

    jq -c '.servers[]' "$CONFIG_FILE" | while read -r server; do
        HOST=$(echo $server | jq -r '.host')
        USER=$(echo $server | jq -r '.user')

        echo "Backing up server: $HOST"

        DATABASES=$(list_databases "$HOST" "$USER")

        if [ -z "$DATABASES" ]; then
            echo "No databases found on $HOST. Skipping."
            continue
        fi

        echo "Databases found:"
        echo "$DATABASES"

        echo "$DATABASES" | while read -r DB; do
            create_backup "$HOST" "$USER" "$DB"
            cleanup_old_backups "$HOST" "$DB"
        done
    done

    echo "Backup process completed."
}

main
```

---

## Setting It Up

**1. Set up `.pgpass` for password-free authentication**

Since the cron job runs as root, the file needs to live at `/root/.pgpass`:

```bash
sudo touch /root/.pgpass
sudo chmod 600 /root/.pgpass
sudo nano /root/.pgpass
```

Add one line per server using the format `hostname:port:database:username:password`. Use `*` for the database field to match all databases on that server:

```
localhost:5432:*:local_pg_user:your_password
db-replica.local:5432:*:backup_user:your_password
```

**2. Make the script executable and do a test run**

```bash
chmod +x postgres_backup.sh
sudo ./postgres_backup.sh
```

---

## Directory Structure

After a few days of running, your backup directory looks like this:

```
/backups/postgres/
├── db-primary.local/
│   ├── myapp/
│   │   ├── daily/
│   │   │   ├── myapp-2026-03-28.dump
│   │   │   ├── myapp-2026-03-29.dump
│   │   │   └── myapp-2026-03-30.dump
│   │   └── monthly/
│   │       └── myapp-2026-03.dump
│   └── analytics/
│       ├── daily/
│       └── monthly/
└── db-replica.local/
    └── ...
```

Each server gets its own directory, each database gets `daily/` and `monthly/` subdirectories, and retention cleanup only touches `daily/`.

---

## Setting Up the Cron Job

Add it to root's crontab to run daily at 2 AM:

```bash
sudo crontab -e
```

```
0 2 * * * /path/to/postgres_backup.sh
```

---

## How It Works

1. **Reads** `config.json` for backup directory, retention policy, and server list
2. **Connects** to each server and queries `pg_database` for all non-system databases
3. **Dumps** each database using `pg_dump` in custom format (`-F c`) — the most flexible format for restores
4. **Copies** the daily dump to the monthly directory on the 1st of each month
5. **Cleans up** daily dumps older than the configured retention period

### Why Custom Format?

The `-F c` flag produces a custom-format archive that supports:

- Selective restore (individual tables, schemas)
- Parallel restore with `pg_restore -j`
- Compression built-in (no need to pipe through gzip)

### Restoring a Backup

```bash
# Full database restore
pg_restore -h localhost -U postgres -d myapp \
    /backups/postgres/db-primary.local/myapp/daily/myapp-2026-03-30.dump

# Restore a single table
pg_restore -h localhost -U postgres -d myapp \
    -t users \
    /backups/postgres/db-primary.local/myapp/daily/myapp-2026-03-30.dump
```

---

## Limitations and Improvements

This is intentionally simple. A few things to consider:

- **Credentials via `.pgpass`** — Much better than passwords in a config file, but make sure the file is `chmod 600` and owned by the user running the script.
- **No encryption at rest** — The dumps sit unencrypted on disk. Pipe through `gpg` or store them on an encrypted volume if needed.
- **No alerting** — If the cron job fails silently, you won't know. Pipe output to a log file and monitor it, or wrap the script in something that sends notifications on failure.
- **Sequential execution** — Servers are backed up one at a time. For a large fleet, you could parallelize with `xargs` or GNU `parallel`.

For my use case — a handful of servers with modest databases — this has been running reliably for over a year without any changes.

---

## Dynamic OG Images in Laravel with Intervention Image

URL: https://mackhankins.com/blog/dynamic-og-images-in-laravel-with-intervention-image
Published: 2026-04-05
Reading time: 4 min

When you share a blog post on Twitter or Discord, the link preview is often the first impression. A plain URL with no image? Easy to scroll past. A branded card with the post title and your site's design language? That stops the thumb.

Most solutions involve external services or complex headless browser setups. I wanted something simpler: a single Laravel controller that generates social card PNGs on the fly, using my site's own fonts and color palette, cached for a week.

## The Setup

One package, no external services:

```bash
composer require intervention/image
```

Intervention Image v4 ships with both GD and Imagick drivers. I went with Imagick for better text rendering, but GD works fine too.

I also downloaded my site's display fonts (Syne and Newsreader) as TTF files into `resources/fonts/` so the generated images match the site's typography exactly.

## The Route

A single route appended to the blog post URL pattern:

```php
Route::get('/blog/{post:slug}/og-image.png', OgImageController::class)
    ->name('blog.og-image');
```

The `.png` extension is intentional. Social platform crawlers expect an image URL, and this makes it unambiguous.

## The Controller

The entire generator is a single invokable controller. The `__invoke` method handles caching, and a private `generate` method does the actual image composition:

```php
class OgImageController extends Controller
{
    private const WIDTH = 1200;
    private const HEIGHT = 630;

    public function __invoke(Post $post): Response
    {
        $cacheKey = "og-image-{$post->id}-{$post->updated_at->timestamp}";

        $imageData = Cache::remember($cacheKey, now()->addDays(7), function () use ($post) {
            return $this->generate($post);
        });

        return response($imageData, 200, [
            'Content-Type' => 'image/png',
            'Cache-Control' => 'public, max-age=604800',
        ]);
    }
}
```

The cache key includes the post's `updated_at` timestamp, so editing a post automatically busts the cache. No manual invalidation needed.

## Building the Canvas

The `generate` method composes the image layer by layer:

```php
private function generate(Post $post): string
{
    $manager = new ImageManager(Driver::class);
    $canvas = $manager->createImage(self::WIDTH, self::HEIGHT);

    $canvas->fill('#0a0c12');

    $this->drawGradientBackground($canvas);

    // Decorative amber accent line
    $canvas->drawRectangle(function ($draw) {
        $draw->size(60, 4);
        $draw->at(80, 80);
        $draw->background('#c9a872');
    });

    $titleY = $this->drawTitle($canvas, $post->title, self::WIDTH - 80);

    if ($post->excerpt && $titleY < 380) {
        $titleY = $this->drawExcerpt(
            $canvas, $post->excerpt, $titleY + 24, self::WIDTH - 80
        );
    }

    // Tags, branding...
    $this->drawBranding($canvas);

    return $canvas->encode(new PngEncoder)->toString();
}
```

Each `draw` method positions elements using absolute coordinates. The layout adapts based on title length — shorter titles get a larger font size, longer titles scale down and wrap.

## Word Wrapping

Intervention Image doesn't have built-in word wrap for text, so I wrote a simple helper using PHP's `imagettfbbox` to measure text width and break lines:

```php
private function wordWrap(
    string $text, string $fontPath, int $fontSize, int $maxWidth
): array {
    $words = explode(' ', $text);
    $lines = [];
    $currentLine = '';

    foreach ($words as $word) {
        $testLine = $currentLine === '' ? $word : $currentLine . ' ' . $word;
        $box = imagettfbbox($fontSize, 0, $fontPath, $testLine);

        if ($box && ($box[2] - $box[0]) > $maxWidth && $currentLine !== '') {
            $lines[] = $currentLine;
            $currentLine = $word;
        } else {
            $currentLine = $testLine;
        }
    }

    if ($currentLine !== '') {
        $lines[] = $currentLine;
    }

    return $lines;
}
```

This measures each word as it's added to the current line. When the line would exceed `$maxWidth`, it wraps to the next line. Simple, but it handles every title I've thrown at it.

## The Background Texture

A flat dark rectangle looks lifeless. I added two subtle touches:

```php
// Subtle teal glow in bottom-right
$canvas->drawEllipse(function ($draw) {
    $draw->size(600, 400);
    $draw->at(self::WIDTH - 100, self::HEIGHT + 50);
    $draw->background('rgba(91, 168, 168, 0.05)');
});

// Horizontal scan lines for grain texture
for ($i = 0; $i < self::HEIGHT; $i += 4) {
    $alpha = ($i % 8 === 0) ? 0.03 : 0.015;
    $canvas->drawLine(function ($draw) use ($i, $alpha) {
        $draw->from(0, $i);
        $draw->to(self::WIDTH, $i);
        $draw->color("rgba(255, 255, 255, {$alpha})");
        $draw->width(1);
    });
}
```

Nearly invisible glows and scan lines. You don't consciously notice them, but they give the card depth that a flat background doesn't have.

## Wiring Up the Meta Tags

The blog post view passes the OG image route to the layout:

```blade

```

And the layout renders the standard Open Graph and Twitter Card tags:

```html





```

The `og:image:width` and `og:image:height` tags tell platforms the exact dimensions upfront, so they can render the card correctly without downloading the image first.

## The Result

Every blog post now gets a branded social card that matches my site's design. No external services, no headless browsers, no build step. Just a Laravel controller, three font files, and about 200 lines of PHP.

The cards are generated on first request and cached for a week. Editing a post busts the cache automatically. Total generation time is under 200ms with Imagick.

Sometimes the simplest solution is the right one.

---

## Replacing Two Legacy Systems with a Single Python Service

URL: https://mackhankins.com/blog/replacing-two-legacy-systems-with-a-single-python-service
Published: 2026-04-02
Reading time: 5 min

## The Problem

The preservation array at MDAH holds irreplaceable digitized historical documents, photographs, government records, and audiovisual materials mounted over NFS. We need to know exactly what changed, when, and maintain a durable audit trail. The old setup technically did this, but:

- The bash script just appended to a log file with no structured data
- The snapshot pipeline required coordinating three servers and two queues
- When the Redis queue backed up or the processor crashed, events were silently lost
- Nobody wanted to touch any of it

## The Architecture

Preservation Monitor is a single systemd-managed Python service that does everything the two old systems did. The core pipeline looks like this:

```python
# The event pipeline: inotifywait -> correlator -> audit log + database
def run(self):
    """Main event loop using select() for non-blocking reads."""
    self.watcher.start()
    self.last_flush = time.monotonic()

    while self.running:
        if not self.watcher.is_alive():
            self._flush_all()
            time.sleep(restart_delay)
            self.watcher.start()
            continue

        ready, _, _ = select.select([self.watcher.fileno()], [], [], 1.0)

        if ready:
            line = self.watcher.readline()
            raw_event = self.watcher.parse_line(line)

            # Write to audit log immediately - never lose an event
            self.audit_log.write_raw(raw_event)

            # Correlate MOVED_FROM/MOVED_TO pairs into single renames
            processed = self.correlator.process(raw_event)
            if processed:
                self.buffer.append(processed)

        if self._should_flush():
            self._flush_to_db()
```

The service spawns `inotifywait` as a subprocess watching the entire preservation tree recursively. Events stream in through stdout, get parsed, and flow through the pipeline. Every raw event is written to a daily audit log immediately, so even if the database is completely down, we never lose a record.

## Move Correlation

One of the trickier parts was handling file renames. `inotifywait` doesn't emit a single "renamed" event. Instead, it fires `MOVED_FROM` followed by `MOVED_TO` as two separate lines. If you just record them individually, your audit trail shows a confusing delete-then-create instead of a rename.

The `MoveCorrelator` solves this by holding a pending `MOVED_FROM` event for a short window. If a `MOVED_TO` follows, they get combined into a single `MOVE` event with both the old and new paths:

```python
class MoveCorrelator:
    """Correlate MOVED_FROM/MOVED_TO pairs into single MOVE events."""

    def __init__(self, timeout: float = 2.0):
        self.timeout = timeout
        self._pending: Optional[FileEvent] = None

    def process(self, event: FileEvent) -> Optional[FileEvent]:
        if event.event_type == "MOVED_FROM":
            flushed = self._flush_pending()
            self._pending = event
            self._pending_time = time.monotonic()
            return flushed

        if event.event_type == "MOVED_TO":
            if self._pending:
                # Correlate: combine into a MOVE event
                move_event = FileEvent(
                    event_type="MOVE",
                    file_path=event.file_path,
                    timestamp=event.timestamp,
                    old_path=self._pending.file_path,
                )
                move_event.stat_file()
                self._pending = None
                return move_event
            else:
                # MOVED_TO without MOVED_FROM - treat as CREATE
                event.event_type = "CREATE"
                event.stat_file()
                return event
```

If the `MOVED_TO` never comes (file moved outside the watched tree), the timeout expires and the orphaned `MOVED_FROM` becomes a `DELETE`. Clean and deterministic.

## Database Resilience

The service writes to a PostgreSQL database, but the NFS mount and the database server are on different machines. Network hiccups happen. The original snapshot pipeline would just lose events during outages. Preservation Monitor handles this differently:

- Events are always written to the daily audit log first, regardless of database state
- Database writes are buffered and batched (up to 100 events or every 5 seconds)
- If the database is unreachable, the buffer grows in memory and the service retries every 30 seconds
- A replay mode can re-process audit log files to backfill the database after extended outages

The audit log files are the source of truth. The database is a queryable index over them.

## The Originals Table

Beyond just tracking changes, Preservation Monitor maintains a live inventory of every file on the array in an `originals` table. Creates trigger upserts, deletes remove rows, and renames update paths in place. A separate reconcile mode can do a full filesystem walk to rebuild this table from scratch if needed, which is invaluable for disaster recovery planning.

## What We Replaced

| Before | After |
|--------|-------|
| Bash script + cron on the NAS | Single Python service on one server |
| ZFS snapshot pipeline across 3 servers | Direct inotifywait event stream |
| Redis queue for event transport | In-memory buffer with audit log backup |
| No structured data, just log files | PostgreSQL with queryable change history |
| Silent failures when Redis backed up | Automatic retry with audit log replay |

## Lessons Learned

**Separate your transactions carefully.** The first version of this service ran `record_changes()` and `sync_originals()` as if they were a single operation. When `sync_originals` failed (a NOT NULL constraint on a file that had disappeared before we could stat it over NFS), the exception prevented the buffer from clearing, but the `record_changes` INSERT had already committed. Every 30-second retry re-inserted the entire buffer. The `file_changes` table grew to 27GB in a matter of days before we caught it. The fix was straightforward: treat the audit insert and the inventory sync as independent operations. If the inventory update fails, log it and move on. The audit data is already safely committed.

**NFS and stat() don't always agree.** When a file is created and quickly deleted (or is a transient temp file), `stat_file()` can fail by the time we get to it. The code now gracefully handles NULL size and inode values rather than letting them cascade into a database constraint violation.

**Simpler is more reliable.** Three servers, two queues, and a cron job turned out to be harder to maintain than a single Python process reading lines from stdout. The new system has fewer moving parts, fewer failure modes, and when something does go wrong, there's one log file on one machine to check.

---

*Preservation Monitor is running in production at the Mississippi Department of Archives and History, quietly watching over Mississippi's digitized historical record.*


---

## Process Files 4.0 — Batch File Processing for Digital Archives

URL: https://mackhankins.com/blog/process-files-40-batch-file-processing-for-digital-archives
Published: 2026-03-26
Reading time: 5 min

Every digital archive eventually hits the same problem: you have a directory full of original files — TIFFs from scanners, Word documents, spreadsheets, audio recordings — and you need web-friendly versions of all of them. Thumbnails, preview images, zoomable tilesets, PDFs, compressed audio. Doing this by hand doesn't scale.

Process Files is the tool I built to handle this. Point it at a source directory, tell it which formats to process, and it handles the rest.

## The Architecture

The core idea is simple: a main orchestrator script (`start.sh`) that discovers and runs format-specific modules. Each module is a self-contained Bash script responsible for one file type.

```bash
# Run all modules against a source directory
./start.sh /path/to/collection

# Or pick specific formats with watermarking enabled
./start.sh -m tif,pdf -w /path/to/collection
```

The orchestrator parses CLI flags, sets up working and output directories, then loops through either all modules or just the ones you requested:

```bash
# If no modules specified, run them all
if [ -z $MODULES ]; then
    for module in $MODULES_DIR*.sh; do
        bash $module $SOURCE $WORKING_DIR $PROCESSED_DIR $NOTILES $WATERMARK
    done
fi

# Otherwise, run only the requested modules
for i in $(echo $MODULES | sed "s/,/ /g"); do
    if [ -f $MODULES_DIR$i".sh" ]; then
        bash $MODULES_DIR$i".sh" $SOURCE $WORKING_DIR $PROCESSED_DIR $NOTILES $WATERMARK
    fi
done
```

## Directory Structure Preservation

One of the trickier parts was preserving nested folder structures from the source. Archives aren't flat — they have subdirectories for series, boxes, folders. The output needs to mirror that hierarchy.

Each module walks the source directory, strips the base path to extract relative paths, and tracks unique directories and files in text manifests:

```bash
for file in $(find $SOURCE -type f -iname "$FILE_PATTERN" \
    ! -path "$PROCESSED_DIR*" ! -path "$WORKING_DIR*"); do

    raw=${file#"$SOURCE"}
    filename=${raw#"/"}
    directory=${raw%/*}
    directory=${directory#"/"}

    if [ ! -z "$directory" ]; then
        if ! grep -Fxq $directory $MODULE_WORKING_DIRECTORY/directories.txt; then
            echo "$directory" >> $MODULE_WORKING_DIRECTORY/directories.txt
        fi
    fi

    if ! grep -Fxq $filename $MODULE_WORKING_DIRECTORY/files.txt; then
        echo $filename >> $MODULE_WORKING_DIRECTORY/files.txt
    fi
done
```

These manifests drive the rest of the pipeline — directory creation, file conversion, and output placement all read from them.

## The Image Pipeline

The TIFF module is the most involved. For each source TIFF, it:

1. Converts to JPEG (with optional watermark)
2. Generates a 100x100 thumbnail
3. Generates a 500px preview
4. Creates a Zoomify tileset for deep-zoom viewing

```bash
while read line; do
    no_ext="${line%.*}"

    if [ "$WATERMARK" == "true" ]; then
        apply_watermark "$SOURCE/$line" "$MODULE_WORKING_DIRECTORY/jpeg/$no_ext.jpg"
    else
        convert "$SOURCE/$line" "$MODULE_WORKING_DIRECTORY/jpeg/$no_ext.jpg"
    fi

    convert -resize 100x100 "$MODULE_WORKING_DIRECTORY/jpeg/$no_ext.jpg" \
        "$PROCESSED_DIR/100/$no_ext.jpg"
    convert -resize 500x500 "$MODULE_WORKING_DIRECTORY/jpeg/$no_ext.jpg" \
        "$PROCESSED_DIR/500/$no_ext.jpg"

    if [ "$NOTILES" == "false" ]; then
        vips dzsave "$MODULE_WORKING_DIRECTORY/jpeg/$no_ext.jpg"[autorotate=true] \
            "$PROCESSED_DIR/zoom/$no_ext" --layout zoomify --vips-progress
    fi
done < $MODULE_WORKING_DIRECTORY/files.txt
```

The tileset generation uses libvips with the Zoomify layout, which produces a tile pyramid that can be loaded by any deep-zoom viewer. The `autorotate=true` flag handles images with EXIF rotation metadata so they display correctly regardless of how the scanner saved them.

## Adaptive Watermarking

The watermark system picks between two watermark sizes based on image dimensions. Larger images get a smaller repeating watermark tile so it doesn't dominate the image, while smaller images get a proportionally larger one:

```bash
apply_watermark() {
    local infile="$1"
    local outfile="$2"

    basew=$(identify -format '%w' "$infile")
    baseh=$(identify -format '%h' "$infile")

    if [ $baseh -gt 1024 ] || [ $basew -gt 1024 ]; then
        wmfile="$SCRIPT_DIR/watermark_small.png"
    else
        wmfile="$SCRIPT_DIR/watermark.png"
    fi

    composite -tile "$wmfile" "$infile" "$outfile"
}
```

ImageMagick's `composite -tile` repeats the watermark PNG across the entire image surface — simple and effective.

## Document and Audio Conversion

The other modules follow the same pattern but with different tools. Word documents go through unoconv to produce PDFs, HTML files get converted via Pandoc and LaTeX, and WAV files are compressed to 320kbps MP3 with LAME:

```bash
# Word to PDF
unoconv -f pdf -o "$PROCESSED_DIR/pdf/$line.pdf" "$SOURCE/$line"

# WAV to MP3
lame -b 320 -h "$SOURCE/$line" "$PROCESSED_DIR/audio/$line.mp3"
```

## What I'd Do Differently

This tool has been through four major versions. If I were starting fresh, I'd probably reach for a language with better error handling and parallelism — Bash gets the job done but swallowing errors silently is its default behavior, and there's no easy way to process files concurrently across CPU cores.

That said, the dependency on external tools like ImageMagick, VIPS, and unoconv means Bash is actually a reasonable glue language here. The real work happens in those tools — the script is just orchestration.

The module system has held up well. Adding a new format means dropping a new `.sh` file into the modules directory. The orchestrator picks it up automatically. Sometimes the simplest architecture is the right one.


---