Stage 2 · Tools
DevOps Scripts in Practice
Backup Automation
Incremental backups, compression, rotation, and verification for data safety.
Backup Strategies
| Strategy | Description | Best For |
|---|---|---|
| Full | Complete copy of all data | Small datasets, initial backups |
| Incremental | Only changes since last backup | Large datasets, frequent backups |
| Differential | Changes since last full backup | Balance of speed and restore |
The 3-2-1 rule: keep 3 copies of data, on 2 different media, with 1 offsite. Automation ensures backups happen consistently.
Database Backups
#!/usr/bin/env bash
set -euo pipefail
readonly BACKUP_DIR="/backups/postgres"
readonly DB_NAME="myapp"
readonly RETENTION_DAYS=7
mkdir -p "$BACKUP_DIR"
# Dump with timestamp
TIMESTAMP=$(date '+%Y%m%d-%H%M%S')
BACKUP_FILE="$BACKUP_DIR/$DB_NAME-$TIMESTAMP.sql.gz"
pg_dump "$DB_NAME" | gzip > "$BACKUP_FILE"
echo "Backup created: $BACKUP_FILE"
echo "Size: $(du -h "$BACKUP_FILE" | cut -f1)"
# Verify backup is not empty
if [[ ! -s "$BACKUP_FILE" ]]; then
echo "ERROR: Backup file is empty"
rm -f "$BACKUP_FILE"
exit 1
fi
# Cleanup old backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete
echo "Cleaned up backups older than $RETENTION_DAYS days"pg_dump creates a SQL dump. gzip compresses it. The timestamp ensures unique filenames. The verification step catches empty backups. find cleans up old backups.
#!/usr/bin/env bash
set -euo pipefail
readonly BACKUP_DIR="/backups/mysql"
readonly DB_NAME="myapp"
readonly TIMESTAMP=$(date '+%Y%m%d-%H%M%S')
mkdir -p "$BACKUP_DIR"
mysqldump \
--user="$DB_USER" \
--password="$DB_PASS" \
--single-transaction \
--routines \
--triggers \
"$DB_NAME" | gzip > "$BACKUP_DIR/$DB_NAME-$TIMESTAMP.sql.gz"
echo "MySQL backup complete"--single-transaction creates a consistent snapshot without locking tables. --routines and --triggers include stored procedures and triggers. Use environment variables for credentials.
File Backups
#!/usr/bin/env bash
set -euo pipefail
readonly SOURCE="/var/www/app"
readonly DEST="/backups/files"
readonly TIMESTAMP=$(date '+%Y%m%d-%H%M%S')
# Incremental backup with rsync
rsync -av --delete \
--link-dest="$DEST/latest" \
"$SOURCE/" "$DEST/$TIMESTAMP/"
# Update latest symlink
ln -snf "$DEST/$TIMESTAMP" "$DEST/latest"
echo "File backup complete: $DEST/$TIMESTAMP"rsync --link-dest uses hard links for unchanged files. This gives incremental backup efficiency with full backup convenience. Unchanged files are shared between backups, saving disk space.
Compression and Encryption
#!/usr/bin/env bash
# gzip — fast, good compression
tar -czf backup.tar.gz /data
# bzip2 — slower, better compression
tar -cjf backup.tar.bz2 /data
# xz — slowest, best compression
tar -cJf backup.tar.xz /data
# pigz — parallel gzip (multi-core)
tar -I pigz -cf backup.tar.gz /data
# zstd — modern, fast compression (recommended)
tar --zstd -cf backup.tar.zst /data
# Encrypted backup
tar -czf - /data | gpg -c -o backup.tar.gz.gpg
# Encrypted with symmetric key
tar -czf - /data | \
openssl enc -aes-256-cbc -salt -out backup.tar.gz.enczstd offers the best balance of speed and compression. pigz uses multiple cores for faster gzip. For sensitive data, encrypt backups with gpg or openssl. Store encryption keys securely.
Retention and Rotation
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="/backups"
KEEP_DAILY=7
KEEP_WEEKLY=4
KEEP_MONTHLY=12
rotate_backups() {
echo "Starting backup rotation..."
# Delete daily backups older than 7 days
find "$BACKUP_DIR/daily" -name "*.tar.gz" -mtime +$KEEP_DAILY -delete
# Delete weekly backups older than 4 weeks
find "$BACKUP_DIR/weekly" -name "*.tar.gz" -mtime +$((KEEP_WEEKLY * 7)) -delete
# Delete monthly backups older than 12 months
find "$BACKUP_DIR/monthly" -name "*.tar.gz" -mtime +$((KEEP_MONTHLY * 30)) -delete
echo "Rotation complete"
}
# Daily backup
daily_backup() {
mkdir -p "$BACKUP_DIR/daily"
local file="$BACKUP_DIR/daily/backup-$(date +%Y%m%d).tar.gz"
tar -czf "$file" /data
echo "Daily backup: $file"
}
# Weekly backup (Sundays)
if [[ $(date +%u) -eq 7 ]]; then
mkdir -p "$BACKUP_DIR/weekly"
tar -czf "$BACKUP_DIR/weekly/backup-$(date +%Y%m%d).tar.gz" /data
fi
# Monthly backup (1st of month)
if [[ $(date +%d) -eq 1 ]]; then
mkdir -p "$BACKUP_DIR/monthly"
tar -czf "$BACKUP_DIR/monthly/backup-$(date +%Y%m).tar.gz" /data
fi
rotate_backupsRetention policies manage backup disk usage. Keep daily backups for a week, weekly for a month, monthly for a year. This provides good recovery options while managing storage costs.
Backup Verification
#!/usr/bin/env bash
set -euo pipefail
verify_backup() {
local backup_file="$1"
echo "Verifying: $backup_file"
# Check file exists and is not empty
if [[ ! -s "$backup_file" ]]; then
echo "FAIL: File is empty or missing"
return 1
fi
# Check archive integrity
if ! tar -tzf "$backup_file" >/dev/null 2>&1; then
echo "FAIL: Archive is corrupted"
return 1
fi
# Check file count
local file_count
file_count=$(tar -tzf "$backup_file" | wc -l)
echo " Files in archive: $file_count"
if (( file_count < 10 )); then
echo "WARN: Suspiciously few files"
return 1
fi
echo "PASS: Backup is valid"
return 0
}
# Verify all backups in directory
for backup in /backups/*.tar.gz; do
verify_backup "$backup" || {
echo "ALERT: Invalid backup found: $backup"
# Send notification
}
doneVerify backups by checking file existence, archive integrity (tar -tzf), and file count. Automated verification catches corrupted backups before you need to restore them.
An untested backup is not a backup. Regularly restore backups to a test environment to verify they work. A backup that cannot be restored is worthless.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.