GitOps Disaster Recovery

Rebuilding clusters from Git, backups, registry mirrors, CRD ordering, and bootstrap credentials.

8 min readCI/CD & GitOpsPlatform

DR Model

GitOps disaster recovery is uniquely powerful: the entire cluster state is defined in Git. To recover from a disaster, provision a new cluster and point it at the Git repository. The GitOps controller reconciles the desired state. This is the core promise of GitOps — reproducible infrastructure.

However, not everything can be recovered from Git. Persistent data, secrets, certificates, and cluster-specific configurations need separate backup strategies. A complete DR plan combines Git-based recovery with backup restoration.

Cluster Rebuild from Git

Bootstrap script for new cluster


#!/bin/bash
# bootstrap.sh - Rebuild cluster from Git

set -euo pipefail

CLUSTER_NAME="${1:-production}"
GIT_REPO="https://github.com/myorg/gitops-repo.git"
GIT_BRANCH="main"

echo "Bootstrapping cluster: $CLUSTER_NAME"

# 1. Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 2. Wait for ArgoCD to be ready
kubectl wait --for=condition=available deployment/argocd-server \
  -n argocd --timeout=300s

# 3. Install ArgoCD CLI
curl -sSL -o argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
chmod +x argocd
sudo mv argocd /usr/local/bin/

# 4. Login and configure
ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)
argocd login localhost:8080 --username admin --password "$ARGOCD_PASSWORD"

# 5. Create root application
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: $GIT_REPO
    targetRevision: $GIT_BRANCH
    path: apps
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

echo "Bootstrap complete. ArgoCD will sync all applications."

This bootstrap script provisions a new cluster with ArgoCD and a root application. The root application points to the apps directory in the Git repository. ArgoCD automatically creates and syncs all applications defined there.

Backup Strategies

Velero backup configuration


apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - production
      - staging
    includedResources:
      - "*"
    excludedResources:
      - events
      - events.events.k8s.io
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 720h

---
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: myorg-velero-backups
    prefix: clusters/production
  config:
    region: us-east-1

Velero backs up Kubernetes resources and persistent volumes. The schedule runs daily at 2 AM. ttl: 720h keeps backups for 30 days. Velero can restore to a new cluster during disaster recovery.

Registry Mirrors

Containerd registry mirror configuration


# /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["https://mirror.example.com"]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."ghcr.io"]
  endpoint = ["https://ghcr-mirror.example.com"]

[plugins."io.containerd.grpc.v1.cri".registry.configs."mirror.example.com".tls]
  ca_file = "/etc/containerd/ca.crt"
  cert_file = "/etc/containerd/client.crt"
  key_file = "/etc/containerd/client.key"

Registry mirrors cache container images locally. During a disaster, if the public registry is unavailable, the mirror provides cached images. This prevents cluster rebuilds from failing due to registry outages.

CRD Ordering

CRD sync waves in ArgoCD


# CRDs must be applied before their instances
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: rollouts.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-10"

---
# CRD instances depend on CRDs
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
  annotations:
    argocd.argoproj.io/sync-wave: "0"

---
# Services depend on Rollouts
apiVersion: v1
kind: Service
metadata:
  name: myapp
  annotations:
    argocd.argoproj.io/sync-wave: "1"

CRDs must be installed before any resources that use them. Sync waves with negative values ensure CRDs are created first. Without this ordering, ArgoCD fails to create resources that depend on CRDs.

Bootstrap Credentials

Secure bootstrap credential distribution


# Use sealed secrets for bootstrap credentials
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: git-credentials
  namespace: flux-system
spec:
  encryptedData:
    username: AgBy3i4OJSWK+PiTySYZZA9rO43...
    password: AgBTdVZz4r8+ZzHJ4Q5x4R5N...

---
# Use external secrets for production credentials
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: git-credentials
  namespace: flux-system
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: git-credentials
  data:
    - secretKey: password
      remoteRef:
        key: production/git-credentials/password

Bootstrap credentials are encrypted in Git using Sealed Secrets or fetched from an external secret store. This prevents plaintext credentials from being stored in the Git repository while still enabling automated recovery.

Test your DR plan quarterly

A disaster recovery plan that has never been tested is not a plan — it is a hypothesis. Quarterly DR drills verify that your recovery process works, identify gaps, and build team confidence. Simulate a cluster loss and time the recovery.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 5 · Platform

GitOps Operations