Drift Remediation

Handling manual kubectl changes with automated sync, prune policies, ignoreDifferences, and audit logs.

7 min readCI/CD & GitOpsPlatform

Remediation Approaches

Drift remediation restores the cluster to the desired state after manual changes. There are three approaches: automatic correction (self-heal), manual sync (human-initiated), and ignore (accept the drift). The right approach depends on the field that drifted.

Approach	When to Use	Risk
Self-heal	Replicas, labels, annotations	May conflict with HPA or controllers
Manual sync	Complex changes, debugging	Requires human intervention
Ignore differences	HPA-managed fields, mutating webhooks	Intentional drift accepted

ArgoCD Self-Heal

ArgoCD self-heal with ignoreDifferences


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-repo.git
    targetRevision: main
    path: applications/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    # HPA manages replicas
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    # Service mesh sidecar injects containers
    - group: ""
      kind: Pod
      jqPathExpressions:
        - .spec.containers[] | select(.name == "istio-proxy")
    # Mutating webhook adds annotations
    - group: ""
      kind: Service
      jsonPointers:
        - /metadata/annotations
    # CronJob schedule may be adjusted
    - group: batch
      kind: CronJob
      jsonPointers:
        - /spec/schedule

ignoreDifferences tells ArgoCD to not consider certain fields when calculating drift. This prevents conflicts with controllers like HPA that manage replicas, or service meshes that inject sidecars.

Flux Remediation

Flux Kustomization remediation


apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  path: ./overlays/production
  sourceRef:
    kind: GitRepository
    name: myapp
  prune: true
  force: true
  # Wait for resources to be ready
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: production
    - apiVersion: v1
      kind: Service
      name: myapp
      namespace: production
  timeout: 5m
  # Retry on transient failures
  retryInterval: 1m
  dependsOn:
    - name: infrastructure

Flux reconciles every interval. force: true uses server-side apply to handle conflicts. healthChecks ensures resources are healthy after reconciliation. retryInterval controls how often Flux retries failed reconciliations.

Ignore Differences

Comprehensive ignoreDifferences


ignoreDifferences:
  # Replicas managed by HPA
  - group: apps
    kind: Deployment
    jsonPointers:
      - /spec/replicas
  - group: apps
    kind: StatefulSet
    jsonPointers:
      - /spec/replicas

  # Resource quotas modified by admission controllers
  - group: ""
    kind: ResourceQuota
    jsonPointers:
      - /status

  # Status fields managed by controllers
  - group: networking.k8s.io
    kind: Ingress
    jsonPointers:
      - /status

  # Labels added by service mesh
  - group: ""
    kind: Service
    jqPathExpressions:
      - .metadata.labels | select(. != null) | to_entries[] | select(.key | startswith("service.istio.io"))

  # Annotations added by cert-manager
  - group: ""
    kind: Service
    jqPathExpressions:
      - .metadata.annotations | select(. != null) | to_entries[] | select(.key == "cert-manager.io/issuer")

Multiple ignore rules can target different fields in different resource types. Use jqPathExpressions for complex selections. jsonPointers are simpler but less flexible.

Prune Configurations

Prune policies in ArgoCD


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      # Delete resources removed from Git
      - PrunePropagationPolicy=foreground
      # Delete after applying new resources
      - PruneLast=true
      # Only prune annotated resources
      - Prune=true

---
# Annotation to opt-in to pruning
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
  annotations:
    argocd.argoproj.io/prune: "true"

---
# Annotation to opt-out of pruning
apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-config
  annotations:
    argocd.argoproj.io/prune: "false"

PrunePropagationPolicy=foreground ensures dependents are deleted before owners. PruneLast applies new resources before deleting old ones. Resources without the prune annotation are never pruned.

Audit Logging

Kubernetes audit logging configuration


apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all changes to production namespace
  - level: RequestResponse
    namespaces: ["production"]
    resources:
      - group: ""
        resources: ["pods", "deployments", "services", "configmaps"]
    verbs: ["create", "update", "patch", "delete"]

  # Log all kubectl exec requests
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec"]
    verbs: ["create"]

  # Don't log read-only requests
  - level: None
    verbs: ["get", "list", "watch"]

Kubernetes audit logging captures all API server requests. Configure policies to log writes to production resources. Combine with SIEM tools to detect unauthorized changes and trigger alerts.

Monitor for sync failures

Set up alerts for ArgoCD and Flux sync failures. A sync failure means the desired state was not applied. This could indicate a misconfiguration, permission issue, or resource conflict that needs investigation.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 5 · Platform

GitOps Operations