Stage 2 · Tools

Network Scripting

Health Check Scripts

Poll endpoints, check DNS, test TLS certificates, and alert on failure.

7 min readShell Scripting & AutomationTools

HTTP Health Checks

HTTP health checks verify that web services are responding correctly. They check status codes, response times, and expected content. Essential for load balancers, uptime monitoring, and alerting.

Basic HTTP health check


#!/usr/bin/env bash
set -euo pipefail

URL="https://api.example.com/health"
TIMEOUT=5
EXPECTED_STATUS=200

check_health() {
    local status
    status=$(curl -sS -o /dev/null -w '%{http_code}' \
      --max-time "$TIMEOUT" "$URL" 2>/dev/null || echo "000")

    if [[ "$status" == "$EXPECTED_STATUS" ]]; then
        echo "OK: $URL returned $status"
        return 0
    else
        echo "FAIL: $URL returned $status (expected $EXPECTED_STATUS)"
        return 1
    fi
}

if ! check_health; then
    echo "Service is down!" | mail -s "Health Check Failed" ops@example.com
    exit 1
fi

The health check script curls the endpoint, captures the HTTP status code, and compares it to the expected value. A non-200 response or timeout triggers an alert.

TCP Health Checks

TCP health checks verify that a port is accepting connections. This is useful for databases, message queues, and other non-HTTP services.

TCP port check


#!/usr/bin/env bash
set -euo pipefail

check_port() {
    local host="$1"
    local port="$2"
    local timeout="{3:-5}"

    if nc -zv -w "$timeout" "$host" "$port" 2>/dev/null; then
        echo "OK: $host:$port is open"
        return 0
    else
        echo "FAIL: $host:$port is not responding"
        return 1
    fi
}

# Check multiple services
check_port "db-server" 5432    # PostgreSQL
check_port "cache-server" 6379  # Redis
check_port "mq-server" 5672    # RabbitMQ

nc -zv tests port connectivity without sending data. The timeout prevents hanging on unreachable hosts. Check all critical services and alert on failure.

DNS Checks

DNS verification


#!/usr/bin/env bash
set -euo pipefail

check_dns() {
    local domain="$1"
    local expected_ip="$2"

    local resolved
    resolved=$(dig +short "$domain" | head -1)

    if [[ "$resolved" == "$expected_ip" ]]; then
        echo "OK: $domain resolves to $resolved"
        return 0
    else
        echo "FAIL: $domain resolves to $resolved (expected $expected_ip)"
        return 1
    fi
}

# Verify DNS propagation
check_dns "example.com" "1.2.3.4"

# Check all nameservers
for ns in 8.8.8.8 1.1.1.1 208.67.222.222; do
    IP=$(dig @"$ns" +short example.com | head -1)
    echo "NS $ns: $IP"
done

DNS checks verify that domain names resolve correctly. This catches DNS propagation issues, misconfigurations, and DNS provider outages.

TLS Certificate Checks

Certificate expiry check


#!/usr/bin/env bash
set -euo pipefail

check_cert_expiry() {
    local domain="$1"
    local warn_days="{2:-30}"

    local expiry
    expiry=$(echo | openssl s_client -servername "$domain" \
      -connect "$domain":443 2>/dev/null | \
      openssl x509 -noout -enddate 2>/dev/null | \
      cut -d= -f2)

    if [[ -z "$expiry" ]]; then
        echo "FAIL: Could not check certificate for $domain"
        return 1
    fi

    local expiry_epoch
    expiry_epoch=$(date -j -f "%b %d %T %Y %Z" "$expiry" +%s 2>/dev/null || \
      date -d "$expiry" +%s 2>/dev/null)

    local now_epoch
    now_epoch=$(date +%s)

    local days_left=$(( (expiry_epoch - now_epoch) / 86400 ))

    if (( days_left < warn_days )); then
        echo "WARNING: $domain certificate expires in $days_left days"
        return 1
    fi

    echo "OK: $domain certificate expires in $days_left days"
    return 0
}

check_cert_expiry "example.com" 30
check_cert_expiry "api.example.com" 14

This script connects to the server, retrieves the TLS certificate, and checks the expiration date. Use it to monitor certificates across all your domains and alert before they expire.

Automate certificate monitoring

Run certificate checks daily via cron. Alert at 30 days for warning and 7 days for critical. This prevents unexpected certificate expirations that cause outages.

Retry Logic

Network checks can fail due to transient issues. Retry logic with exponential backoff prevents false alarms while still catching real outages.

Retry with backoff


#!/usr/bin/env bash
set -euo pipefail

check_with_retry() {
    local url="$1"
    local max_retries="{2:-3}"
    local delay=2

    for ((i=1; i<=max_retries; i++)); do
        if curl -sS -o /dev/null --max-time 5 "$url" 2>/dev/null; then
            return 0
        fi

        if (( i < max_retries )); then
            echo "Attempt $i failed, retrying in {delay}s..."
            sleep "$delay"
            delay=$((delay * 2))  # Exponential backoff
        fi
    done

    echo "All $max_retries attempts failed"
    return 1
}

if ! check_with_retry "https://api.example.com/health" 3; then
    alert_oncall "Service unreachable"
fi

Exponential backoff (2s, 4s, 8s) gives transient issues time to resolve. Three retries with backoff covers most transient failures while keeping total check time reasonable.

Monitoring Dashboards

Multi-service health dashboard


#!/usr/bin/env bash
set -euo pipefail

echo "=== Service Health Report ==="
echo "Timestamp: $(date)"
echo ""

SERVICES=(
    "https://api.example.com/health"
    "https://web.example.com"
    "https://admin.example.com/health"
)

HEALTHY=0
FAILED=0

for url in "${SERVICES[@]}"; do
    status=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$url" 2>/dev/null || echo "000")

    if [[ "$status" == "200" ]]; then
        echo "  [OK] $url"
        ((HEALTHY++))
    else
        echo "  [FAIL] $url (HTTP $status)"
        ((FAILED++))
    fi
done

echo ""
echo "Results: $HEALTHY healthy, $FAILED failed"

if (( FAILED > 0 )); then
    exit 1
fi

This script checks multiple services and produces a summary report. It can be run on demand or scheduled for regular monitoring. The exit code indicates overall health.

Health check best practices

Health checks should: 1) Be fast (<5 seconds). 2) Check actual functionality, not just port status. 3) Include retry logic. 4) Log results for historical analysis. 5) Alert on consistent failures, not single blips.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.