Platform Engineering

Graceful Shutdown Isn't Optional: The Rolling Deploy That Dropped 3% of Requests

Every deploy, every day, for months, we silently dropped a small percentage of real requests — and nobody noticed until we actually measured it, because 3% doesn't look like an outage.

July 10, 20267 min read

Kubernetesgraceful shutdownrolling deploysSIGTERM

Nobody filed an incident. There was no page, no red dashboard, no angry customer email that anyone connected to a deploy. And yet every single rolling deploy — dozens per week — was silently dropping around 3% of in-flight requests, for months, because our application never handled SIGTERM correctly. It took someone deliberately measuring request success rate *during* a deploy window, not just before and after, to notice it at all.

How we even found this

A teammate was investigating unrelated flaky end-to-end tests and noticed the failures clustered suspiciously close to deploy timestamps. Pulling error-rate metrics filtered to a tight 30-second window around each rolling deploy (rather than the usual hourly aggregate, which completely smooths this out) showed a small, consistent, unmistakable spike: roughly 3% of requests returning connection-reset or 502 errors, every single time, for the duration of each pod replacement.

What Kubernetes actually does when it terminates a pod

During a rolling update, Kubernetes sends SIGTERM to the container, then waits up to terminationGracePeriodSeconds (default 30s) before force-killing it with SIGKILL. Separately, and concurrently, it removes the pod's IP from the Service's endpoint list. The critical detail almost nobody accounts for: these two things happen in parallel, not in a guaranteed safe order. There is a real window where a pod has received SIGTERM (and, in the naive case, immediately stops accepting connections) while some in-flight requests are still being routed to it, or new connections are still arriving because the endpoint removal hasn't fully propagated to every kube-proxy / load balancer yet.

The race between SIGTERM and endpoint-list propagation

Rendering diagram…

The gap that actually caused the drops

Our application's default HTTP server behavior on SIGTERM was to exit almost immediately — Go's default net/http server doesn't gracefully drain by itself unless you explicitly call Shutdown(). So the moment SIGTERM arrived, the process exited within milliseconds, abruptly closing any connections that were mid-flight or had just been routed to it in that small propagation-delay window.

The fix: a deliberate three-part shutdown sequence

The fix has three parts, and all three matter — most guides only mention the second one, which alone isn't sufficient.

A pre-stop delay before anything else happens. Give the endpoint-removal propagation time to actually complete across the cluster before the app stops accepting new connections.
Stop accepting new connections, but let in-flight ones finish. This is what http.Server.Shutdown() actually does in Go — it's the part everyone thinks is the whole fix.
A termination grace period long enough for the slowest realistic in-flight request, not just the average one.

Part 1: preStop hook — the part most teams skip

containers:
  - name: api
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "5"]
    terminationGracePeriodSeconds: 30

This alone fixed roughly 60% of the drops we measured. SIGTERM isn't sent until after this sleep completes, giving the Service's endpoint list time to propagate the pod's removal to every kube-proxy/load balancer before the app even starts shutting down. Requests stop arriving before the app stops accepting them.

Gopart-2:-graceful-http-shutdown-(the-part-everyone-remembers).go

21 linesLn 1, Col 1Go

srv.Shutdown() closes the listener immediately (no new connections accepted) but lets already-accepted requests finish naturally, up to the given deadline. Without this, the naive default is the process exiting the instant SIGTERM arrives, killing every in-flight request mid-response.

Part 3: grace period sized for the slowest real request, not the average

terminationGracePeriodSeconds: 35
# 5s preStop delay + up to 20s for Shutdown() to drain + 10s buffer.
# If your p99 request latency is 8s, don't set this to 15 total —
# a burst of slow requests right as a deploy starts will get killed anyway.

We initially set this too low based on average latency and still saw occasional drops — the requests being cut off were specifically the slow-tail ones (p99+), which is exactly the population most likely to still be in-flight when a pod terminates.

The one more thing: readiness probes need to fail fast on shutdown too

We also updated the app to fail its readiness probe the instant it receives SIGTERM, before even starting the drain sequence. This gives Kubernetes an additional, faster signal (beyond just the endpoint removal from pod deletion) to stop routing new traffic to a terminating pod — belt-and-suspenders on top of the preStop delay.

Result: 3% dropped requests per deploy → effectively 0%

After all three changes, the same tight-window measurement around deploy timestamps showed no measurable increase in error rate during rolling updates — deploys became genuinely invisible to real traffic, not just invisible to our hourly-aggregated dashboards.

How to check if this is happening to you right now

Don't trust hourly or even 5-minute error-rate graphs — they average away exactly this kind of short, sharp spike. Pull error rate at 10-15 second granularity, filtered to a tight window around your actual deploy timestamps (most CI/CD systems log these). If you see even a small, repeatable bump exactly at pod-replacement time, you have this bug, whether or not anyone's ever complained about it.