Project challenges / verified progress
Beacon: turn a program into a service

The engineering notebook

Shut down politely

What should happen between SIGTERM and the process actually exiting?

Loading Vault account...
Phase 4 — Process behaviour

Step 01 of 06

Understand the idea

When a service needs to stop, its supervisor usually gives it time to finish current work. In this lesson, Beacon responds to SIGTERM, stops accepting new work, and exits before its shutdown deadline.

THIRTY SECOND EXITsecondsSIGTERMstop requested0Drainfinish work25sDeadlineforce exit5s left
A graceful shutdown is a budget. Stop accepting new work immediately, give current work a bounded window, and exit before the supervisor reaches for SIGKILL.
Step 01

Key technical ideas

Signals are process-level messages

SIGTERM is the standard polite stop signal. SIGINT usually comes from Ctrl-C. SIGKILL cannot be caught, delayed or cleaned up after. Your code only gets a chance to close databases and drain requests if it responds to catchable signals before the outside system loses patience.

NotifyContext connects signals to Go cancellation

signal.NotifyContext returns a context cancelled when selected signals arrive. Passing that context to the scheduler, server and checks creates one stop line through the process. Remember to call the returned stop function so signal handling is released during cleanup.

HTTP shutdown is not server close

Server.Close drops active connections. Server.Shutdown(ctx) first closes listeners so new requests stop, then waits for active handlers until the context deadline. That is the behaviour users expect during deploys: no new work, but current requests get a fair chance.

Kubernetes will not wait forever

In Kubernetes, a pod receives SIGTERM and has terminationGracePeriodSeconds, often 30s, before SIGKILL. If Beacon needs 45s to drain, Kubernetes will cut it off. Designing a 25s shutdown deadline inside a 30s grace period leaves room for runtime and network slop.

Cancel on SIGTERM
package main

import (
	"context"
	"fmt"
	"os"
	"os/signal"
	"syscall"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	<-ctx.Done()
	fmt.Println("shutdown requested")
}

The program waits until a signal cancels the context. Beacon will use the same cancellation to stop the scheduler and begin HTTP server shutdown.

Stop signals

SignalCatchableUse

SIGINT

Yes

Terminal interrupt

SIGTERM

Yes

Supervisor stop

SIGKILL

No

Forced kill

SIGHUP

Yes

Reload in some daemons

Technical terms used at work

  • Drain — Let in-flight work finish while refusing new work.

  • SIGTERM — A catchable request for a process to terminate.

  • SIGKILL — An uncatchable forced process kill.