Stage 2 · Tools

Process & System Management

Writing systemd Services

Create unit files, manage dependencies, and write clean ExecStart scripts for services.

7 min readShell Scripting & AutomationTools

systemd Overview

systemd is the init system on most modern Linux distributions. It manages services, mounts, timers, and more. Writing proper systemd units makes your scripts production-ready with automatic restart, logging, and dependency management.

systemd commands


# List running services
systemctl list-units --type=service

# Check service status
systemctl status myapp

# Start/stop/restart
sudo systemctl start myapp
sudo systemctl stop myapp
sudo systemctl restart myapp

# Enable at boot
sudo systemctl enable myapp

# Reload after config change
sudo systemctl daemon-reload

systemctl is the command-line interface for systemd. status shows detailed information including recent log entries, PID, memory usage, and cgroup information.

Service Unit Files

Service unit files define how a service runs. They specify the command to execute, restart behavior, resource limits, and dependencies. Unit files live in /etc/systemd/system/ for custom services.

Basic service unit


# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
Documentation=https://example.com/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/start.sh
ExecStop=/opt/myapp/bin/stop.sh

# Restart policy
Restart=on-failure
RestartSec=5

# Environment
Environment=NODE_ENV=production
EnvironmentFile=/opt/myapp/.env

# Resource limits
LimitNOFILE=65535
MemoryMax=1G

[Install]
WantedBy=multi-user.target

The [Unit] section defines metadata and dependencies. [Service] defines how the service runs. [Install] defines how it is installed. Type=simple means the process is the main service process.

Use EnvironmentFile for secrets

Never put secrets directly in unit files. Use EnvironmentFile to load them from a separate file with restricted permissions: chmod 600 /opt/myapp/.env.

ExecStart Scripts

ExecStart specifies the command to run. It must use absolute paths. The script must not fork — systemd tracks the main PID. For scripts that fork, use Type=forking.

ExecStart patterns


# Direct command (no wrapper script needed)
ExecStart=/usr/bin/python3 /opt/myapp/server.py

# Shell script (must set -e and not fork)
ExecStart=/opt/myapp/bin/start.sh

# With arguments
ExecStart=/usr/bin/node /opt/myapp/dist/server.js --port=8080

# For daemonizing scripts (Type=forking)
ExecStart=/opt/myapp/bin/daemon.sh
PIDFile=/var/run/myapp.pid

# Pre-start command
ExecStartPre=/opt/myapp/bin/migrate.sh

# Post-start command
ExecStartPost=/opt/myapp/bin/healthcheck.sh

For Type=simple, the command must not fork. The process IS the service. For Type=forking, the command forks and the parent exits — systemd tracks the child via PIDFile.

Restart Policies

Restart policies determine when systemd restarts a failed service. This is essential for high availability — services should recover from crashes automatically.

Policy	Behavior
no	Never restart (default)
on-success	Restart only on clean exit (code 0)
on-failure	Restart on non-zero exit or signal (except SIGTERM)
on-abnormal	Restart on signal, timeout, or watchdog
on-abort	Restart only on unclean signal
always	Always restart (default for Type=oneshot)

Restart configuration


# Restart on any failure with delay
Restart=on-failure
RestartSec=5

# Restart immediately
Restart=always
RestartSec=0

# Limit restart rate (max 5 restarts in 60 seconds)
StartLimitIntervalSec=60
StartLimitBurst=5

# Watchdog (restart if not responsive)
WatchdogSec=30

StartLimitBurst prevents restart loops — if the service fails too many times, systemd stops trying. WatchdogSec restarts the service if it stops heartbeating.

Dependencies and Ordering

systemd manages service dependencies with Wants/Requires (what) and After/Before (when). Understanding these is essential for correct startup ordering.

Dependency patterns


# After: start after this unit (but don't require it)
After=network-online.target

# Wants: try to start this unit (non-fatal if it fails)
Wants=postgresql.service

# Requires: this unit must be running (fatal if it fails)
Requires=redis.service

# BindsTo: stop this unit if the dependency stops
BindsTo=docker.service

# Conflicts: cannot run at the same time
Conflicts=iptables.service

After controls ordering. Wants/Requires control binding. BindsTo means the service stops if its dependency stops. Conflicts prevents simultaneous execution.

Logging with journald

systemd captures stdout/stderr from services automatically via journald. You do not need to configure log files — journalctl provides powerful querying.

journalctl patterns


# View service logs
journalctl -u myapp

# Follow logs (like tail -f)
journalctl -u myapp -f

# Logs since last boot
journalctl -u myapp -b

# Logs in time range
journalctl -u myapp --since "2025-01-15" --until "2025-01-16"

# Logs with priority
journalctl -u myapp -p err  # Errors only

# Logs from specific PID
journalctl _PID=1234

# Persistent logs (survive reboot)
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

journalctl -u is the primary way to view service logs. -p filters by priority (emerg, alert, crit, err, warning, notice, info, debug). -b limits to current boot.

Use StandardOutput=syslog for remote logging

By default, journald stores logs locally. For centralized logging, configure StandardOutput=syslog to forward to rsyslog, or use systemd-journal-remote for network forwarding.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.