Stage 3 · Build
Network Debugging in Production
Debugging Connection Failures
Timeouts, refused connections, DNS resolution failures, and firewall debugging.
Connection Refused
Connection refused (ECONNREFUSED) means the TCP SYN was rejected. The server sent a RST packet because no process is listening on that port, or a firewall explicitly rejected the connection.
# Check if anything is listening on the port
ss -tlnp | grep :8080
netstat -tlnp | grep :8080
# Check if the process is running
ps aux | grep myapp
# Check if it is bound to the right address
# 0.0.0.0 = all interfaces
# 127.0.0.1 = localhost only
# 10.0.0.1 = specific interface
# Check with nc
nc -zv localhost 8080
nc -zv 10.0.0.1 8080
# Check firewall rules
sudo iptables -L -n | grep 8080
sudo ufw status | grep 8080The most common cause is the service not running or not listening on the expected port. The second most common is binding to 127.0.0.1 instead of 0.0.0.0 — the service is running but only accepts connections from localhost.
Connection Timeout
# Timeout with no response (vs refused)
# Refused = RST packet received (fast)
# Timeout = SYN sent, no response (waits)
# Test with nc timeout
nc -zv -w 5 api.example.com 443
# Test with curl timeout
curl --connect-timeout 5 https://api.example.com
# Check if route exists
ip route get 10.0.1.50
# Check if host is reachable
ping -c 3 -W 2 10.0.1.50
# Check if port is reachable through firewall
sudo tcpdump -i eth0 host 10.0.1.50 and port 443 -c 5
# If SYN goes out but no SYN-ACK comes back, firewall is blockingTimeout means the packet went out but no response came back. This is typically a firewall issue — either on the server, in the network, or in a security group (AWS). The tcpdump output confirms whether the SYN leaves and whether a SYN-ACK returns.
Firewall Debugging
# Linux iptables
sudo iptables -L -n -v
sudo iptables -t nat -L -n -v
# Check for rules blocking your traffic
sudo iptables -L -n | grep -E "(REJECT|DROP)"
# Cloud security groups (AWS)
aws ec2 describe-security-groups --group-ids sg-abc123
# Check if security group allows your traffic
# Inbound: source IP, port, protocol
# Outbound: destination IP, port, protocol
# Test connectivity through firewall
# From allowed IP
curl -v http://10.0.1.50:8080/health
# From blocked IP (should timeout or refuse)
curl -v --connect-timeout 3 http://10.0.1.50:8080/healthFirewalls can be iptables on the host, security groups in the cloud, or network ACLs in the VPC. Check all three. A common mistake: security group allows port 8080 but network ACL blocks return traffic on ephemeral ports.
DNS Resolution Failures
# Check DNS configuration
cat /etc/resolv.conf
# Test DNS resolution
dig example.com +short
# If dig works but application fails, check nsswitch
cat /etc/nsswitch.conf | grep hosts
# Should include: files dns
# Check if /etc/hosts has overrides
grep example.com /etc/hosts
# Test with specific resolver
dig @8.8.8.8 example.com +short
# Check systemd-resolved (Linux)
resolvectl status
resolvectl query example.comDNS failures are often misconfigured /etc/resolv.conf or a DNS server that is down. resolvectl query shows the full resolution chain including which resolver was used and whether the answer came from cache.
Connection Tracking
# View connection tracking table
sudo conntrack -L
# Count tracked connections
sudo conntrack -C
# Check tracking table size limit
cat /proc/sys/net/netfilter/nf_conntrack_max
# Check for dropped connections due to full table
dmesg | grep conntrack
# "nf_conntrack: table full, dropping packet"
# Increase if needed
sudo sysctl -w net.netfilter.nf_conntrack_max=262144
# Reduce timeout for stale entries
sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600Connection tracking is used by NAT and stateful firewalls. When the tracking table is full, new connections are dropped. This causes intermittent connection failures that are hard to diagnose. Monitor conntrack -C and the max limit.
If dmesg shows 'table full, dropping packet', your conntrack table is exhausted. This happens with many short-lived connections (NAT, load balancers). Increase the max and reduce timeouts for stale entries. Or disable conntrack for high-throughput connections (ip rules).
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.