Stage 3 · Build
Network Debugging in Production
Latency Debugging
Measuring latency, traceroute, MTU issues, and bufferbloat.
Measuring Latency
Latency has multiple components: DNS lookup time, TCP handshake time, TLS handshake time, time to first byte (TTFB), and content transfer time. Understanding which component is slow is the first step to fixing it.
# Full timing breakdown with curl
curl -w "DNS: %{time_namelookup}s\nTCP: %{time_connect}s\nTLS: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
-o /dev/null -s https://api.example.com
# Example output:
# DNS: 0.012s <- DNS resolution
# TCP: 0.025s <- TCP connect (13ms RTT to server)
# TLS: 0.089s <- TLS handshake (64ms for key exchange)
# TTFB: 0.150s <- Time to first byte (61ms server processing)
# Total: 0.152s <- Content download
# Measure with ping (ICMP RTT)
ping -c 10 8.8.8.8
# Measure with hping3 (TCP RTT)
sudo hping3 -S -p 443 -c 10 api.example.comcurl timing breaks down the total request time. DNS, TCP, and TLS are network costs. TTFB minus TLS is server processing time. Total minus TTFB is content transfer time. Focus optimization on the largest component.
TCP Connection Metrics
# View TCP connection details
ss -ti dst api.example.com
# Key metrics in the output:
# rto: retransmission timeout
# rtt: round-trip time (smoothed)
# cwnd: congestion window (bytes in flight)
# retrans: retransmission count
# bytes_sent: total bytes sent
# bytes_acked: total bytes acknowledged
# Monitor retransmissions in real time
ss -ti state established | grep retransThe rtt field gives you the smoothed round-trip time. cwnd shows the congestion window — if it is small, TCP is being conservative due to packet loss. retrans shows cumulative retransmissions — high numbers indicate packet loss.
Traceroute Analysis
# Standard traceroute
traceroute -n api.example.com
# Example output analysis:
# 1 10.0.0.1 1.2 ms <- Your gateway (fast)
# 2 10.0.0.254 2.1 ms <- ISP router
# 3 172.16.0.1 5.3 ms <- ISP backbone
# 4 * * * <- ICMP blocked (normal)
# 5 198.51.100.1 12.4 ms <- Peering point
# 6 93.184.216.34 15.2 ms <- Destination
# Key patterns:
# Sudden jump in latency between two hops = congestion
# Asterisks = ICMP blocked (not necessarily a problem)
# Inconsistent times = intermittent congestionA latency jump between two hops indicates where the delay occurs. If hop 3 is 5ms and hop 6 is 12ms, the delay is in the ISP backbone between those points. You cannot control that, but it tells you where the bottleneck is.
MTU and Fragmentation
# Check current MTU
ip link show eth0 | grep mtu
# Test path MTU (set DF bit, vary packet size)
# Standard Ethernet MTU is 1500 bytes
# With overhead: 1500 - 20 (IP) - 20 (TCP) = 1460 payload
# Test if 1500-byte packets pass
ping -M do -s 1472 -c 3 api.example.com
# 1472 + 28 (IP+ICMP headers) = 1500 total
# If that fails, try smaller
ping -M do -s 1400 -c 3 api.example.com
# Check for fragmentation issues
sudo tcpdump -i eth0 'ip[6:2] & 0x20 != 0 or (ip[6:1] & 0xf0 != 0)'
# Captures packets with Don't Fragment bit or fragment offsetIf a 1500-byte packet does not pass but a 1400-byte packet does, your path MTU is between 1400 and 1500. VPNs, tunnels, and PPPoE add overhead that reduces the effective MTU. Set your MTU to match the path.
Bufferbloat
Bufferbloat occurs when network equipment has oversized buffers. Packets queue up instead of being dropped, causing high latency under load. The buffer does not drop packets (which would signal congestion to TCP), so latency increases while throughput stays high.
# Test bufferbloat with speedtest
# Run: https://www.waveform.com/tools/bufferbloat
# Check queue discipline
tc qdisc show dev eth0
# Typical output with bufferbloat:
# qdisc fq_codel 0: root refcnt 2 limit 10240p
# If you see qdisc pfifo_fast or qdisc netem,
# you may have bufferbloat issues
# Fix with fq_codel or cake qdisc
sudo tc qdisc replace dev eth0 root fq_codelfq_codel and CAKE are queue disciplines that manage bufferbloat by actively managing queue sizes and dropping packets early. This signals congestion to TCP sooner, keeping latency low under load.
Latency at light load is often fine. The real test is latency under load. Run a continuous ping while saturating the link with iperf3. If latency spikes under load, you have bufferbloat or insufficient bandwidth.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.