Stage 4 · Provision
Asynchronous Systems
Event Streaming with Kafka
Topics, partitions, consumer groups, offsets, and compaction in Kafka clusters.
Kafka Architecture
Kafka is a distributed event streaming platform built around an append-only log. Producers write events to topics. Topics are split into partitions distributed across brokers. Consumers read events at their own pace, tracking their position with offsets.
Broker 1 Broker 2 Broker 3
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Topic A │ │ Topic A │ │ Topic A │
│ Part 0 │ │ Part 1 │ │ Part 2 │
│ (leader) │ │ (leader) │ │ (leader) │
└──────────┘ └──────────┘ └──────────┘
│ │ │
└── replicas ────┼────────────────┘
│
ZooKeeper/KRaft: Cluster metadata and leader electionEach partition has a leader broker. Producers write to the leader. Followers replicate for fault tolerance. KRaft (Kafka 3.3+) replaces ZooKeeper for metadata management.
Topics and Partitions
A topic is a category of events. Partitions are the unit of parallelism — each partition is an ordered, immutable log. More partitions mean more parallelism but also more open file handles and replication overhead.
Topic: order-events (6 partitions, replication factor 3)
Partition 0: Broker 1 (leader), Broker 2, Broker 3
Partition 1: Broker 2 (leader), Broker 1, Broker 3
Partition 2: Broker 3 (leader), Broker 1, Broker 2
Partition 3: Broker 1 (leader), Broker 2, Broker 3
Partition 4: Broker 2 (leader), Broker 1, Broker 3
Partition 5: Broker 3 (leader), Broker 1, Broker 2
Producer sends order for user_123:
hash("user_123") % 6 = Partition 4
→ Write to Broker 2 (leader of Partition 4)The partition count is fixed at topic creation. Choose based on throughput requirements. A common guideline: target 10-20 MB/s per partition.
Consumer Groups
Consumer groups enable parallel consumption of a topic. Each consumer in a group reads from a different partition. The number of consumers cannot exceed the number of partitions. If a consumer fails, its partitions are reassigned to remaining consumers.
Topic: orders (6 partitions)
Consumer Group: order-processor
Before rebalance:
Consumer 1: Partitions 0, 1
Consumer 2: Partitions 2, 3
Consumer 3: Partitions 4, 5
Consumer 3 crashes → Rebalance:
Consumer 1: Partitions 0, 1, 4
Consumer 2: Partitions 2, 3, 5
Consumer 3 recovers → Rebalance:
Consumer 1: Partitions 0, 1
Consumer 2: Partitions 2, 3
Consumer 3: Partitions 4, 5Rebalancing redistributes partitions when consumers join or leave. During rebalance, consumption pauses briefly. Minimize rebalancing frequency for low latency.
Offset Management
Offsets track how far each consumer group has processed. When a consumer processes a batch of messages, it commits the offset. On restart, the consumer resumes from the last committed offset. Uncommitted messages are reprocessed.
from kafka import KafkaConsumer
consumer = KafkaConsumer(
'orders',
group_id='order-processor',
auto_offset_reset='earliest', # Start from beginning if no offset
enable_auto_commit=False, # Manual commit for reliability
max_poll_records=100
)
for message in consumer:
try:
process_order(message.value)
# Commit offset only after successful processing
consumer.commit()
except Exception as e:
# Do NOT commit — message will be reprocessed
log.error(f"Failed: {e}")Manual offset commit ensures at-least-once processing. Only commit after successful processing. Auto-commit risks losing messages on crash.
Log Compaction
Log compaction retains only the latest value for each key. This is useful for changelog topics where you only need the current state. Compaction runs in the background and removes old records with the same key.
Compacted topics store the latest state for each key. A new consumer can read the entire compacted topic to rebuild its state. This is essential for event sourcing and CQRS patterns.
Kafka Operations
- Monitor consumer lag — Alert when lag exceeds threshold.
- Partition reassignment — Move partitions for load balancing.
- Topic creation — Pre-provision partitions for expected growth.
- Retention policy — Set time-based or size-based retention.
- MirrorMaker 2 — Replicate across Kafka clusters for DR.
Consumer lag is the most important Kafka metric. High lag means messages are delayed. Set up alerts for lag > 1000 messages or lag > 5 minutes. This catches processing bottlenecks before they become outages.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.