19. Message Queues & Event Streaming

Async messaging decouples producers from consumers. Done well, it absorbs spikes, enables independent scaling, and provides reliability through retry. Done poorly, it creates inscrutable distributed bugs.

19.1 Why use a queue

• Decoupling: producer doesn't wait for consumer.
• Buffering / smoothing: absorb load spikes.
• Retry / resilience: failed messages can be retried later.
• Fan-out: one message to many consumers.
• Background work: long-running tasks off the request path.
• Audit / replay: persistent log lets you re-process history.

If your work could be sync and < 100ms, don't add a queue. Queues add latency, complexity, debugging pain.

19.2 Two paradigms

Message queue (point-to-point)

One message → one consumer. Worker pool drains the queue.

Examples: RabbitMQ (classic queue), AWS SQS, Redis Streams (consumer groups), BullMQ.

Pub/sub (broadcast)

One message → all subscribers. Each consumer gets its own copy.

Examples: Redis pub/sub (no persistence), NATS, Google Pub/Sub, Kafka (topic with multiple consumer groups).

Log (event streaming)

Append-only durable log. Many consumers read independently at their own pace. Replay is first-class.

Examples: Kafka, Pulsar, Amazon Kinesis, Redpanda, Google Cloud Pub/Sub Lite.

The log paradigm has won most large-scale architectures.

19.3 Delivery semantics

Three flavors. There is no true "exactly once delivery." There is "at-least-once + idempotent processing."

At-most-once

Send and forget. No retries. Messages may be lost. Used for: metrics, "fire and forget" telemetry.

At-least-once

Retries until ack. Duplicates possible. Default for most queues. Requires idempotent consumers.

Exactly-once (semantic)

At-least-once + idempotent dedup → effectively once. Kafka offers "transactional producer + idempotent consumer offsets" for exactly-once processing within a Kafka pipeline.

Even Kafka exactly-once doesn't extend to side effects (sending an email, updating a non-Kafka DB). For those, you need outbox + idempotency keys.

19.4 Kafka deep dive

Kafka is the de-facto standard for event streaming. Worth understanding deeply.

Concepts

• Topic: named log.
• Partition: each topic is split into N partitions; each is an ordered log.
• Producer: appends messages to a partition (chooses by key hash or round robin).
• Consumer: reads from partitions.
• Consumer group: multiple consumers share work; each partition consumed by exactly one consumer in a group.
• Offset: position in a partition; consumer commits offsets.
• Replication factor: N copies of each partition; one is leader, others are in-sync replicas (ISR).

Ordering

• Within a partition: total order.
• Across partitions: no order.
• Practical implication: choose your partition key wisely; messages with the same key are ordered.

Throughput

Single broker can handle ~1 MB/s × 10K partitions = 10 GB/s in some benchmarks. Real systems run 100s of brokers, millions of msgs/sec.

Retention

Configurable per topic: by time (retention.ms=604800000 = 7 days) or by size. Compaction retains the latest message per key (useful for "current state" topics).

Why it's fast

• Sequential disk writes (LSM-style).
• Zero-copy (sendfile) for reads.
• Page cache amplification.
• Batched producer + consumer fetch.

Replication

• Leader + N followers per partition.
• acks=0 (fire and forget), acks=1 (leader ack), acks=all (all ISRs ack — durable).
• min.insync.replicas enforces minimum surviving replicas; refuses writes if violated.

Producer guarantees

• Idempotent producer (enable.idempotence=true): broker dedups based on producer ID + sequence number. No duplicates from retries.
• Transactional producer: atomic write across multiple partitions. Required for exactly-once.

Consumer offsets

Stored in __consumer_offsets (a topic). Consumers commit as they go (auto or manual). On crash, restart from last committed offset.

Kafka without ZooKeeper

KIP-500 / KRaft mode (production since Kafka 3.5): Kafka uses an internal Raft for metadata, removing ZooKeeper. Simpler ops; default for new clusters.

19.5 RabbitMQ

Classic AMQP broker. Different model than Kafka.

• Exchange + binding + queue: producer sends to exchange; bindings route to queues; consumers read from queues.
• Exchange types: direct, topic (pattern routing), fanout, headers.
• Acks: per-message; broker holds until acked.
• Persistence: optional per message.
• Priority queues: yes (Kafka doesn't easily).
• Delayed messages: via plugins.
• Lower throughput than Kafka: ~100K msgs/sec range.

When to pick: complex routing, per-message ack, RPC patterns, priorities, delayed scheduling, low-to-mid throughput.

19.6 SQS, Pub/Sub, Pulsar

AWS SQS

Simple managed queue.

• Standard (at-least-once, no order, ~unlimited throughput) or FIFO (in-order, exactly-once-ish, ~3K msgs/sec/group).
• Visibility timeout: in-flight message hidden until consumer ack or timeout.
• DLQ (dead-letter queue) for poison messages.
• Long polling to reduce empty receives.

Google Pub/Sub

Managed global pub/sub. Push and pull modes. Good Pub/Sub++ feature: ordering keys.

Apache Pulsar

Open-source competitor to Kafka. Tiered storage (hot in BookKeeper, cold in S3). Cleaner multi-tenancy. Less mature ecosystem.

19.7 Common patterns

Work queue

Producers enqueue jobs; worker pool consumes. Auto-scale workers by queue depth.

Pub/sub fan-out

Event published once; many services react. Order routing: fulfillment, billing, notification.

Retry with backoff + DLQ

On failure: requeue with delay (exponential backoff + jitter). After N retries, send to dead-letter queue for manual inspection.

Saga choreography

Each service emits events; others react. (See chapter 16.)

Outbox

DB writes + event publish in one local transaction via an outbox table. CDC streams to broker. The most important pattern in event-driven systems.

Event sourcing

Store all state changes as events; rebuild state by replaying. (See chapter 22.)

CQRS

Separate write model (commands) from read model (queries). Often paired with event sourcing.

19.8 Backpressure and flow control

When consumers can't keep up:

• Buffering: broker absorbs; risk OOM if unbounded.
• Backpressure to producer: broker pushes back; producer slows or buffers locally.
• Drop / sampling: shed low-priority work.

Kafka: producer pushes; broker has effectively unlimited buffering (disk-backed). If consumers fall behind, lag grows; eventually retention policy drops old data they haven't read. Watch consumer lag.

19.9 Poison messages

A bad message that crashes every consumer. Two strategies:

1. Move to DLQ after N retries (most queues do this automatically with config).
2. Skip and log (Kafka requires you to commit a later offset; you do so explicitly after logging the bad message).

DLQ is essential. Without it, one bad message hangs the whole pipeline forever.

19.10 Monitoring queues

Critical SLIs:

• Queue depth / lag (consumer offset vs latest offset). Alert > N.
• Message age (oldest message in queue). Tells you SLA breach.
• Producer rate vs consumer rate. Mismatch = trouble brewing.
• Error rate / DLQ rate. Spike = poison or bug.
• Per-partition skew. Uneven keys = hot partition.

19.11 Choosing a queue

Need	Pick
Massive event log, replay	Kafka
Complex routing, low-volume	RabbitMQ
Fully managed, AWS	SQS
Fully managed, GCP	Pub/Sub
In-process, Redis already there	Redis Streams / BullMQ
Stream processing, multi-tenant	Pulsar

For Google interviews: know Kafka cold. It's the lingua franca of event-driven systems.

19.12 Anti-patterns

• Queue as a database: messages aren't a substitute for storage. Persist data in a DB; messages signal change.
• Sync RPC with a queue in front: defeats the point.
• No DLQ: poison messages destroy systems silently.
• No idempotency: duplicates cause double charges.
• Single partition for ordering: can't scale; one slow consumer blocks everything.
• Unbounded consumer lag with no alerts: you'll find out from users.

19.13 Latency math

Producer-to-consumer end-to-end latency:

• Producer batch wait (1-100 ms typical)
• Network to broker (< 1 ms in DC)
• Broker disk fsync (1-10 ms)
• Replication ack (1-10 ms in DC)
• Consumer poll interval (1-100 ms)
• Total: ~10-200 ms typical for Kafka

Real-time? Yes (streaming). Sync API? No.

Key takeaways

• Async messaging decouples and absorbs spikes; cost is latency and complexity.
• Kafka = log paradigm, dominant choice for high-throughput event pipelines.
• Delivery: pick at-least-once + idempotent consumers. "Exactly once" is mostly marketing.
• Outbox pattern is essential for transactional producers.
• DLQ + monitoring queue depth are non-negotiable.