Queues, Streams & Async Workflows

Why async messaging

When service A calls service B synchronously, A's reliability is bounded by B's. If B is slow, A is slow. If B is down, A is down. Async messaging breaks this coupling: A drops a message into a queue and moves on; B picks it up when it can. This buys reliability, smoothing of traffic spikes, and the ability to retry on failure.

Message queues vs event streams vs pub/sub

• MESSAGE QUEUE (RabbitMQ, AWS SQS) — each message is consumed by exactly one consumer. Great for jobs and tasks. Often FIFO with at-least-once delivery.
• EVENT STREAM (Apache Kafka, AWS Kinesis) — append-only log. Many consumers can read independently, each at their own offset. Replayable — perfect for analytics, audit logs, and data pipelines.
• PUB/SUB (Redis Pub/Sub, Google Pub/Sub) — fire-and-forget broadcast to many subscribers. Often no persistence — slow subscribers miss messages.

Delivery semantics

• At-most-once — fast, may lose messages. Use only when loss is acceptable (metrics, telemetry).
• At-least-once — guaranteed delivery, but the message may be delivered MULTIPLE times. The standard for most queues. Consumers must be idempotent.
• Exactly-once — what everyone wants. Hard to achieve end-to-end; usually built on at-least-once + idempotency.

Idempotency

An idempotent operation produces the same effect whether run once or many times. \"Set user.email to X\" is idempotent. \"Add $50 to user.balance\" is NOT — running twice doubles the deposit.

To make a non-idempotent operation safe under at-least-once delivery, attach a unique idempotency key to each message and record processed keys. If you see a duplicate, skip it.

Common patterns

Task queue

Web request comes in → enqueue a job (e.g. \"send welcome email\") → return 202 Accepted → background worker picks up the job. The user experience is fast; the slow work happens off the request path.

Outbox pattern

When you must update a database AND publish a message, you can't do both atomically. The outbox pattern: write the message to an OUTBOX table in the same DB transaction as the data change, then have a separate poller publish from the outbox to the queue. Either both happen or neither.

Event-driven architecture

Services emit events about state changes (\"OrderCreated\", \"PaymentProcessed\") to a stream. Other services subscribe and react. No direct coupling — you can add new consumers without touching the producer. The downside: tracing what happened across services becomes a forensic exercise; invest in correlation IDs and observability.