Series: System Design · Distributed Systems — Pillar 8 of 8

Systems Design

#	Post	What it covers
00	Distributed Systems: What Happens When Machines Disagree	Twenty concepts covering network partitions, consensus, clocks, distributed transactions, CDC, erasure coding, and observability. The final pillar.
01	Network Partitions: The Failure Mode You Can't Design Away	Network partitions are inevitable. Learn what happens when nodes can't communicate, how systems choose between availability and consistency, and what that means in practice.
02	Split-Brain: When Two Nodes Both Think They're the Leader	Split-brain occurs when two nodes both believe they're the primary. Learn how it happens, why it causes data corruption, and how STONITH and fencing prevent it.
03	Heartbeats: How Nodes Know Their Peers Are Alive	Heartbeats let nodes detect peer failures. Learn how timeouts, phi accrual failure detectors, and the tradeoff between false positives and detection speed work.
04	Leader Election: Agreeing on Who's in Charge	Leader election coordinates which node acts as primary. Learn the bully algorithm, Raft-based election, and why exactly-one-leader guarantees are hard to achieve.
05	Consensus Algorithms: Agreeing on a Value Across Failures	Consensus lets distributed nodes agree on a value despite failures. Learn what FLP impossibility means, what Paxos and Raft provide, and where consensus is used.
06	Quorum: How Many Nodes Must Agree?	Quorum determines how many nodes must agree for an operation to succeed. Learn how R + W > N ensures consistency in distributed databases like Cassandra and DynamoDB.
07	Paxos: The Algorithm That Started It All	Paxos is the foundational distributed consensus algorithm. Learn how its two phases work, why it's hard to implement, and what systems use it in production.
08	Raft: Consensus Made Understandable	Raft makes distributed consensus understandable. Learn how leader election, log replication, and safety work in the algorithm that powers etcd, CockroachDB, and TiKV.
09	Gossip Protocol: Decentralised Cluster Communication	Gossip protocols propagate information across a cluster without a central coordinator. Learn how epidemic spreading works and where it's used in production.
10	Logical Clocks: When Physical Time Isn't Enough	Physical clocks drift and can't establish event order in distributed systems. Logical clocks track causality instead. Learn why this matters and how it works.
11	Lamport Timestamps: Ordering Events Without a Global Clock	Lamport timestamps assign logical counters to events to establish causal order in distributed systems. Learn how they work and what they can and can't tell you.
12	Vector Clocks: Knowing When Events Are Truly Concurrent	Vector clocks detect causality and concurrency in distributed systems. Learn how they work, how Dynamo uses them for conflict detection, and their limitations.
13	Distributed Transactions: When One Machine Isn't Enough	Distributed transactions are hard. Learn why cross-service atomicity is expensive, when to use it, and when eventual consistency is the right alternative.
14	Two-Phase Commit: Coordinating a Distributed Decision	2PC ensures distributed atomicity through prepare and commit phases. Learn how it works, the coordinator failure problem, and why it's rarely used in modern systems.
15	Three-Phase Commit: Solving 2PC's Blocking Problem	3PC adds a pre-commit phase to eliminate 2PC's blocking problem. Learn how it works, what assumptions it requires, and why it's rarely used in production.
16	Delivery Semantics: What Does "Delivered" Actually Mean? ← you are here	Message delivery guarantees define system reliability. Learn what at-most-once, at-least-once, and exactly-once mean, what they cost, and when each is appropriate.
17	Change Data Capture: Streaming Your Database in Real Time	CDC streams database changes in real time by reading the write-ahead log. Learn how Debezium works, what CDC enables, and when to use it.
18	Erasure Coding: Fault Tolerance Without Full Replication	Erasure coding stores data across nodes using math, not full replication. Learn how Reed-Solomon works, how S3 uses it, and when it beats 3x replication.
19	Merkle Trees: Efficiently Finding What's Different	Merkle trees efficiently detect which parts of a large dataset differ between nodes. Learn how Bitcoin, Cassandra, and Git use them for verification and anti-entropy.
20	Observability: Understanding Your System at Runtime	Logs, metrics, and distributed traces are how you understand a system at runtime. Learn what each pillar provides, the tools involved, and how they work together.
21	Distributed Systems: Wrap-Up	A recap of all 20 distributed systems concepts and the complete URL shortener architecture spanning all 8 pillars. The final post in the series.

---

Delivery Semantics: What Does "Delivered" Actually Mean?

The problem

Your URL shortener's click pipeline publishes events to Kafka. The analytics consumer processes each event and increments a counter. What guarantees does the system provide?

If the consumer crashes after processing an event but before committing its Kafka offset, Kafka will re-deliver the event on restart. The counter gets incremented twice for one click — double-counting.

If the consumer commits its offset before processing, and then crashes during processing, the event is marked consumed but was never processed — the counter was never incremented. The click is lost.

Both outcomes are wrong. The question is: which wrongness are you willing to accept, and what does it cost to avoid it?

---

The core idea

Delivery semantics describe what a messaging system guarantees about how many times a message is delivered to a consumer: at most once (possibly zero), at least once (possibly more than once), or exactly once (exactly one). Each guarantee reflects a different tradeoff between performance, reliability, and implementation complexity.

---

The analogy: postal delivery policies

At-most-once: the postal service attempts delivery once. If no one is home, the letter is discarded — no second attempt. You might not get the letter (under-delivery), but you'll never get the same letter twice.

At-least-once: the postal service attempts delivery and tries again if unconfirmed. You will eventually receive the letter — possibly multiple copies if the first delivery wasn't confirmed but was received.

Exactly-once: the postal service guarantees you receive the letter exactly once. This requires tracking every delivery, detecting duplicates, and deduplicating before handing you the envelope. Much more work for the postal service.

---

At-Most-Once

A message is delivered zero or one times. If something fails, the message may be lost — but will never be processed twice.

How: the producer sends a message and doesn't retry on failure. The consumer acknowledges before processing — if it crashes after acknowledging but before processing, the message is gone.

Producer → sends event → does not retry on failure
Consumer → acks message → processes event
         ↑
         Ack before processing: if crash here, message lost

Use cases: metrics, telemetry, non-critical logs. Losing an occasional event is acceptable; duplicate processing would corrupt aggregates.

Cost: possible message loss.

---

At-Least-Once

A message is delivered one or more times. The consumer will process every message, but may process some more than once.

How: the producer retries until the broker acknowledges. The consumer processes before acknowledging — if it crashes after processing but before acknowledging, the message is re-delivered and re-processed.

Producer → sends event
  → timeout → retry (message sent again)
  → broker acks → stop retrying

Consumer → processes event
         → acks message
         ↑
         If crash here, message is re-delivered and re-processed

The idempotency requirement: if a consumer might process the same message twice, it must be idempotent — processing the same message twice produces the same result as processing it once.

# Non-idempotent (wrong for at-least-once):
def process_click(event):
    db.execute("UPDATE links SET click_count = click_count + 1 WHERE id = ?", event.link_id)
    # If this runs twice: click_count is incremented twice

# Idempotent (correct):
def process_click(event):
    db.execute("INSERT INTO click_events (id, link_id, ...) VALUES (?, ?, ...)",
               event.id, event.link_id, ...)
    # If event.id already exists (UNIQUE constraint): ignore duplicate silently

Use cases: most event-driven workloads. The most common production choice.

Cost: consumers must be idempotent; some events are processed more than once (but with idempotency, the end result is correct).

---

Exactly-Once

A message is delivered exactly once — no loss, no duplicates. The hardest guarantee.

The problem: "exactly once" in the context of distributed systems requires coordinating the producer, broker, and consumer atomically. The consumer must atomically: process the event AND mark it as consumed — such that if either step fails, both are retried or rolled back together.

Kafka's exactly-once implementation:

Kafka 0.11+ provides exactly-once semantics within a Kafka-to-Kafka workflow:

1. Idempotent producer: each producer has a unique ID and sequence number per partition. Kafka deduplicates retried produces — the same sequence number from the same producer is committed only once.

2. Transactional API: a consumer reads from topic A, processes the event, and writes results to topic B — atomically, within a Kafka transaction. Either all three steps succeed (consume + process + produce) or none do.

producer.init_transactions()
try:
    producer.begin_transaction()
    for record in consumer.poll():
        result = process(record)
        producer.send("output-topic", result)
    producer.send_offsets_to_transaction(consumer.offsets, group_id)
    producer.commit_transaction()
except Exception:
    producer.abort_transaction()

Kafka exactly-once is only within Kafka. If the consumer writes to a database (not another Kafka topic), exactly-once requires the database write and Kafka offset commit to be in the same transaction — which requires 2PC across Kafka and the database. This is either unsupported or impractical.

The practical approach for Kafka → Database: use at-least-once delivery with idempotent writes to the database (deduplication by event ID). The result is semantically exactly-once at the application level, without Kafka's transactional complexity.

---

Choosing delivery semantics

Can you tolerate message loss?
  Yes → At-most-once (fastest, simplest)

Can your consumers be idempotent?
  Yes → At-least-once with idempotent consumers (most common, practical)
  No → Must work toward exactly-once or idempotent redesign

Are all your writes Kafka-to-Kafka?
  Yes → Kafka exactly-once transactions
  No → At-least-once + idempotent writes to database

---

Tradeoffs

At-most-once: lowest latency and overhead. Wrong for any workload where losing events matters.

At-least-once: the practical default. Requires idempotent consumers — which is a good design principle regardless. Doubles some processing; correctly designed systems are unaffected.

Exactly-once: the highest overhead and complexity. Truly necessary only when the consumer cannot be made idempotent and duplicate processing has visible effects. Often "exactly-once at the application level through idempotency" is a better approach.

---

The one thing to remember

At-least-once delivery with idempotent consumers is the practical choice for almost all event-driven systems. Accept that messages will occasionally be delivered more than once; design consumers to handle duplicates gracefully (by event ID deduplication, INSERT OR IGNORE, or natural idempotency). Exactly-once is achievable within Kafka-to-Kafka pipelines but not across systems — for cross-system exactly-once semantics, idempotency at the application level is almost always the simpler and more robust approach.

---

← Previous: Three-Phase Commit — the protocol designed to eliminate 2PC's blocking problem, and why it's rarely used in practice despite solving it.

→ Next: Change Data Capture — streaming database changes in real time by reading the write-ahead log.