case study · go · distributed systems
Raft KV — consensus, from the paper up
A distributed key-value store where multiple servers agree on data even when nodes crash — every line of the consensus layer written by hand in Go.
The problem
How do multiple servers agree on the same data when any of them can crash, restart, or get partitioned? Consensus is the foundation under etcd, Consul, and CockroachDB — and reading the Raft paper is not the same as making it work. I wanted the version of understanding that only comes from debugging your own leader elections at 2am.
What I built
AppendEntries with prevLogIndex/prevLogTerm consistency checks, conflict truncation, and per-follower nextIndex back-off
Safety — commits only entries from the current term, and plants a no-op barrier on election so prior-term writes commit correctly instead of silently disappearing
Persistence and recovery — a write-ahead log plus atomically written snapshots (temp file + rename), with an InstallSnapshot RPC to catch up a follower that has fallen behind the log
Linearizable reads via the ReadIndex protocol — the leader re-confirms it still holds a majority before serving, so a partitioned stale leader returns NOT_LEADER rather than a stale value
Dynamic single-server membership — add and remove nodes on a live cluster, replayed correctly across restarts
Engineering decisions
The hardest part wasn't the happy path — it was the edges: a stale leader that doesn't know it's stale, a follower whose log diverged during a partition, election storms when timeouts are too tight. Key choices:
hashicorp/raft, no etcd/raft. The only third-party import is OpenTelemetry, and only at the server edge for optional tracing. Using someone else's Raft would have defeated the point.
Hand-rolled RPC transport. RequestVote, AppendEntries, and InstallSnapshot ride as JSON over plain TCP, dispatched by message type — so the wire format stays legible instead of vanishing into a framework.
State machine as a pure function. Applying a log entry is deterministic and side-effect free, which makes replay, snapshots, and per-client dedup easy to reason about.
Heartbeat-driven replication. A deliberate simplicity choice with a known cost — writes wait up to one 50 ms heartbeat — so I benchmarked exactly what that trade-off buys and what it costs.
Measured, not asserted
Rather than claim it “works,” I built a benchmark harness that boots a real 3-node cluster as child processes, then kills leaders and partitions nodes to measure recovery from the client's point of view:
Single-machine loopback numbers, self-reported from the harness in docs/benchmarks.md. The 50 ms latency floor and the throughput cliff past 8 clients are both direct, expected consequences of heartbeat-batched writes under a single lock — the benchmark exists to make those trade-offs visible, not to hide them.
Where it stands, honestly
The consensus core is real and tested — election, replication, current-term safety, WAL persistence and crash recovery, snapshots with InstallSnapshot, ReadIndex reads, and single-server membership all work end-to-end and are exercised by 84 passing tests plus the cluster-spinning benchmark harness. The limits are deliberate and documented in ADRs: single-server (not joint-consensus) membership, no pre-vote or leader leases, and no batching or pipelining of replication — each traded away in favor of a codebase you can actually read against the paper. Around that core sits a larger cloud-native ops layer (Docker, a Helm chart, GitOps, signed images); that scaffolding is built but is a portfolio milestone, not a hardened production datastore.
Distributed systems fail in ways sequential code never prepares you for — a leader that doesn't yet know it's been deposed, a log that diverged during a partition. Building Raft by hand gave me a working mental model for every consensus-backed system I touch now, and the discipline to measure recovery instead of assuming it.