Raft, explained by building it
Reading the paper is one thing. Watching your own leader election flap because of a heartbeat bug is another.
Why build it
Raft was designed to be understandable — that's literally the paper's thesis. But "understandable" when reading and "understood" after implementing are different states of knowledge. I built Raft KV to close that gap, and these are the notes I wish I'd had at the start.
"In Search of an Understandable Consensus Algorithm" — Ongaro & Ousterhout, 2014. The paper is genuinely readable; go read it first.
Leader election, and the bug that taught me the most
Every node is a follower until it stops hearing heartbeats. Then it becomes a candidate, votes for itself, and asks the cluster for votes. Simple — until your timeouts are too close together and the cluster elects nobody, forever. Randomized election timeouts aren't an optimization; they're the whole trick.
// The fix: jitter the election timeout per node.
// 150–300ms spread means one node usually times out
// first and wins cleanly.
timeout := 150*time.Millisecond +
time.Duration(rand.Intn(150))*time.Millisecond
select {
case <-n.heartbeat:
// leader is alive, reset timer
case <-time.After(timeout):
n.becomeCandidate()
}My first implementation used a fixed timeout. Three nodes, three simultaneous candidates, split vote, retry, split vote again. The logs looked like an argument between toddlers. One line of rand.Intn fixed a bug I'd spent a night on.
Log replication is a conversation
The leader doesn't push entries blindly — every AppendEntries call carries the index and term of the entry right before the new ones. If the follower doesn't have that entry, it refuses, and the leader walks backwards until they agree. It's a negotiation protocol disguised as replication: find the last point where two histories agree, then overwrite everything after it.
The subtle part: only commit your own term
Here's the bug you won't hit in a demo but will hit in production. A freshly elected leader must not treat an old entry as committed just because it's replicated on a majority — that reasoning is exactly the hole Raft's safety proof closes. A leader may only commit entries from its current term, and to carry the older ones across the line it appends a no-op entry the instant it wins. Leave that out and a value that looked committed can quietly disappear after a leader change. It's a handful of lines guarding the one invariant the whole system rests on, and it took me writing a test that kills the leader at exactly the wrong moment to believe I needed it.
Reads are the hard part, not writes
Writes are the easy case — they go through the log, so the log's rules protect them. Reads are where linearizability quietly breaks: a node that still believes it's leader after a partition will happily hand you a stale value. My reads use the ReadIndex protocol — before answering, the leader confirms it still commands a majority; if it can't, it returns NOT_LEADER instead of lying. A read that refuses to answer is more correct than a fast wrong one.
Measure it, don't trust it
Claiming Raft “works” is cheap. So I wrote a harness that boots a real three-node cluster, kills the leader mid-run, and times how long until the next write commits: p50 around 244 ms — one election timeout plus a round trip, exactly what the paper predicts. It also surfaced the ugly parts. Writes bottom out at a ~50 ms latency floor because replication is heartbeat-driven, and throughput falls off a cliff past eight concurrent clients thanks to a single global lock and a synchronous fsync per entry. Those aren't failures to bury — they're the next things to fix, and now I can point at the number instead of guessing.
What transfers to real work
Every consensus-backed system I touch now — etcd under Kubernetes, DynamoDB's replication, Kafka's controller quorum — maps onto machinery I've debugged myself. That's the return on building from scratch: production incidents become recognition, not archaeology.