Learning Notes
Apr 11, 2026 · 6 min read
Learning notes: the log at the heart of every system
Kafka, Raft, write-ahead logs, event sourcing — the same data structure keeps showing up.
The pattern
These are my notes on Jay Kreps' essay "The Log: What every software engineer should know about real-time data's unifying abstraction". The core claim: an append-only, totally-ordered sequence of records is the primitive underneath databases, replication, messaging, and stream processing. After building Flume's storage engine, I believe it.
Where it shows up
Databases: the write-ahead log is the source of truth; the tables are just a cache of it.
Raft: consensus is literally agreement on the contents of a replicated log — nothing more.
Kafka: the log is the product. Consumers are just offsets into it.
Takeaway
When a system confuses you, look for its log. Once I started asking "what's the ordered history here, and who replays it?", architectures that looked complicated — CDC pipelines, event sourcing, read replicas — collapsed into one idea applied at different scales.
kafka
storage
papers