Kafka: a Distributed Messaging System for Log…

Kafka turns a queue into a retained distributed log. Append-only partitions hold ordered records while independent consumers pull from their own offsets, replay after failure, and advance at different speeds.

Reading focus: How a topic becomes ordered append-only partitions whose records are addressed by logical offsets rather than deleted after one consumer sees them. Why consumers pull at their own pace and keep their own checkpoints, making independent replay possible while exposing duplicate-processing and retention tradeoffs. How batching, sequential I/O, filesystem page cache, sendfile, and partition ownership produced the paper's 2011 throughput, and why its no-acknowledgement and no-replication limits matter.

NetDB 2011. Kreps, Narkhede, and Rao. 35 min read, easy difficulty.