---
title: "Kafka: a Distributed Messaging System for Log Processing"
description: "Kafka turns a queue into a retained distributed log. Append-only partitions hold ordered records while independent consumers pull from their own offsets, replay after failure, and advance at different speeds."
canonical_url: "https://fanout.sh/daily/2026-08-28-kafka-distributed-log"
md_url: "https://fanout.sh/daily/2026-08-28-kafka-distributed-log.md"
last_updated: "2026-08-28"
access: "public"
---

# Kafka: a Distributed Messaging System for Log Processing

Kafka turns a queue into a retained distributed log. Append-only partitions hold ordered records while independent consumers pull from their own offsets, replay after failure, and advance at different speeds.

## Paper details

- Authors: Kreps, Narkhede, and Rao

- Venue: NetDB 2011

- Track: Systems

- Difficulty: Easy

- Reading time: 35 min

- Original paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/09/Kafka.pdf

## What you will learn

- How a topic becomes ordered append-only partitions whose records are addressed by logical offsets rather than deleted after one consumer sees them.

- Why consumers pull at their own pace and keep their own checkpoints, making independent replay possible while exposing duplicate-processing and retention tradeoffs.

- How batching, sequential I/O, filesystem page cache, sendfile, and partition ownership produced the paper's 2011 throughput, and why its no-acknowledgement and no-replication limits matter.

## Continue exploring

[Explore Kafka design on Fanout](https://fanout.sh/system/archive/source-kafka-design-docs): Continue into Fanout's Kafka source summary and connect partitions, offsets, consumer groups, retention, replay, replication, and stream-processing semantics to the modern system.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md