Design: a Kafka-style log

The core idea

A publishing service needs search and analytics to receive the same post changes.

Search may be current while analytics is an hour behind.

In this lesson

  • What are we promising?
  • Partition the order we need
  • Follow one accepted publication
  • Replace a failed leader safely
  • Handle producer uncertainty
  • Read, apply, then save progress
  • Store batches, expire segments
  • Size the recovery work
  • Run the smaller recovery experiment
  • Check a failure, not just a happy path

Continue reading

Pro unlocks video lessons, visual walkthroughs, full notes, diagrams, case studies, and implementations across the System Design archive.