---
title: "Production systems"
description: "Architecture reports, incidents, and migrations from teams running inference at scale."
canonical_url: "https://fanout.sh/inference-eng/production"
md_url: "https://fanout.sh/inference-eng/production.md"
access: "public"
---

# Production systems

Architecture reports, incidents, and migrations from teams running inference at scale.

## Public overview

In-House LLM Serving: From Prototype to Production: Teaches you what a serving platform has to own beyond the engine: model versioning, cold starts, CPU-side bottlenecks, structured generation, scheduler failures. After it you can list what a vLLM prototype is still missing before it is production.

How Superhuman and Databricks built a 200K QPS inference platform: Shows you how to sequence design decisions when the QPS target is fixed before anything else: SLOs first, FP8 to fit the budget, then routing, autoscaling, and quality gates. After it you expect the bottleneck to be CPU RPC overhead, not the GPU.

How OpenAI Delivers Low-Latency Voice AI at Scale: Makes the network path part of the serving design: transport, session state, edge routing, and failure isolation are inference decisions here. After it you can design a realtime voice endpoint from the socket inward.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md