Production systems

Architecture reports, incidents, and migrations from teams running inference at scale.

In-House LLM Serving: From Prototype to Production: Teaches you what a serving platform has to own beyond the engine: model versioning, cold starts, CPU-side bottlenecks, structured generation, scheduler failures. After it you can list what a vLLM prototype is still missing before it is production.

How Superhuman and Databricks built a 200K QPS inference platform: Shows you how to sequence design decisions when the QPS target is fixed before anything else: SLOs first, FP8 to fit the budget, then routing, autoscaling, and quality gates. After it you expect the bottleneck to be CPU RPC overhead, not the GPU.

How OpenAI Delivers Low-Latency Voice AI at Scale: Makes the network path part of the serving design: transport, session state, edge routing, and failure isolation are inference decisions here. After it you can design a realtime voice endpoint from the socket inward.