Math for Memory, Throughput & Speedups
Napkin arithmetic for LLM inference: count the bytes a model needs in GPU memory, turn bandwidth into a speed ceiling, and check what a speedup is worth.
Napkin arithmetic for LLM inference: count the bytes a model needs in GPU memory, turn bandwidth into a speed ceiling, and check what a speedup is worth.