Quantization

Shrink the model itself: store weights in 8-bit or 4-bit instead of 16-bit floats. Post-training quantization is nearly free to apply and surprisingly robust; the trade is a quality haircut that grows as bits shrink. The reason a 70B model runs on a single consumer GPU at all.