The Roofline model for AI inference

Peak FLOP per second is a ceiling, not a forecast. An inference kernel can leave most arithmetic units idle because the processor is waiting for weights, activations, or cache data to cross a memory boundary.

The Roofline model turns that tension into one picture. It compares the work performed per byte moved with a compute ceiling and a bandwidth ceiling, then takes the smaller attainable bound.

The picture is valuable precisely because it is incomplete. It classifies a bottleneck and suggests which direction an optimization must move; it does not predict latency for every request or replace measurement.

Two ceilings compete for every kernel

Let P peak be peak arithmetic throughput in FLOP per second. Let BW be sustainable bandwidth in bytes per second across the memory level being studied. Let I be operational intensity in FLOP per byte transferred.

The bound is P attainable equals the minimum of P peak and BW times I. The horizontal roof is compute capacity. The sloped roof says a kernel cannot consume data faster than bandwidth supplies bytes times the work extracted from each byte.

Williams, Waterman, and Patterson introduced this visual model in the original Roofline report. Their model uses operational intensity based on traffic after cache filtering, which makes the selected memory boundary part of the definition.

Units make the equation legible

Multiply bytes per second by FLOP per byte and bytes cancel, leaving FLOP per second. That is why BW times I can be compared with P peak.

If a workload performs one trillion operations but moves an unspecified number of bytes, the Roofline point is not defined. If a benchmark reports bandwidth without naming the memory tier and traffic accounting, its intensity can be misleading.

The terms arithmetic intensity and operational intensity are often used loosely. In this guide, I means measured or modeled operations per byte at the chosen memory boundary, matching the operational-intensity interpretation of the original model.

The ridge point separates the regimes

Set the two ceilings equal: P peak equals BW times I ridge. Solving gives I ridge equals P peak divided by BW.

To the left, BW times I is smaller, so the kernel is bandwidth-bound in this model. To the right, P peak is smaller, so additional intensity alone cannot raise the bound above the compute roof.

The ridge is hardware and boundary specific. A faster compute unit with unchanged bandwidth moves the ridge right. More bandwidth with unchanged compute moves it left.

Move one operating point across the roof

The workbench fixes a 100 TFLOP per second compute ceiling and 2 TB per second bandwidth, giving a ridge at 50 FLOP per byte. Slide intensity and watch the operating point follow the sloped roof until it meets the flat roof.

The topology is the lesson. On the sloped side, doubling sustainable bandwidth doubles the model's bound if intensity stays fixed. On the flat side, the same bandwidth change does not raise the compute ceiling.

Open the Roofline formula case for a typed symbol table. Then return to the plot and explain why the minimum operator, rather than addition, combines the limits.

Inference is a collection of points, not one point

Prefill processes many prompt tokens in parallel and can expose matrix multiplications with substantial reuse. Token-by-token decode often applies large weight matrices to a smaller active batch, which can reduce reuse per byte of weight traffic.

Attention, normalization, routing, sampling, and cache manipulation have different operation counts and memory traffic. Assigning one intensity to an entire model can hide the kernel that actually controls latency.

Batch size, sequence length, quantization, fusion, sparsity, cache reuse, and parallel placement can all move a point. A performance claim should identify workload phase and shape, not simply say that LLM inference is memory-bound.

Reducing traffic can move the point right

If an exact computation performs the same useful operations while transferring fewer bytes across the measured boundary, I increases. The point moves right and the bandwidth roof permits more throughput until another ceiling takes over.

FlashAttention is explicitly IO-aware exact attention. Its analysis focuses on reads and writes between GPU memory levels, showing why data movement can be an algorithm-design concern rather than a postscript to FLOP counts.

Fusion, tiling, and reuse may reduce traffic, but operation counts can also change. Recompute sometimes performs more arithmetic to avoid storing intermediates. Recalculate both F and M instead of assuming every memory optimization moves horizontally.

Use measured ceilings with a named boundary

A specification-sheet bandwidth number may exceed the sustained bandwidth available to a real access pattern. A useful Roofline study measures or justifies the ceiling under relevant conditions.

NVIDIA's Nsight Compute Roofline documentation describes hierarchical Roofline analysis and profiler metrics. It is a practical reminder that different cache and memory levels can produce different roofs.

Count bytes consistently. Reading weights once from HBM and reusing them from on-chip storage is different from reading them repeatedly. Mixing theoretical traffic in the numerator with measured traffic in the denominator produces a point with no clear interpretation.

Check invariants and extreme cases

As intensity approaches zero, the bandwidth ceiling approaches zero. As intensity grows without bound, the minimum approaches P peak and never exceeds it.

Below the ridge, doubling intensity doubles the simple bound. Above the ridge, doubling intensity leaves the bound unchanged. At the ridge, both ceilings are equal.

Those statements assume fixed sustainable bandwidth and peak compute. Thermal throttling, power limits, occupancy, instruction mix, synchronization, and communication can lower the real ceiling.

The roof is a bound, not a latency model

The basic model does not capture launch overhead, queueing, dependencies, tail latency, small-problem underutilization, network collectives, or quality-of-service interference. A point below the roof may have several explanations.

It also does not say whether an optimization improves user-visible tokens per second. A kernel may speed up while end-to-end time remains dominated by another kernel, scheduling, or communication.

Use the diagram to form a falsifiable hypothesis: which bytes or operations should change, across which boundary, for which shape. Then verify that hypothesis with profiler counters and end-to-end measurements.

Ask a mechanism-level question

A kernel is clearly left of the ridge and its operational intensity stays fixed. If sustainable bandwidth doubles while peak compute does not change, what happens to the simple Roofline bound.

It doubles until the point reaches the compute roof. The qualifier matters: once BW times I is no longer the smaller term, further bandwidth increases no longer raise this two-ceiling bound.

Frequently asked questions

Does memory-bound mean memory capacity is full

No. Capacity measures how many bytes fit. A bandwidth bound concerns how quickly bytes cross a boundary relative to the arithmetic work available per byte. A workload can fit comfortably and still be bandwidth-bound.

Does fewer FLOP always make inference faster

No. Reducing operations may also reduce reuse or leave memory traffic unchanged. On the bandwidth roof, removing arithmetic that was not the limiting resource can have little effect on throughput.

Can one model have several ridge points

The hardware has different ceilings for different precisions and memory levels. A hierarchical analysis can show several sloped roofs, while separate kernels and shapes occupy different points against them.

Continue from the roof to the bytes

Use the KV-cache memory guide to calculate a concrete byte payload, or build the wider reading habit in the Math Decoder curriculum.

The dependable routine is to name the memory boundary, calculate operations and bytes for one workload shape, locate the point relative to the ridge, and treat the result as a bound that still needs measurement.