LLM inference interview questions that matter

The weakest inference interviews reward vocabulary. A candidate says PagedAttention, continuous batching, tensor parallelism, and quantization, then nobody checks whether the pieces fit.

A useful interview changes one constraint at a time and asks the candidate to reason from measurements.

The questions below include an answer rubric. They are not scripts to memorize. A strong answer states assumptions, estimates scale, names tradeoffs, and proposes a measurement.

Questions about the request lifecycle

1. What happens between receiving a prompt and streaming the first token?

A strong answer covers tokenization, admission, queueing, batching, model execution, sampling, serialization, and network delivery.

It separates prefill from decode and names time to first token as a user-facing measure.

The candidate should ask whether tokenization is local, whether the model is warm, and whether the request waited for capacity.

2. Why can two requests with the same token count have different latency?

Prompt and output tokens use different execution patterns. Sequence shapes, cache state, batching peers, sampling, hardware state, and queue position also matter.

The answer should reject one universal "milliseconds per token" constant.

Read prefill vs decode for the phase distinction.

3. Which latency metrics would you expose?

Look for queue time, time to first token, inter-token latency, end-to-end latency, and cancellation or timeout rate.

Percentiles matter. Split by model, prompt length, output length, priority, region, and cache status.

The vLLM metrics reference is a useful example of production instrumentation.

Questions about memory and the KV cache

4. Estimate KV cache memory for this model

Give layers, key-value heads, head dimension, sequence length, batch size, and bytes per element.

A strong candidate writes the formula, checks units, and clarifies whether the batch value means active sequences or reserved capacity.

They should notice grouped-query attention and avoid using total attention heads when key-value heads differ.

Use Fanout's KV cache calculator to practice checking the result.

5. Why does PagedAttention improve serving capacity?

The answer should focus on allocation and fragmentation, not claim that attention math itself became cheaper.

PagedAttention stores logical cache blocks in non-contiguous physical blocks and maps between them.

A stronger answer discusses copy-on-write or sharing, block-size tradeoffs, metadata, and what happens under preemption.

6. The GPU reports free memory, but a new request is rejected. Why?

Possible causes include reservation policy, fragmentation, graph capture pools, workspace memory, safety margins, model replicas, and an admission estimate based on maximum growth.

The candidate should request allocator and cache-occupancy metrics instead of trusting one free-memory number.

7. How would you reduce memory without changing the model's weights?

Options include shorter context limits, lower concurrency, KV cache quantization, fewer key-value heads in a different architecture, prefix reuse, and moving some state.

Each option changes capacity, quality, latency, or product behavior. "Clear the cache" is not a serving policy.

Questions about scheduling and batching

8. What problem does continuous batching solve?

Static batches hold capacity until the group finishes. Continuous batching can admit new work at iteration boundaries as sequences complete.

The answer should connect variable output lengths to wasted slots and head-of-line blocking.

Orca is the useful primary reference.

9. Can higher throughput make latency worse?

Yes. Larger batches and deeper queues can increase device utilization while increasing queue time or inter-token latency.

A strong answer asks for the service objective and separates maximum throughput from goodput within that objective.

10. How would you prevent a long prompt from hurting chat traffic?

Possible designs include separate prefill and decode capacity, chunked prefill, priority queues, admission limits, and workload isolation.

DistServe is a useful reference for separating prefill and decode.

The candidate should discuss starvation and capacity waste rather than promise perfect isolation.

11. Design a scheduler for free and paid users

Look for explicit objectives, weights or reserved capacity, aging to prevent starvation, per-tenant limits, and overload behavior.

A good answer includes cancellation and abandoned requests. Serving output after the client has gone is expensive throughput.

Questions about GPU performance

12. Why is decode often memory-bound?

Each decode step performs a small amount of new-token work while reading model weights and a growing cache.

The candidate should explain arithmetic intensity and avoid claiming every kernel or model shape behaves identically.

Ask what profiler evidence would confirm the diagnosis.

13. What would you inspect before writing a custom CUDA kernel?

Look for a profiler trace, kernel time, launch count, memory throughput, occupancy, shapes, existing fused kernels, and framework overhead.

The CUDA Best Practices Guide gives a sound measurement vocabulary.

Writing a kernel before locating the bottleneck is an expensive way to optimize the wrong path.

14. Why might INT4 use less memory without reducing latency?

The format may require dequantization, use a slower kernel, create poor shapes, or move the bottleneck elsewhere.

A strong answer measures prefill and decode separately and checks quality, not only model load time.

15. What does kernel fusion buy?

Fusion can remove intermediate memory traffic and reduce launch overhead.

It may increase register pressure, reduce flexibility, or perform poorly for some shapes.

The candidate should name the operations being fused and propose a benchmark across realistic sequence and batch sizes.

Questions about parallelism

16. When would you use tensor parallelism?

Use it when the model or target latency needs multiple tightly connected GPUs.

The answer should discuss collective communication, topology, per-layer synchronization, and why fast links matter.

More GPUs can lower compute time and raise communication cost. The balance changes between prefill and decode.

17. Compare tensor and pipeline parallelism

Tensor parallelism splits operations inside layers. Pipeline parallelism splits groups of layers into stages.

Look for discussion of collectives, pipeline bubbles, microbatches, stage balance, latency, and fault domains.

A diagram of data movement is better than a list of names.

18. Why can two identical GPUs perform differently in production?

Thermal state, power limits, clocks, NUMA placement, PCIe topology, background work, firmware, driver versions, and error recovery can differ.

A strong candidate checks host and topology data beside GPU metrics.

Questions about speculative decoding

19. How can speculative decoding preserve output distribution?

A cheaper proposer suggests tokens. The target model verifies them in parallel and an acceptance rule preserves the target distribution.

Fast Inference from Transformers via Speculative Decoding reports exact sampling with faster generation in its experiments.

The candidate should not say the draft model's output is blindly accepted.

20. When can speculative decoding lose?

Low acceptance, expensive proposals, extra memory, high-QPS throughput pressure, poor model pairing, and incompatible features can remove the gain.

The right benchmark measures inter-token latency, throughput, acceptance length, memory, and quality across actual sampling settings.

The speculative decoding guide develops this answer.

A system design prompt for the final round

Design a service for two models, streaming responses, bursty traffic, a 500 ms time-to-first-token objective, and a hard GPU budget.

Ask the candidate to clarify traffic, prompt and output distributions, regions, availability, quality constraints, and what can queue.

Then change one fact. Double prompt lengths. Lose a GPU. Add a priority tenant. Require a new model to roll out without dropping streams.

A strong design covers:

• Admission, queues, priorities, and backpressure.
• Model placement, warm capacity, and deploy strategy.
• Cache allocation, sequence limits, and overload.
• Streaming, cancellation, retries, and idempotency.
• Metrics, traces, cost attribution, and capacity planning.
• Failure modes for workers, dependencies, and regions.

The Fanout inference engineering path provides the technical sequence behind these questions.

How to practice without rehearsing slogans

Choose one open model and one serving engine. Measure it on hardware you can access.

For every result, explain the workload, configuration, precision, model revision, hardware, and metric definition.

Change one variable at a time: prompt length, output length, concurrency, cache size, batch policy, quantization, or parallelism.

Write a short incident after each failure. What did the user observe? Which metric moved first? What guardrail should have contained it?

An interview answer becomes credible when it contains a model of the system and a way to test that model.

Fanout's latency lab and back-of-the-envelope lab are good places to practice estimates before opening a profiler.