Interview questions

Questions organized by technical track, difficulty, and source evidence.

Implement a PyTorch autoregressive inference loop, then explain where and how you would add KV-cache reuse. The strong answer separates prefill from decode, appends K/V per step instead of recomputing, and can state the cache tensor shapes.

Implement batched multi-head self-attention with a causal mask, without using a prebuilt attention module. Shapes end to end, (B, H, T, d_k), scaling by the root of d_k, masking before softmax, and the final output projection.

Debug a pre-norm transformer block containing an incorrect residual or normalization path. Know the dataflow cold: x + Attn(LN(x)), then x + MLP(LN(x)). Planted bugs usually move the LayerNorm or drop a residual.