Sparse attention

Full attention costs O(n²) in sequence length, so long-context models attend selectively: sliding windows, dilated patterns, or learned sparsity. What survives at 128k+ contexts, and the long-range-reference problem sparse patterns must not break (the pronoun that points 1000 words back).