Mamba: Linear-Time Sequence Modeling with…

Mamba asks how a recurrent model with a fixed-size state can decide what information deserves to survive. Read it to see how input-dependent state-space parameters create content-aware persistence and reset, and how a fused parallel scan makes the resulting time-varying recurrence practical on GPUs.

Reading focus: Why fixed linear state-space dynamics can be trained as a convolution but cannot change their update rule according to token content. How input-dependent Δ, B, and C let a bounded recurrent state persist useful history, admit selected inputs, and reset at boundaries. Why Mamba needs kernel fusion, parallel scan, SRAM-local state expansion, and recomputation after giving up the global-convolution shortcut.

COLM 2024. Gu and Dao. 55 min read, medium difficulty.