How AI memory works: five systems, not one
“Memory” is one of the least precise words in AI.
It can mean knowledge stored in model weights, tokens inside the current context, a KV cache on a GPU, a recurrent state inside the architecture, or records retrieved from an external store.
Those systems retain different things, for different lengths of time, with different failure modes. Treating them as one feature makes both model behavior and infrastructure harder to reason about.
The cleanest mental model has five layers.
1. Model weights are learned memory
Training changes the model’s parameters. Patterns that help predict the next token become distributed across billions of weights.
This is durable, parametric memory. It survives between requests because the weights stay fixed at inference time.
It is not a notebook of exact training examples. A fact is not normally stored in one address that can be read, corrected, or deleted in isolation.
The original RAG paper uses a useful distinction: model parameters are parametric memory, while a retrieved document index is non-parametric memory.
That distinction explains why weights are broad but awkward to update. Changing one fact usually means more training, fine-tuning, or an external system that supplies newer evidence.
Weights answer, “What patterns did training leave in the model?”
They do not answer, “What did this user say ten minutes ago?”
2. The context window is the working set
At inference time, the model receives a sequence of tokens. The context window limits how many input and generated tokens can participate in the current computation.
This is working memory in a product sense. Instructions, examples, tool results, and conversation history are visible only when the application includes them in the request.
A longer context window increases capacity, but capacity is not the same as reliable recall.
The model still has to find the relevant token among everything else. Position, repetition, formatting, and competing evidence can change what it uses.
Context also has a direct systems cost. Full attention compares tokens with earlier tokens, while the serving stack keeps state for those positions during generation.
The context answers, “What information is available to this request right now?”
When the request ends, those tokens are not automatically a persistent user memory.
3. The KV cache remembers computation
Autoregressive generation emits one token at a time. Recomputing attention keys and values for the entire prefix before every new token would waste most of the work.
The KV cache stores those tensors for reuse. Each decoding step computes the new token’s key and value, appends them, and attends over the cached prefix.
This is a performance memory. It remembers intermediate computation, not a semantic fact chosen for future conversations.
The distinction matters because KV cache is often described as if it were the model’s long-term memory.
It usually lives only for an active request or a reusable prefix. Evicting it may make the next request slower, but it does not erase knowledge from the model weights.
The PagedAttention paper shows why cache layout affects serving throughput. Dynamic request lengths make contiguous allocation wasteful and constrain batching.
Fanout’s KV-cache Lab exposes the core formula. Layers, KV heads, head dimension, tokens, batch, and precision all multiply.
The KV cache answers, “Which attention computations can this serving process avoid repeating?”
4. Recurrent attention keeps a fixed-size state
Full attention retains token-level keys and values. Linear attention takes another route: it summarizes the sequence into a recurrent state that can be updated as tokens arrive.
Transformers are RNNs showed how changing the attention formulation permits a linear-time recurrent implementation.
The gain comes with a constraint. A fixed-size state has to compress an arbitrarily long history.
Simple additive updates can blend two values written under similar keys. The memory needs a rule for correcting old associations and making room for new ones.
DeltaNet uses a delta update. It reads the current association and writes the difference between that value and the new target.
Gated DeltaNet adds learned decay. It can weaken old state before writing a targeted correction.
Kimi Linear makes the decay channel-specific through Kimi Delta Attention, or KDA.
Some channels can retain stable information while others forget quickly enough to accept an update.
Kimi K3 combines three KDA layers with one exact Gated MLA layer in each block. The hybrid carries compact state most of the time and periodically restores global token-to-token attention.
This state answers, “What compressed history is the architecture carrying forward?”
It is still request state, not a database of past sessions.
5. Retrieval supplies persistent external memory
An application can store documents, messages, facts, summaries, or events outside the model. Before a response, it searches that store and inserts selected records into the context.
This is the usual foundation for persistent product memory.
The original RAG design combines a neural retriever with a generator. The model weights provide parametric knowledge, while a dense index provides explicit, updateable evidence.
A user-memory system often adds two steps around retrieval:
- Decide what should be captured
- Store it with useful metadata
- Retrieve candidates for the current request
- Rank, filter, and place them in context
- Let the model answer from the assembled working set
Storage alone is not memory quality. A system that saves everything can retrieve stale, private, duplicated, or irrelevant records.
Retrieval also does not guarantee use. The selected item still has to survive the context assembly and the model’s attention.
External memory answers, “What durable records should be brought back for this request?”
Product memory is an orchestration layer
When an assistant appears to remember a preference across sessions, the model has not usually rewritten its own weights.
The product has captured a record, stored it, retrieved it later, and placed it into a new context.
This separation is useful because each step can be inspected.
You can ask what was saved, why it was selected, whether it is still valid, and how the user can correct or delete it.
It also creates responsibilities. Persistent memory needs access control, retention limits, provenance, correction paths, and a rule for sensitive data.
The model may be the most visible component, but memory quality often depends more on the surrounding data system.
The same failure can come from different layers
Suppose an assistant gives an old meeting room after the room changed.
The cause could be stale parametric knowledge, an old message still in context, a retrieved record that was never updated, or a recurrent state that blended the old and new values.
Those failures need different fixes.
More context will not repair a retrieval index that keeps ranking the wrong record.
A vector database will not repair a KV cache allocation problem.
Longer retention in a recurrent state may preserve both conflicting values rather than selecting the newer one.
Good debugging starts by naming which memory layer produced the evidence.
A practical memory stack
Use model weights for broad learned capability.
Use the context window for the exact instructions and evidence needed now.
Use the KV cache to reuse attention computation during serving.
Use recurrent state when the architecture needs bounded sequence memory.
Use external retrieval for durable, inspectable, updateable information.
Most real systems combine several layers. The design question is not whether the AI has memory. It is which memory owns each kind of information.
Fanout’s How AI remembers Lab makes the architectural progression visible with six small systems and one changing fact.
Continue with the RAG Daily note for external memory, or the Kimi K3 architecture guide for hybrid recurrent and exact attention.
Once the layers are separate, “the model forgot” stops being an explanation. It becomes the start of a useful systems question.