Build a Tiny Autoregressive Generation Loop
Rebuild model.generate() as a dozen-line autoregressive loop, watch greedy decoding pick each next token, and find the flaw the KV cache exists to fix.
Rebuild model.generate() as a dozen-line autoregressive loop, watch greedy decoding pick each next token, and find the flaw the KV cache exists to fix.