Trace the shapes through a transformer
Trace four tokens through a small decoder. Check matrix shapes and find mistakes involving position vectors, token mixing, and the two softmax operations.
Trace four tokens through a small decoder. Check matrix shapes and find mistakes involving position vectors, token mixing, and the two softmax operations.