Tiled Attention & Online Softmax

The last lesson closed on a claim it did not prove: the traffic it had just counted is optional. Proving it takes one observation about softmax. The function subtracts the largest score in a row before exponentiating, a largest score is a fact about the whole row, and knowing a fact about the whole row means having already seen all…

In this lesson

• Why the row has to exist
• A running maximum and a running normalizer
• Watching the correction fire
• From a row to a tile
• Running it

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.