Four Ways to Average the Past
From a loop mean to scaled causal self-attention, following every tensor shape through the implementation.
Index / term
From a loop mean to scaled causal self-attention, following every tensor shape through the implementation.
A shifted batch, 4,096 simultaneous next-character bets, and the first numerical check that tells me whether the model is alive.