The Extra Code Is Not Another Brain
Flash Attention, mixed precision, DDP, shards, checkpointing, and HellaSwag without pretending I trained GPT-2.
Index / term
Transformers
Width Paid, Context Lied, Last Step Lost
Tiny Shakespeare ablations for context, depth, and width—and the caveats that matter more than the leaderboard.
A Live Wire Is Not a Language
The complete Tiny Shakespeare baseline as an instrument: diagnostics, curve, checkpoint, and one revealing broken specimen.
The Residual Stream Is the Model
Pre-LN, identity gradients, residual initialization, and why a block is two writes into one persistent stream.
Four Ways to Average the Past
From a loop mean to scaled causal self-attention, following every tensor shape through the implementation.
The Batch Is a Factory of Futures
A shifted batch, 4,096 simultaneous next-character bets, and the first numerical check that tells me whether the model is alive.