Index / term

Nanogpt

note 10Transformers

The Extra Code Is Not Another Brain

Flash Attention, mixed precision, DDP, shards, checkpointing, and HellaSwag without pretending I trained GPT-2.

note 05Transformers

The Batch Is a Factory of Futures

A shifted batch, 4,096 simultaneous next-character bets, and the first numerical check that tells me whether the model is alive.