Blog

Notes on what I'm building and learning: projects from my GitHub, experiments with models, and ideas I'm working through.

Oct 29, 2026 · 9 min read

How I trained GPT-2 small from scratch

I wrote GPT-2 by hand in PyTorch, then trained GPT-2 small (124 million parameters) from random weights on 9.85 billion tokens. It took one rented GPU, 15.6 hours of training, and about $12 of compute in all. On held-out educational web text, the kind it trained on, it scores a lower loss than OpenAI's original.

  • LLMs
  • PyTorch
  • Training
Read post