In February 2019, OpenAI released GPT-2 small, a 124-million-parameter language model (117 million in the original paper, a count OpenAI later said was wrong). In October 2026, I trained my own from scratch, with code I built. To compare them, I held back 1% of my dataset, 99.6 million tokens of educational web pages, so my model never trained on it. Then I had both models predict each next token across that text. Mine averaged a loss of 3.05, and OpenAI’s averaged 3.30.
From one book to 10 billion tokens
The project started with a 6.8-million-parameter model trained on The War of the Worlds in about a minute on my M1 Pro MacBook. The code is written to be read, with one small file per part and plain names (token_embedding, not wte). OpenAI’s GPT-2 small weights also load into it and give the same outputs as Hugging Face’s implementation, to within 1e-13 in 64-bit math (0.0000000000001), which confirms the architecture matches GPT-2’s exactly.
Scaling up from one book to 10 billion tokens meant rebuilding the training pipeline piece by piece. The biggest changes:
- Tokenize once. Ten billion tokens can’t be re-tokenized every run or held in memory, so the text is tokenized one time into files of 2-byte tokens, which take 20 GB on disk instead of 80 GB as PyTorch’s default 64-bit integers.
- Read straight off disk. Batches come from memory-mapped files, so the data never has to fit in RAM.
- Gradient accumulation. A 524,288-token batch doesn’t fit on the GPU at once, so each step adds up the gradients of 32 smaller batches before it updates the weights.
- Survive disconnects. Google Colab can drop a session at any time, so data prep and training both pick up where they stopped.
- Speed. bf16 math, FlashAttention, and
torch.compile(more on that last one below).
The data
The model learned from FineWeb-Edu, a set of Common Crawl web pages that Hugging Face filtered for educational quality. I used its 10-billion-token sample, pinned to one exact version so a prep resumed days later reads the same pages. Every 100th page, whole, was held back for validation and never used in training.
Prep took about two hours on the Colab machine’s 12 CPU cores. It streams pages from Hugging Face, tokenizes them with GPT-2’s tokenizer, marks where each page ends, and writes 99 training files and 10 validation files straight to Google Drive. Training then reads the files once, in order, over 18,794 steps of 524,288 tokens. Since the model never sees the same text twice, there’s little to overfit, so dropout is off.
The recipe
Most of the settings come from the GPT-3 paper’s recipe for its 125M model:
| Setting | Value |
|---|---|
| Batch | 524,288 tokens per step (16 sequences of 1,024 tokens, accumulated 32 times) |
| Learning rate | warmup over 715 steps to 6e-4, then a cosine curve down to 6e-5 |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on the 2-D weights |
| Gradient cap | total gradient norm at most 1.0 |
| Dropout | 0.0 (the GPT-3 paper doesn’t give one; nanoGPT and llm.c use 0.0) |
Testing before spending money
Before paying for GPU time, I tested everything on my Mac, and that turned up a real PyTorch bug. A practice run with torch.compile on the Mac’s GPU ended at a validation loss of 5.92, while the same run without it reached 5.70. Tracking it down showed that on Apple GPUs, PyTorch’s compiler fused two operations incorrectly, which made the position-embedding gradient too big by a factor of the batch size. It turned out to be a known bug, already reported to PyTorch and since fixed in its development code. The repo now refuses to use torch.compile on a Mac GPU.
The full-size model did run on my M1 Pro, but only at about 1,750 tokens per second, with 23 GB of swap on a 16 GB laptop. At that speed, 9.85 billion tokens would take more than two months, so the real run went to a rented NVIDIA A100 (40 GB) on Google Colab Pro+.
The final rehearsal took about 35 minutes on the A100 itself, covering a small slice of data, a GPU check, and a short practice run.
- FlashAttention really runs, and the compiled model’s gradients match the plain model’s to within 1.7 millionths.
- It trains at about 176,000 tokens per second, using 14 GB of the GPU’s 40 GB.
- I stopped it at step 110 and restarted it, and it resumed at step 100 with the same losses as before, to four decimal places.
The run
Training ran overnight for 15.6 hours, from about 3:20 PM on October 23 to 7 AM the next morning, at a steady 176,000 tokens per second with the GPU 99% busy.
The loss falls off a cliff in the first half-billion tokens, from 10.96 (no better than random guessing) to 4.12, as the model picks up the basics of English. The remaining 9.3 billion tokens bring it down slowly, from 4.12 to 3.01. The last check was the best one, so the model was still improving when the data ran out. The training loss wobbles, most likely because the files are read in order and the text drifts through different years of the web crawl. The validation loss, measured on the same text every time, stays smooth.
Partway through, the Colab page said “Saving failed,” and later “disconnected,” but the training kept running on the GPU. I kept an eye on it from Colab’s terminal instead, with nvidia-smi and the timestamps on the saved checkpoints.
The score
After training, I scored my model and OpenAI’s GPT-2 small on all 99,631,104 held-out tokens, which took 27 minutes.
| Model | Validation loss | Perplexity |
|---|---|---|
| My GPT-2 small | 3.05 | 21.2 |
| OpenAI’s GPT-2 small (2019) | 3.30 | 27.0 |
In perplexity terms, my model is about as unsure of each next token as if it were choosing among 21 equally likely options, and OpenAI’s among 27.
The loss chart above ends at 3.01, not 3.05. Both numbers come from the same finished model, tested on different amounts of held-out text.
- 3.01 is from the quick checks during training. Every 250 steps, the run scored the model on only the first 327,680 held-out tokens, since scoring all 99.6 million each time would have added more than 30 hours. The last check, on the finished model, gave 3.01.
- 3.05 is from scoring the finished model on all 99,631,104 held-out tokens. That is its true validation loss, and the number I compare with OpenAI’s.
Both models are the same size and architecture, so the difference most likely comes from the data and the recipe. FineWeb-Edu is filtered for educational quality, while OpenAI’s 2019 training set, WebText, was about 40 GB of text from 8 million web pages linked from Reddit. Most of my settings also come from the GPT-3 paper, which came out more than a year after GPT-2. The comparison does favor my model, though, because the held-out text is FineWeb-Edu, the kind of text it trained on and OpenAI’s didn’t. Mine is better at predicting educational web text, which isn’t the same as being better at everything.
What it learned
Before training, the model’s top guesses for the next token are noise. After training, they make sense:
| Prompt | Before training | After training |
|---|---|---|
| “In this lesson, we will learn” | “akh” 0.022%, “interface” 0.015%, “Brittany” 0.014% | “how” 36.0%, “about” 27.3%, “the” 10.5% |
| “The Martians” | “dump” 0.017%, “adhesive” 0.015%, “analyze” 0.015% | “were” 9.6%, “,” 7.6%, “are” 5.5% |
With the prompt in bold, here’s what the untrained model writes:
In this lesson, we will learn contraceptive bull 75 aheadarchingatted congratulations exileditairepees point663akh 171 Hull 03 point FROM ADHD1027…
And the trained model, from the same prompt:
In this lesson, we will learn some useful and practical concepts relating to the topic of water chemistry.
It also learned which words mean similar things. In GPT-2 small, each token is a row of 768 numbers, and related words should end up pointing in similar directions. Cosine similarity measures this from the angle between two rows, on a scale from −1 to 1:
- 1 means the rows point the same way (0° apart)
- 0 means they’re unrelated (90° apart)
- −1 means they point in opposite directions (180° apart)
Before training, the rows are random, so almost every pair sits near 0, unrelated. Training pulls related words toward 1. Monday and Tuesday ended at 0.85 (32° apart), seven and eight at 0.80 (37°), and red and blue at 0.47 (62°).
Monday · Tuesday
cosine 0.00 → 0.85
angle 90° → 32°
seven · eight
cosine −0.01 → 0.80
angle 91° → 37°
red · blue
cosine −0.06 → 0.47
angle 93° → 62°
Facts are hit or miss at this size. Given France: Paris / Italy: Rome / Spain:, it answers Madrid, just like OpenAI’s GPT-2 small. But both also say Canada’s capital is Montreal and Australia’s is Sydney, naming a well-known city instead of the capital. With random sampling, mine once announced that Paris is the capital of the United States. As a base model, it continues text rather than looking facts up, and it isn’t a chatbot.
Sharing it
The trained weights are on Hugging Face under the MIT license. To publish them, an export script renames every weight to GPT-2’s original names (token_embedding becomes transformer.wte), saves them in the standard safetensors format, and checks that the converted model gives the same outputs before saving. Anyone can load it with the transformers library:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("John-Machado/gpt2-small-fineweb-edu")
tokenizer = AutoTokenizer.from_pretrained("John-Machado/gpt2-small-fineweb-edu")
ids = tokenizer("The water cycle is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_k=50, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0]))
To train your own copy, the repo’s README walks through every step of this run on Colab.
What it cost
| Step | Time | Colab compute units |
|---|---|---|
| GPU check and practice run | about 35 min | ~4 |
| Data prep | about 2 h | ~15 |
| Training | 15.6 h | ~83 |
| Scoring against OpenAI’s model | 27 min | ~2.5 |
| Idle time between steps | ~10 | |
| Total | about 19 h, ~20 min of it hands-on | ~115, about $12 |
Colab Pro+ costs $50 a month and includes 500 compute units, so the whole project used about a quarter of one month’s units.
What’s next
Next, I plan to train the bigger GPT-2 sizes, each on about 20 tokens per parameter, the compute-optimal ratio from the Chinchilla paper. Medium reuses the same 10 billion tokens, a bit more than the ~7 billion that ratio gives it. Large and XL need more data, about 15 billion and 31 billion tokens. My rough estimates, scaled from this run:
| Size | Parameters | GPU time | GPU cost (est.) |
|---|---|---|---|
| Medium | 355M | ~44 A100-hours | ~$23 |
| Large | 774M | ~150 A100-hours | ~$80 |
| XL | 1.56B | ~590 A100-hours | ~$310 |
Costs at Colab’s A100 rate (~$0.53 an hour); rented GPUs elsewhere price differently.
Medium is next, on the same data. XL is a bigger jump. On one GPU it would take about 12 days even on an H100, so it really wants a machine with several GPUs and the multi-GPU code to use it.