N-gram hinted language modeling: NanoGPT Speedrun in 6.7s

TLDR; How I improve the NanoGPT Speedrun world record from 39.9s to 6.7s with retrieval-augmented n-gram hints. This is a larger improvement than all prior improvements since 2025 combined.

Play again

NanoGPT Speedrun world-record time since January 2025. Every record since then cut the time by 81.4% in total. Retrieval-augmented n-gram hints cut it by another 83.2%, from 39.9s to 6.7s.

Introduction

The NanoGPT Speedrun challenges you to train a model to GPT-2-level loss on an 8xH100 node as fast as possible. Since its start in May ‘24, it has seen nearly 100 records, cutting training time from 45 minutes to 39.9 seconds.

Until now, the CPUs were mostly idle, so I use them to build a hint corpus: a lookup table of which tokens follow each n-gram (a sequence of n tokens) in the training data. We give the model the most common next tokens as hints, and it learns how much to trust them, resulting in much lower loss on hinted tokens.

This post contains two parts. First, I show that text repeats and how we can exploit it. Then I walk through how we build a large hint corpus quickly.

Web text is repetitive, with validation n-grams also appearing in training3FineWeb is scraped from the web, where the same page can be crawled twice and the same boilerplate shows up on many sites. 6-grams and even 20-grams are short, so they also catch cookie banners, book excerpts and facts shared by otherwise different documents. Even with fuzzy deduplication of train and validation documents, the hints give large gains, with training finishing in around 10 seconds.. For a 20-gram that also appears in training, the most common next token is correct 98% of the time, and 54% for 6-grams. If we record which token follows each n-gram in the training data, we can give the model the most common ones as hints for what is likely to come next. The hints come only from the training data, just like the model's weights, and we never build hints from validation tokens.

n = 2

val tokens with a hint 99.7%

next token right, given a match 22.9%

n = 3

val tokens with a hint 93.4%

next token right, given a match 28.9%

n = 4

val tokens with a hint 75.3%

next token right, given a match 34.8%

n = 5

val tokens with a hint 53.0%

next token right, given a match 42.8%

n = 6

next token right, given a match 54.1%

val tokens with a hint 35.5%

n = 7

next token right, given a match 66.9%

val tokens with a hint 25.0%

n = 8

next token right, given a match 78.1%

val tokens with a hint 19.3%

n = 9

next token right, given a match 85.9%

val tokens with a hint 16.4%

n = 10

next token right, given a match 90.6%

val tokens with a hint 14.7%

n = 11

next token right, given a match 93.3%

val tokens with a hint 13.7%

n = 12

next token right, given a match 94.8%

val tokens with a hint 13.1%

n = 14

next token right, given a match 96.2%

val tokens with a hint 12.2%

n = 16

next token right, given a match 97.0%

val tokens with a hint 11.6%

n = 20

next token right, given a match 97.7%

val tokens with a hint 10.7%

n = 24

next token right, given a match 98.2%

val tokens with a hint 10.1%

n = 32

next token right, given a match 98.7%

val tokens with a hint 9.1%

n = 48

next token right, given a match 99.2%

val tokens with a hint 7.9%

n = 64

next token right, given a match 99.4%

val tokens with a hint 7.2%

0020204040606080801001002346812203264next token right,given a match (%)n-gramval tokens witha hint (%)

n = 2

val tokens with a hint 99.7%

next token right, given a match 22.9%

n = 3

val tokens with a hint 93.4%

next token right, given a match 28.9%

n = 4

val tokens with a hint 75.3%

next token right, given a match 34.8%

n = 5

val tokens with a hint 53.0%

next token right, given a match 42.8%

n = 6

next token right, given a match 54.1%

val tokens with a hint 35.5%

n = 7

next token right, given a match 66.9%

val tokens with a hint 25.0%

n = 8

next token right, given a match 78.1%

val tokens with a hint 19.3%

n = 9

next token right, given a match 85.9%

val tokens with a hint 16.4%

n = 10

next token right, given a match 90.6%

val tokens with a hint 14.7%

n = 11

next token right, given a match 93.3%

val tokens with a hint 13.7%

n = 12

next token right, given a match 94.8%

val tokens with a hint 13.1%

n = 14

next token right, given a match 96.2%

val tokens with a hint 12.2%

n = 16

next token right, given a match 97.0%

val tokens with a hint 11.6%

n = 20

next token right, given a match 97.7%

val tokens with a hint 10.7%

n = 24

next token right, given a match 98.2%

val tokens with a hint 10.1%

n = 32

next token right, given a match 98.7%

val tokens with a hint 9.1%

n = 48

next token right, given a match 99.2%

val tokens with a hint 7.9%

n = 64

next token right, given a match 99.4%

val tokens with a hint 7.2%

For each validation token, we check whether the n tokens before it (n-gram) also appear somewhere in the first 37B training tokens. Pink: the share of validation tokens with such a match. Purple: how often the token that most often follows the n-gram in training is the correct next token. Longer n-grams match fewer tokens, but their hints are almost always right.

Learning to use the hints

Long hints are usually correct, but how much we trust a hint depends on its length, how often the n-gram appears in the training data (count) and how often the n-gram is followed by the same token (purity). It should also depend on how confident the model is. Rather than mixing the model prediction and n-gram hint distribution, as infini-gram4Liu et al. (2024) build an n-gram model over trillions of tokens and interpolate its next-token distribution with a neural language model's at inference time, without training the model to use them. does, we give the hints to our model and let it decide how to use them5Training with the hints makes our backbone transformer model slightly worse when evaluated without hints, given the same number of steps..

To give the model a hint, we take the hinted token's embedding from the model's own embedding table. Each kind of hint, given by its length, count and purity6The length is the number of tokens in the n-gram (6, 10 or 20). The count is how many times the n-gram appears in the training corpus, and the purity is how often it is followed by the hinted token.We group the counts into 7 buckets and the purities into 6, so with 3 lengths there are 7 × 6 × 3 = 126 trust amounts and trust embeddings., has two learned parameters: a trust amount that scales the hint embedding up or down, and a trust embedding that we add to it, telling the model what kind of hint it is. We add the hints to the residual stream in three places. Right after the embedding, in the middle of the layers, and right before the LM head, each time with its own learned scale7The scales differ since the residual norm grows with depth..

EmbeddingTransformer blockslayers 0–6Transformer blockslayers 7–10Output headlogitstrust embeddinghint embeddingpuritycountn-gram length {6, 10, 20}trust amount

svg]:block [&>svg]:h-auto [&>svg]:w-full [&>svg]:overflow-visible">EmbeddingTransformer blockslayers 0–6Transformer blockslayers 7–10Output headlogitstrust amounthint embeddingtrust embeddingn-gram length{6, 10, 20}countpurity

The pink path carries the hint. ht is the embedding of the hint's next token. Its n-gram length ℓ, count bucket c and purity bucket p select a learned trust amount σ and trust embedding β. The result is added to the residual stream after the embedding, after layer 6 and before the LM head, each time scaled by its own α.

Scaling the hint corpus has strong returns

We train a model on 70.8M tokens with hints drawn only from tokens we have already trained on and evaluate it with hint corpora from 75M to 37B tokens. If we train on a large hint corpus from the start, the backbone model relies on the hints too much, so a small hint corpus during training and a large one during validation works better8Switching to a large hint corpus midway through training is faster, but timing is more fragile, since the CPU needs to finish building the hints right before the GPU needs them.. Loss keeps improving with hint corpus size, and each doubling helps more than the previous. A larger hint corpus gives more hints. 1% of validation tokens have a 20-gram match in a 75M-token hint corpus, which increases to 11% in a 37B-token one9With fuzzy document deduplication, 6.6% of validation tokens still have a 20-gram match..

75M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.7219

target loss (3.28) 3.2800

150M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.7092

target loss (3.28) 3.2800

300M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6968

target loss (3.28) 3.2800

600M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6782

target loss (3.28) 3.2800

1.2B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6562

target loss (3.28) 3.2800

2.4B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6267

target loss (3.28) 3.2800

4.8B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.5828

target loss (3.28) 3.2800

9.6B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.5209

target loss (3.28) 3.2800

19.2B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.4253

target loss (3.28) 3.2800

37B

no hint corpus 3.7454

target loss (3.28) 3.2800

hint corpus (70.8M transformer tokens) 3.2739

3.33.43.53.63.775M300M1.2B4.8B37Bhint corpus tokens Nval loss

75M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.7219

target loss (3.28) 3.2800

150M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.7092

target loss (3.28) 3.2800

300M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6968

target loss (3.28) 3.2800

600M

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6782

target loss (3.28) 3.2800

1.2B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6562

target loss (3.28) 3.2800

2.4B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.6267

target loss (3.28) 3.2800

4.8B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.5828

target loss (3.28) 3.2800

9.6B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.5209

target loss (3.28) 3.2800

19.2B

no hint corpus 3.7454

hint corpus (70.8M transformer tokens) 3.4253

target loss (3.28) 3.2800

37B

no hint corpus 3.7454

target loss (3.28) 3.2800

hint corpus (70.8M transformer tokens) 3.2739

We train one model on 70.8M tokens, with hints drawn only from tokens it has already trained on. We then evaluate that same model with hint corpora built from the first N training tokens, from 75M (about what it trained on) to 37B (record's hint corpus size). The dashed line is the same model evaluated without hints. Validation loss shows no sign of saturating as the hint corpus grows.

We build the hint corpus on the CPUs while the GPUs train, so a faster builder means a larger hint corpus. Our best hint builder is 500x faster than a naive Python implementation and 11x faster than an auto-researched implementation. The next sections build it step by step, bringing the record from 39.9s to 6.7s.

The simplest hint builder is a hash map in Python. The key is a hashed n-gram and the value is a list (with duplicates) of the next tokens. To look up a hint during validation, we compute the hash of the n-gram and count how many times each next token appears. It is easy to