Larger Tokenizers Suffer More Multi-Epoch Regret

On a fixed amount of text trained for 8 epochs, larger tokenizers suffer more multi-epoch regret.

同样多的文本训 8 个 epoch，分词器越大，multi-epoch regret 越高。

A tokenizer turns text into token ids from a fixed vocabulary. Each token gets an input embedding row (an input vector) and an output softmax row (weights to predict the next token). Byte-level BPE1 starts from the 256 single bytes. First, a pre-tokenizer splits text into words, numbers, and spaces. Then BPE repeatedly merges the most common adjacent pair until the vocabulary is full. To encode new text, it applies these merges in the same order. The diagram below shows this for the Marin tokenizer (Llama 3's BPE, with 128,000 ordinary tokens ranked by merge order).

分词器把文本转成固定词表里的 token 编号。每个 token 在输入端有一行 embedding（输入行），在输出端有一行预测下一个 token 的 softmax 权重（输出行）。字节级 BPE1 从 256 个单字节起步：预分词器先把文本切成词、数字和空格，BPE 再反复合并出现最多的相邻对，直到填满词表。编码新文本时，就按训练时的顺序依次做这些合并。下面的示意图用 Marin 分词器演示了这个过程（即 Llama 3 的 BPE，有 128,000 个按合并顺序编号的普通 token）。

8K stops here: it keeps ids below 8,000 8K 在这里停下：只保留编号小于 8,000 的 token

␣memor

izes #16,420 128K 128K

Encoding " memorizes" (with its leading space) from 10 bytes into 2 tokens in 8 merges: or (#269), es (#288), ␣m (#296), em (#336), iz (#450), ␣mem (#1,871), izes (#4,861), and ␣memor (#16,420). The 8K vocabulary keeps ids under 8,000 and stops after 7 merges at ␣mem|or|izes; 128K merges one step further to get ␣memor.把带前导空格的 " memorizes" 从 10 个字节经 8 次合并编成 2 个 token：or（#269）、es（#288）、␣m（#296）、em（#336）、iz（#450）、␣mem（#1,871）、izes（#4,861）和 ␣memor（#16,420）。8K 词表只留编号小于 8,000 的 token，7 次合并后停在 ␣mem|or|izes；128K 多合一步得到 ␣memor。

Keeping only the K lowest-ranked tokens of the 128K vocabulary (for K from 1K to 64K) is the same as stopping BPE at K tokens. We call any token outside the first 8K a rare token. Take " memor" (#16,420) as our running example. The 128K vocabulary gives it its own input and output rows. The 8K vocabulary splits it into " mem" and "or", and the byte vocabulary reads six bytes. Next, we measure what those dedicated rows do across multiple epochs.

只留 128K 词表里编号最小的 K 个 token（K 从 1K 到 64K），就等于让 BPE 在合出 K 个 token 时停手。我们把前 8K 之外的 token 叫作少见 token。全文拿 " memor"（#16,420）举例：128K 给它单独分了输入行和输出行；8K 词表把它拆成 " mem" 和 "or"；字节词表则直接读六个字节。下面我们就来测一测，多训几个 epoch 时，这些专属行会带来什么影响。

Training for multiple epochs on a small dataset hurts validation loss on unseen text2. The opening video shows how we measure this from the 256-byte vocabulary upward. Using FineWeb-Edu documents, we train a matched pair of runs for the same number of steps at each vocabulary size. The 1-epoch run trains for 1 epoch on unique text, while the 8-epoch run trains for 8 epochs on a random eighth of the text. We measure validation loss in macro bits per byte across the 16 Paloma domains, converting loss to bytes so we can compare vocabularies fairly. We define multi-epoch regret as the validation loss of the 8-epoch run, trained on one eighth of the text, minus the validation loss of the 1-epoch run on unique text, with the same steps. The word comes from online learning; here it means a gap in validation loss at the same training step, not a running sum. At 128K, this regret is 1.4667 − 1.3854 = 0.081. We use a 300m model, untied input and output tables, and MuonH, a Muon variant that fixes weight norms on a sphere. Our experiment page has the code and recipe and the runs with their training curves. Next, we track how regret changes as vocabularies add rare tokens like " memor".

拿小份数据训多个 epoch，模型在没见过的文本上验证损失会变差2。开头的视频展示了我们怎么从 256 字节词表开始测量这一点。在同样的 FineWeb-Edu 文档上，每个词表都训一对步数相同的模型：1-epoch 跑在不重复文本上训 1 个 epoch；8-epoch 随机抽八分之一文本训 8 个 epoch。我们用 Paloma 16 个领域的宏平均比特/字节来算损失（也就是把损失折算到每个字节上，方便跨词表比较）。我们把 multi-epoch regret 定义为：在八分之一文本上训 8 个 epoch 的验证损失，减去步数相同、在不重复文本上训 1 个 epoch 的验证损失。regret 这个词来自在线学习，但这里指同一训练步上的验证损失之差，不是累积量。在 128K 上，regret 是 1.4667 − 1.3854 = 0.081。实验用 300m 模型、MuonH 优化器（一种把权重范数固定在球面上的 Muon 变体），输入表和输出表不共享。实验主页放了代码和配方，还有运行记录与训练曲线。下面看看词表加入 " memor" 这类少见 token 后，regret 怎么变。

This study grew out of an earlier experiment. The hashed 2-gram and 3-gram input tables of the Over-Tokenized Transformer (Huang et al., 2025) lower the loss up to 520m and overfit when the data repeats. At 1.2B, which trains for 2.4 epochs, the gain becomes a loss. We therefore asked whether the vocabulary of a plain tokenizer has the same problem.

这项研究源于我们更早的一个实验。Over-Tokenized Transformer（Huang 等人，2025）的哈希 2-gram 和 3-gram 输入表在 520m 及以下能降低损失，但数据一重复就过拟合。到了 1.2B，数据要训 2.4 个 epoch，收益就变成了损失。所以我们想知道，普通分词器的词表是不是也有同样的问题。

Figure 1 shows that multi-epoch regret grows with vocabulary size at 300m. With fixed 4096-token windows (dashed line), regret rises from near zero on the 256-byte vocabulary to about 0.08 bits per byte at 128K. Vocabularies from 512 to 8K sit on a flat plateau. Above 8K, every doubling pushes regret higher and wipes out most of 128K's 1-epoch lead over 8K. Over 8 epochs, 128K lowers training loss only slightly more, yet every memorized bit hurts unseen text much more.

图 1 测了 300m 模型下不同词表的 multi-epoch regret。虚线是固定 4096 token 窗口的结果：regret 从 256 字节词表的几乎为零，涨到 128K 的约 0.08 比特/字节。512 到 8K 词表停在一个平缓的平台上。过了 8K，词表每翻一倍，regret 就往上涨一截，几乎吃掉了 128K 在 1 个 epoch 下对 8K 的优势。训 8 个 epoch 时，128K 的训练损失只多降了一点点，但它每多背下 1 比特，在未见文本上损失都多得多。

Even when every window holds the same text (solid line), multi-epoch regret still rises above 4K. With 4096-token windows, a smaller vocabulary packs less text per window and takes more steps to read the corpus: 54,247 steps for bytes versus 11,443 for 128K. Matching the text per window puts every run at about 11,444 steps. Then vocabularies from 256 bytes to 4K have a regret of 0.02 to 0.03 bits per byte, so bytes no longer sit near zero. Above 4K, regret still climbs with vocabulary size, from 0.031 at 8K to 0.081 at 128K. With equal text per window, 16K has 0.053 regret and 32K has 0.039, flipping their order from the 4096-token runs. Both runs in a setup use the same subset of documents, and changing the window length keeps only about 9 percent of those documents the same. That document shift may explain why 16K and 32K flip, and why byte regret rises from 0.0003 to 0.0175.

词表越小，4096 token 窗口装的文本就越少，读完同样多的文本要走更多步数：字节词表要走 54,247 步，128K 只要 11,443 步。图里的实线改成让每个窗口装同样的文本，每次运行都走约 11,444 步。这么对齐后，256 字节到 4K 词表的 multi-epoch regret 都在 0.02 到 0.03 比特/字节，字节词表不再接近零。4K 往上，regret 还是随词表变大而上升：8K 是 0.031，128K 是 0.081。装同样文本时，16K 的 regret 是 0.053，32K 是 0.039，顺序跟 4096 token 窗口时反了过来。同一套配置的两次运行重复的是同一批文档，换个窗口长度，重合的文档就只剩约 9%。所以 16K 和 32K 顺序颠倒，还有字节词表从 0.0003 涨到 0.0175，都可能来自换了一批被重复的文档。

In the right panel of Figure 1, we test an over-vocab setup on 128K. It adds hashed 2- and 3-gram input tables (25.7M rows in total that map adjacent token groups to vectors) and a second 2-gram output head. Its 1-epoch loss drops by about 0.02 bits per byte, but its multi-epoch regret jumps to 0.71, about nine times 128K's. The same panel shows a one-stage H-Net (Hwang, Wang and Gu, 2025). H-Net learns byte chunks directly from bytes without giving them lookup rows. It aims at the length of 128K tokens (4.74 bytes) and reaches about 4 bytes per chunk on the training text. Its regret is 0.031, under two fifths of 128K's 0.081. In a second run that reaches about 6.8 bytes per chunk, longer than 128K's tokens, the regret is 0.015, although the 1-epoch loss is higher (1.432 against 1.373). So compressing the sequence as far as 128K does not by itself create 128K's regret.

在图 1 右栏，我们给 128K 加上哈希 2-gram 和 3-gram 输入表（共 2570 万行，把相邻两三个 token 哈希到表里查向量），再加第二个预测 2-gram 的输出头，记作 over-vocab。它的 1-epoch 损失降了约 0.02 比特/字节，但 multi-epoch regret 暴涨到 0.71，大概是 128K 的 9 倍。同一栏还画了单阶段 H-Net（Hwang、Wang 和 Gu，2025）。它直接从字节学出块，但不给这些块建查表行。它的目标是 128K token 的长度（4.74 字节），在训练文本上实际约 4 字节一块。它的 regret 只有 0.031，不到 128K（0.081）的五分之二。另一次运行实际约 6.8 字节一块，比 128K 的 token 还长，regret 是 0.015，不过 1-epoch 损失更高（1.432，前者是 1.373）。所以把序列压得和 128K 一样短，本身并不会带来 128K 那样的 regret。

Figure 2 tests 130m and 520m models and finds the same vocabulary ordering as 300m. At 520m, multi-epoch regret wipes out 128K's 1-epoch lead over 8K, leaving them tied. Still, this overall average does not tell us where the regret lands: on common tokens, or on rare tokens like " memor" and unfamiliar domains.

图 2 测了 130m 和 520m 两个规模，词表的排序跟 300m 一样。在 520m 上，multi-epoch regret 抹平了 128K 在 1 个 epoch 下对 8K 的领先，最后两者打平。不过光看整体平均分，还看不出 regret 到底是落在常见 token 上，还是落在 " memor" 这类少见 token 和陌生领域上。

Unfamiliar domains suffer much higher multi-epoch regret at 128K. Following Bergsma et al.3, Figure 3 breaks regret down by domain. On the 8-epoch training subset, 128K memorizes nearly twice as many bits per byte as the 256-byte vocabulary. On unseen text, 128K's regret grows from about 0.02 bits per byte on unread training-distribution text to about 0.32 on code. Meanwhile, 8K and the 256-byte vocabulary barely move.

照着 Bergsma 等人3的做法，图 3 看 multi-epoch regret 落在哪些领域。在训了 8 个 epoch 的那份数据上，128K 每字节背下的比特数快赶上 256 字节词表的两倍。到了没见过的文本上，同分布的未读训练文本只让 128K 多出约 0.02 比特/字节的 regret，到了代码上就涨到约 0.32；而 8K 和 256 字节词表几乎没动。

Much of 128K's multi-epoch regret sits on tokens that the 8-epoch subset rarely holds. Figure 4 groups evaluation positions by how often the target token appears in the 8-epoch subset. Two effects stand out. First, 128K puts many targets into low-count bins, whereas 99.7 percent of byte targets appear over 10,000 times. Second, at the same count (from a few hundred upward), 128K has up to twice the regret per byte of 8K. Splitting tokens into ten frequency bins shows the same spike in 128K's rarest tenth, whether grouped by input token or predicted token. At 128K, tokens seen fewer than 1,000 times make up 9.5 percent of the text, yet hold 45 percent of the regret. The 8K rank cutoff itself matters less: tokens outside the first 8K are 37 percent of the text and 41 percent of the regret. Half of them appear fewer than 45 times, while " memor" appears 8,849 times. Next, we ask whether the byte vocabulary also degrades if we shrink the subset and run more epochs.

图 4 按目标 token 在 8-epoch 训练子集里的出现次数，把评测位置分组。这里能拆出两件事：第一，128K 把很多目标切进了低频档，而字节词表有 99.7% 的目标出现超过一万次；第二，就算出现次数一样（几百次往上），128K 每字节的 multi-epoch regret 最高也是 8K 的两倍。按训练词频分成十等份，不管是按输入 token 还是预测 token 分，128K 最少见的十分之一都会冒出同样的尖峰。在 128K 里，出现不到 1,000 次的 token 只占文本的 9.5%，却占了 45% 的 regret。前 8K 这条分界线本身反倒没那么关键：8K 之外的 token 占文本的 37%、占 regret 的 41%，其中一半出现不到 45 次，而 " memor" 其实出现了 8,849 次。看到这儿还有个问题：要是把训练子集缩得更小、训更多 epoch，字节词表自己会不会也跟着变差？

Does the byte vocabulary ever overfit? Yes, just later. In Figure 5, we fix the step count and shrink the training subset to run more epochs. The 256-byte vocabulary stays flat through 8 epochs and degrades from 16 epochs onward. Still, a smaller vocabulary packs fewer bytes into a window and takes more steps on the corpus. Next, we check whether window length or step count explains the vocabulary gap.

字节词表也会过拟合吗？会，只是来得更晚。图 5 固定总步数，不断缩小训练子集：256 字节词表在 8 个 epoch 以内一直很稳，从 16 个 epoch 开始变差。不过小词表每个窗口装的字节少，跑完语料走的步数多。我们还得排查一下，前面看到的词表差距是不是窗口长度或者步数搞出来的。

Neither a shorter window nor taking more steps explains the multi-epoch regret gap. An 8K token has about a quarter fewer bytes than a 128K token, so an 8K window holds a quarter less text and takes a third more steps. Figure 6 tests both factors. On the left, giving 128K the window bytes or step count of 8K leaves 128K's regret at about 0.08 bits per byte. Also, where in the window the regret sits is flat across positions. On the right, stretching the byte window to match a 128K window's text and step count raises 256-byte regret to about 0.02 bits per byte, still under a quarter of 128K's regret. And as Figure 1 showed, at equal steps the byte vocabulary sits level with every vocabulary up to 4K. Even with matched windows and steps, though, the main network sees " memor" as one token instead of six bytes. We still need to check whether sequence compression causes the regret.

差距不是窗口更短或步数更多造成的。8K 的每个 token 比 128K 少约四分之一字节，所以 8K 的窗口少装四分之一文本，步数多出三分之一。图 6 排查了这两个因素。左栏把 128K 的窗口字节数或步数改成跟 8K 一样，128K 的 multi-epoch regret 还是停在约 0.08 比特/字节，regret 在窗口各个位置上的分布也是平的。右栏反过来，把字节窗口拉长，让它装的文本和走的步