Tokenizers
Each byte is an integer from 0 to 255, so the vocabulary contains just 256 tokens. Furthermore, we no longer need to worry about out-of-vocabulary tokens because any future characters will still be encoded as a sequence…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Each byte is an integer from 0 to 255, so the vocabulary contains just 256 tokens. Furthermore, we no longer need to worry about out-of-vocabulary tokens because any future characters will still be encoded as a sequence of bytes.
The trade-off is sequence length. Since there are currently 172,808 characters in the Unicode standard, some code points must map to more than one byte. This means that all the issues we had around sequence length with a character-level tokenizer are even more exacerbated here.
Note that most of these problems live at the level of compute and memory efficiency, so it is not crazy to imagine a future where we indeed use byte-level tokens. In the meantime, however, we need better solutions.
We are dealing with a trade-off between vocabulary size and sequence length, or the ability of a tokenizer to compress text. We want tokenizers with a relatively small vocabulary size, where each token is dense with semantic meaning and information. See Figure 2.
To achieve this, it makes sense to use subwords as tokens. These are units falling between individual bytes and full words. The question becomes: which subwords should we include in the vocabulary?
The guiding principle is that common sequences should be represented by single tokens, while rare words should be decomposable into smaller, meaningful tokens. For instance, if “the” and “ing” occur frequently in the training data, it may be useful to assign them their own tokens. Rare substrings, by contrast, need not become separate vocabulary entries.
In this sense, a tokenizer is a compression algorithm. It is trained on a text corpus, independently of the language model, and then used to encode and decode text.
There is no single best solution. Several algorithms have been proposed, including WordPiece in 2012, Byte-Pair Encoding (BPE) in 2015 or Unigram in 2018. In what follows, we focus on the BPE tokenizer.
At a conceptual level, the BPE algorithm is very easy to understand.
We start with a vocabulary of 256 tokens, one for each possible byte. We then identify the most frequent pair of adjacent tokens in the corpus, merge that pair into a new token, and repeat until the vocabulary reaches the desired size.
BPE was originally introduced as a data-compression algorithm in 1994. In 2016, it was proposed as a tokenizer at the character level. The method gained wider popularity when OpenAI used a byte-level variant for GPT-2, described in their 2019 paper.
One important addition in GPT-2 was pre-tokenization. OpenAI realized that many versions of common words like “dog” appeared in the final vocabulary in many variations like “dog.”, “dog!” or “dog?”. These tokens get completely different token IDs, even though they are very similar semantically. To avoid this, they used a regular expression to split the text into units across which merges are not allowed.
The full algorithm is shown below, with implementation details abstracted away. In practice, a naive implementation can be made substantially more efficient by caching pair counts instead of recomputing them at every iteration, parallelizing pre-tokenization across a large corpus, and maintaining a heap to efficiently identify the most frequent pair.
Pair = tuple[bytes, bytes] NUM_BYTES = 2**8
def train_bpe( input_path: str, vocab_size: int, special_tokens: list[str] ) -> tuple[dict[int, bytes], list[Pair]]: """Train a BPE tokenizer on the input text file.""" assert vocab_size >= NUM_BYTES + len(special_tokens)
Initialize vocabulary
vocab = initialize_vocab(special_tokens)
Pre-tokenize
pretokens, pretoken_counts = pretokenize( input_path, special_tokens, )
Perform BPE
merges: list[Pair] = [] pair_counts, pair_to_indices = initialize_pair_stats( pretokens, pretoken_counts )
for token_id in range(len(vocab), vocab_size): if not pair_counts: break
Select the most frequent pair,
breaking ties lexicographically
pair = max( pair_counts, key=lambda pair: (pair_counts[pair], pair) ) merges.append(pair) vocab[token_id] = pair[0] + pair[1]
apply_merge( pretokens, pretoken_counts, pair, pair_counts, pair_to_indices, )
return vocab, merges
The output consists of a vocabulary of the desired size and a sequence of merges that together define how the tokenizer encodes and decodes future text. For example, the following are the first merges produced when training BPE on the OpenWebText dataset.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。