Before a model sees your words, a tokenizer chops them into pieces from a fixed vocabulary. Below are two of Meta's real tokenizers running in your browser: OPT-175B's from 2022 (the GPT-2 byte-level BPE, about 50,000 pieces) and Llama 3's from 2024 (128,000 pieces). Type anything, then watch what happens to code, Hindi, or emoji — and how much the bigger vocabulary saves.
pink = the tokenizer had no piece for this and fell back to raw bytes (one token per byte) · hover a token for its id · · marks a leading space · counts exclude the beginning-of-text token
"Every language is a way of seeing the world" — as translated by NLLB-200. A tokenizer trained mostly on English text pays a steep tax elsewhere; Llama 3's 28,000 extra non-English pieces were added to shrink exactly this bill.
| Language | Text | OPT-175B | Llama 3 | Llama 3 vs OPT |
|---|
Byte-pair encoding starts from 256 bytes and repeatedly merges the most frequent adjacent pair seen in training text — 50,000 merges later you have a vocabulary of common chunks like "ing", " the", or "Meta". Rare strings fall apart into bytes.
Llama 2 used a 32,000-piece SentencePiece vocabulary. For Llama 3, Meta switched to a tiktoken-style tokenizer with 100K pieces from tiktoken plus 28K added for non-English languages — reporting that English compression improved from 3.17 to 3.94 characters per token (The Llama 3 Herd of Models, §3.2). More text per token means more context and lower cost.
Tokens are the unit models pay for: context windows, prices, and speed are all counted in tokens. A Hindi sentence that costs several times the tokens of its English twin (see the table above) gets that much less context — and that much bigger a bill.
FAIR's Byte Latent Transformer (Dec 2024) reads raw bytes and groups them into patches by entropy — spending compute where text is surprising, not where a vocabulary happened to split. It matches Llama 3 with up to 50% fewer inference FLOPs.