Byte Latent Transformer
Byte Latent Transformer (BLT)
The architecture that killed the tokenizer
Latest: Dynamic BLT weights 1B & 8B (May 2025)
A tokenizer-free LLM architecture that reads raw bytes, dynamically grouping them into patches sized by entropy — spending compute where text is hard, coasting where it's easy. First byte-level architecture to match tokenization-based models (Llama 3 class) at scale, with up to 50% fewer inference FLOPs; weights for 1B and 8B released 2025.
Why it matters
BLT removed the tokenizer — the last hand-engineered component in modern LLMs — by reading raw bytes and grouping them into entropy-sized patches. It is the first byte-level architecture to match Llama 3-class tokenized models at scale, with up to 50% fewer inference FLOPs and native robustness to typos and character-level noise.
Facts
- It killed the tokenizer — the one hand-engineered relic every modern LLM still depended on.
- Scaling study ran to 8B parameters and 8 trillion training bytes.
- Because there is no tokenizer, it is naturally robust to typos, weird spellings, and the classic 'how many r's in strawberry' failure class.
- Matches Llama 3 training performance with up to 50% fewer inference FLOPs, and is far more robust to typos and character-level noise because it reads raw bytes.
- 14 authors (Pagnoni et al.).
Try it yourself
BLT-1B weights on Hugging Face ↗ Code on GitHub ↗ Read the paper ↗
Sources
arXiv ↗GitHub · blt ↗Meta AI research ↗