Multi-token prediction
Several tokens per step: stronger code models, up to 3x faster
Latest: Paper Apr 2024; 7B code models released Jul 2024
'Better & Faster Large Language Models via Multi-token Prediction' (April 2024): training models to predict several future tokens at once via parallel output heads yields better sample efficiency and stronger code models — 13B models solved 12-17% more HumanEval/MBPP problems — while enabling up to 3x faster inference via self-speculative decoding. 7B weights released for research.
Why it matters
Multi-token prediction showed that training a model to predict several future tokens at once — with zero training overhead — yields better sample efficiency and stronger code models, with 13B models solving 12-17% more HumanEval/MBPP problems, plus up to 3x faster self-speculative decoding. The idea flowed straight into industry speculative-decoding heads.
Facts
- The idea flowed into industry practice (speculative decoding heads); Meta released the 7B multi-token model on Hugging Face under a non-commercial license.
- Zero training-time overhead versus next-token prediction.
Try it yourself
7B code models on Hugging Face ↗ Read the paper ↗