MobileLLM
Proof that sub-billion models could think
Latest: MobileLLM-1.5B + ParetoQ quantized variants (1/1.58/2/3/4-bit); 125M/350M/600M/1B/1.5B line completed Nov 2024
FAIR's sub-billion-parameter architecture study proving that deep-and-thin transformers with embedding sharing, grouped-query attention, SwiGLU, and block-wise layer sharing beat wide-shallow designs on-device. Published at ICML 2024. Checkpoints from 125M to 1.5B released with full weights and training code — all non-commercial (CC-BY-NC 4.0 weights, FAIR NC code).
Why it matters
Reset how sub-billion language models are designed: deep-and-thin transformers with embedding sharing, grouped-query attention and block-wise layer sharing beat wide-shallow designs at equal size. Published at ICML 2024 with full weights and training code, it became the reference architecture for on-device models, including Meta's R1, Pro and Flash lines.
Facts
- MobileLLM-125M beat prior 125M SoTA by 2.7% and the 350M beat SoTA by 4.3% on zero-shot commonsense reasoning (46.3% vs GPT-neo-125M's 42.9%; 51.3% vs Pythia-410M's 46.6%).
- Training the 125M takes ~3 days on 32 A100s over 1T tokens; the 1.5B takes ~18 days.
- GitHub repo has ~1.5k stars.
- The HF collection also holds ParetoQ extreme-quantization variants down to 1-bit.
Try it yourself
Model collection on Hugging Face ↗ Code on GitHub ↗ Read the paper ↗
Lineage
Sources
arXiv ↗GitHub · MobileLLM ↗Hugging Face · MobileLLM 125M ↗Hugging Face · mobilellm ↗