Quantized Llama 3.2
Quantized Llama 3.2 (1B/3B) + ExecuTorch
Llama 3.2 on a phone — 56% smaller, up to 4x faster
Latest: Llama 3.2 1B/3B Instruct QLoRA_INT4_EO8 and SpinQuant_INT4_EO8 (Oct 24, 2024)
Meta's first official quantized Llama releases (Oct 24, 2024): Llama 3.2 1B/3B Instruct in two flavors — Quantization-Aware Training with LoRA adaptors for accuracy, and SpinQuant post-training quantization for portability. 2-4x faster with 56% smaller size and 41% less memory, running on phones via PyTorch's ExecuTorch on Qualcomm, MediaTek, and Arm. Llama 3.2 Community License — commercial use permitted.
Why it matters
Meta's first official quantized Llama releases made small-model on-device deployment a supported path rather than a community hack: QAT with LoRA adaptors for accuracy and SpinQuant for portability, 2-4x faster with 56% smaller size and 41% less memory, running via ExecuTorch on Qualcomm, MediaTek and Arm under the commercial Llama 3.2 license.
Facts
- Measured on a OnePlus 12: 2.5x faster decode, 4.2x faster prefill, 56% average size reduction, 41% less memory vs BF16 — verified also on Samsung S24+/S22.
- QLoRA keeps accuracy within 1.95% of BF16 on the 3B.
- SpinQuant (learned rotation matrices + GPTQ, INT4 groupwise g32 weights with 8-bit dynamic activations) is itself a FAIR research contribution with its own open GitHub repo.
- This is the family's commercially-licensed on-device option.
Try it yourself
Llama 3.2 1B SpinQuant on Hugging Face ↗ Llama 3.2 3B QLoRA on Hugging Face ↗ Run Llama on-device (ExecuTorch example) ↗
Lineage
Sources
GitHub · executorch ↗GitHub · SpinQuant ↗Hugging Face ↗Meta AI blog ↗PyTorch blog ↗