fastText
Language detection for 176 languages in under a megabyte
Latest: v0.9.2 final line; repository archived March 19, 2024
Library for fast word representations and text classification built on subword n-grams, from FAIR in 2016. It shipped pre-trained word vectors for 157 languages and a famously tiny language-identification model, making industrial-strength NLP possible on a laptop CPU. The repo was archived in March 2024 — mission accomplished.
Why it matters
Made industrial-strength text classification and word embeddings run on a laptop CPU using subword n-grams, and shipped pretrained vectors for 157 languages plus a famously tiny language-identification model that still runs inside data pipelines everywhere. Archived in March 2024 after its ideas became standard practice.
Facts
- About 26k GitHub stars.
- Its lid.176 language-identification model recognizes 176 languages and compresses to under 1 MB — still deployed all over the industry years after the repo froze.
Try it yourself
pip install fasttext ↗ Pretrained vectors for 157 languages ↗ Language-ID model on Hugging Face ↗
Sources
arXiv ↗GitHub · fastText ↗fasttext.cc ↗