RoBERTa
BERT, done right
Latest: RoBERTa base/large (Jul 2019)
'A Robustly Optimized BERT Pretraining Approach' (July 2019): Meta showed BERT was severely undertrained, and that longer training on more data (160GB of text) with dynamic masking — no architecture change — topped the GLUE leaderboard. For years it was the default encoder for real-world NLP systems.
Why it matters
RoBERTa showed BERT was badly undertrained: same architecture, more data, longer training and dynamic masking topped GLUE. It is the landmark argument that recipes matter as much as architectures, and years later roberta-base still logs nearly ten million monthly Hugging Face downloads as a default encoder for real-world NLP.
Facts
- Beat BERT using the exact same architecture — the paper is a landmark argument that training recipes matter as much as architectures.
- Still among the most-downloaded models on Hugging Face years later.
Try it yourself
roberta-base on Hugging Face ↗ roberta-large on Hugging Face ↗ Read the paper ↗