Memory Layers at Scale
Capacity without FLOPs: a lookup table the model learns to consult
Latest: Paper + code (Dec 2024)
FAIR work scaling trainable key-value memory layers to add capacity without adding FLOPs: a 1.3B model with memory matched models trained on 2-4x more compute, and beat MoE models at equal budget on factual tasks. Part of the same December 2024 FAIR wave as LCM and BLT — three simultaneous bets against the standard dense transformer.
Why it matters
Memory layers add trainable key-value lookup capacity without adding FLOPs: a 1.3B model with memory matched dense models trained on 2-4x more compute and beat MoE at equal budget on factual tasks, scaling to 128B memory parameters. It is one of FAIR's three December 2024 bets against the standard dense transformer.
Facts
- Scaled memory pools past 1 million keys and 128B memory parameters.
- Especially strong on factual QA — the memory acts like a built-in lookup table the model learns to consult.
Try it yourself
Code on GitHub ↗ Read the paper ↗