Brain & AI: Decoding Perception from MEG/EEG
Reconstructing what you see from brain signals
Latest: Image decoding from MEG (October 2023); speech-perception decoding (2022 preprint, published 2023)
FAIR's neuroscience program showed that self-supervised AI models align with brain activity — and can decode it. A 2023 Nature Machine Intelligence paper decoded perceived speech from non-invasive MEG/EEG using wav2vec 2.0-style contrastive learning; a companion 2023 system reconstructed seen images from MEG signals in near real time using DINOv2 embeddings.
Why it matters
Showed self-supervised AI representations align with brain activity well enough to decode it: contrastive learning between wav2vec 2.0 speech embeddings and MEG/EEG identified perceived speech, and DINOv2 embeddings reconstructed seen images from MEG in near real time. Code covering 175 volunteers and 160+ hours of recordings is open.
Facts
- From 3 seconds of MEG activity, the speech decoder identified the matching audio segment from over 1,500 candidates with up to 41% top-10 accuracy averaged across participants — using no implants at all.
Try it yourself
Code on GitHub (brainmagick) ↗ Speech decoding paper ↗ Image decoding paper ↗
Lineage
Sources
arXiv · 2208.12266 ↗arXiv · 2310.19812 ↗Meta AI blog ↗