CoTracker
Follow any pixel through a video — even when it disappears
Latest: CoTracker3 (Oct 15, 2024)
Meta plus Oxford VGG's line of transformer point trackers that follow any pixel through a video jointly with its neighbors, staying locked on through occlusions and out-of-frame excursions. CoTracker3 (October 2024) simplified the architecture and used pseudo-labeled real videos to beat prior trackers with 1,000x less training data.
Why it matters
CoTracker made dense point tracking practical by tracking points jointly rather than independently, staying locked on through occlusions and off-screen excursions. CoTracker3 then matched or beat prior trackers with 1,000x less training data via pseudo-labeled real videos, and its rainbow point-trail demos made it a research favorite.
Facts
- Tracking points jointly (co-tracking) rather than independently was the key trick; the mesmerizing rainbow point-trail demo videos made it a research-Twitter favorite.
Try it yourself
Run the official demo in Colab ↗ cotracker3 on Hugging Face ↗ Code on GitHub ↗
Sources
GitHub · co-tracker ↗Hugging Face ↗