Memory Consolidation Enables Long-Context Video Understanding
Most transformer-based video encoders are limited to short temporal contexts
due to their quadratic complexity. While various attempts have been made to
extend this context, this has often come at the cost of both conceptual and
computational complexity. We propose to instead re-purpose existing pre-trained
video transformers by simply fine-tuning them to attend to memories derived
non-parametrically from past activations. By leveraging redundancy reduction,
our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
context far into the past and exhibits excellent scaling behavior when learning
from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
long-context video understanding on EgoSchema, Perception Test, and Diving48,
outperforming methods that benefit from orders of magnitude more parameters.
65c5a63e0909ce480fdcabf8
Ivana Balažević
65c5a63e0909ce480fdcabf9
Yuge Shi
65c5a63e0909ce480fdcabfa
Pinelopi Papalampidi
65c5a63e0909ce480fdcabfb
Rahma Chaabouni
65c5a63e0909ce480fdcabfc
Skanda Koppula
admin_assigned
2024-02-09T09:29:39.915Z
65451016321f0393f453cd7b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XBmF81Uyj0N7lPclU1pUG.jpeg
Skanda Koppula
user
skoppula
65c5a63e0909ce480fdcabfd
Olivier J. Hénaff
admin_assigned
2024-02-09T09:29:50.941Z
64f30d8ceb5f2982081db604
/avatars/eedf65a104d099d8a60bbffe69bc2571.svg
Olivier Henaff
user
olivierhenaff
Memory Consolidation Enables Long-Context Video Understanding
Most transformer-based video encoders are limited to short temporal contexts
due to their quadratic complexity. While various attempts have been made to
extend this context, this has often come at the cost of both conceptual and
computational complexity. We propose to instead re-purpose existing pre-trained
video transformers by simply fine-tuning them to attend to memories derived
non-parametrically from past activations. By leveraging redundancy reduction,
our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
context far into the past and exhibits excellent scaling behavior when learning
from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
long-context video understanding on EgoSchema, Perception Test, and Diving48,
outperforming methods that benefit from orders of magnitude more parameters.
A memory-consolidated vision transformer (MC-ViT) extends the context of video understanding by leveraging non-parametrically derived memories from past activations, achieving state-of-the-art performance on long-context video understanding tasks.
transformer-based video encoders quadratic complexity pre-trained video transformers memory-consolidated vision transformer MC-ViT memory attend redundancy reduction long-context video understanding EgoSchema Perception Test Diving48