title: Memory Consolidation Enables Long-Context Video Understanding
summary: Most transformer-based video encoders are limited to short temporal contexts
  due to their quadratic complexity. While various attempts have been made to
  extend this context, this has often come at the cost of both conceptual and
  computational complexity. We propose to instead re-purpose existing pre-trained
  video transformers by simply fine-tuning them to attend to memories derived
  non-parametrically from past activations. By leveraging redundancy reduction,
  our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
  context far into the past and exhibits excellent scaling behavior when learning
  from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
  long-context video understanding on EgoSchema, Perception Test, and Diving48,
  outperforming methods that benefit from orders of magnitude more parameters.
paper_authors._id: 65c5a63e0909ce480fdcabf8
paper_authors.name: Ivana Balažević
paper_authors._id: 65c5a63e0909ce480fdcabf9
paper_authors.name: Yuge Shi
paper_authors._id: 65c5a63e0909ce480fdcabfa
paper_authors.name: Pinelopi Papalampidi
paper_authors._id: 65c5a63e0909ce480fdcabfb
paper_authors.name: Rahma Chaabouni
paper_authors._id: 65c5a63e0909ce480fdcabfc
paper_authors.name: Skanda Koppula
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:29:39.915Z
paper_authors._id: 65451016321f0393f453cd7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XBmF81Uyj0N7lPclU1pUG.jpeg
paper_authors.fullname: Skanda Koppula
paper_authors.type: user
paper_authors.user: skoppula
paper_authors._id: 65c5a63e0909ce480fdcabfd
paper_authors.name: Olivier J. Hénaff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:29:50.941Z
paper_authors._id: 64f30d8ceb5f2982081db604
paper_authors.avatarUrl: /avatars/eedf65a104d099d8a60bbffe69bc2571.svg
paper_authors.fullname: Olivier Henaff
paper_authors.type: user
paper_authors.user: olivierhenaff
paper_title: Memory Consolidation Enables Long-Context Video Understanding
paper_summary: Most transformer-based video encoders are limited to short temporal contexts
  due to their quadratic complexity. While various attempts have been made to
  extend this context, this has often come at the cost of both conceptual and
  computational complexity. We propose to instead re-purpose existing pre-trained
  video transformers by simply fine-tuning them to attend to memories derived
  non-parametrically from past activations. By leveraging redundancy reduction,
  our memory-consolidated vision transformer (MC-ViT) effortlessly extends its
  context far into the past and exhibits excellent scaling behavior when learning
  from longer videos. In doing so, MC-ViT sets a new state-of-the-art in
  long-context video understanding on EgoSchema, Perception Test, and Diving48,
  outperforming methods that benefit from orders of magnitude more parameters.
paper_ai_keywords: transformer-based video encoders quadratic complexity pre-trained video transformers fine-tuning memories past activations redundancy reduction memory-consolidated vision transformer (MC-ViT) scaling behavior long-context video understanding EgoSchema Perception Test Diving48