title: Text-Conditioned Resampler For Long Form Video Understanding
summary: Videos are highly redundant data source and it is often enough to identify a
  few key moments to solve any given task. In this paper, we present a
  text-conditioned video resampler (TCR) module that uses a pre-trained and
  frozen visual encoder and large language model (LLM) to process long video
  sequences for a task. TCR localises relevant visual features from the video
  given a text condition and provides them to a LLM to generate a text response.
  Due to its lightweight design and use of cross-attention, TCR can process more
  than 100 frames at a time allowing the model to use much longer chunks of video
  than earlier works. We make the following contributions: (i) we design a
  transformer-based sampling architecture that can process long videos
  conditioned on a task, together with a training method that enables it to
  bridge pre-trained visual and language models; (ii) we empirically validate its
  efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art
  on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks
  which require longer video contexts and that can thus be used effectively for
  further evaluation of long-range video models.
paper_authors._id: 65825956117b524ef3ef37f5
paper_authors.name: Bruno Korbar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:41:40.441Z
paper_authors._id: 655c8ebc04a116e8ff33cecf
paper_authors.avatarUrl: /avatars/ec1a742f115672b852cd7c098587d30a.svg
paper_authors.fullname: Bruno Korbar
paper_authors.type: user
paper_authors.user: bjuncek
paper_authors._id: 65825956117b524ef3ef37f6
paper_authors.name: Yongqin Xian
paper_authors._id: 65825956117b524ef3ef37f7
paper_authors.name: Alessio Tonioni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:42:01.851Z
paper_authors._id: 63402b30e670ff9cf63d8caa
paper_authors.avatarUrl: /avatars/0aee84d132a78d4ec71663836a57a245.svg
paper_authors.fullname: Alessio Tonioni
paper_authors.type: user
paper_authors.user: Alessiot
paper_authors._id: 65825956117b524ef3ef37f8
paper_authors.name: Andrew Zisserman
paper_authors._id: 65825956117b524ef3ef37f9
paper_authors.name: Federico Tombari
paper_title: Text-Conditioned Resampler For Long Form Video Understanding
paper_summary: Videos are highly redundant data source and it is often enough to identify a
  few key moments to solve any given task. In this paper, we present a
  text-conditioned video resampler (TCR) module that uses a pre-trained and
  frozen visual encoder and large language model (LLM) to process long video
  sequences for a task. TCR localises relevant visual features from the video
  given a text condition and provides them to a LLM to generate a text response.
  Due to its lightweight design and use of cross-attention, TCR can process more
  than 100 frames at a time allowing the model to use much longer chunks of video
  than earlier works. We make the following contributions: (i) we design a
  transformer-based sampling architecture that can process long videos
  conditioned on a task, together with a training method that enables it to
  bridge pre-trained visual and language models; (ii) we empirically validate its
  efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art
  on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks
  which require longer video contexts and that can thus be used effectively for
  further evaluation of long-range video models.
paper_ai_keywords: text-conditioned video resampler (TCR) pre-trained and frozen visual encoder large language model (LLM) cross-attention transformer-based sampling architecture NextQA EgoSchema EGO4D-LTA challenge