Text-Conditioned Resampler For Long Form Video Understanding
Videos are highly redundant data source and it is often enough to identify a
few key moments to solve any given task. In this paper, we present a
text-conditioned video resampler (TCR) module that uses a pre-trained and
frozen visual encoder and large language model (LLM) to process long video
sequences for a task. TCR localises relevant visual features from the video
given a text condition and provides them to a LLM to generate a text response.
Due to its lightweight design and use of cross-attention, TCR can process more
than 100 frames at a time allowing the model to use much longer chunks of video
than earlier works. We make the following contributions: (i) we design a
transformer-based sampling architecture that can process long videos
conditioned on a task, together with a training method that enables it to
bridge pre-trained visual and language models; (ii) we empirically validate its
efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art
on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks
which require longer video contexts and that can thus be used effectively for
further evaluation of long-range video models.
65825956117b524ef3ef37f5
Bruno Korbar
admin_assigned
2023-12-20T09:41:40.441Z
655c8ebc04a116e8ff33cecf
/avatars/ec1a742f115672b852cd7c098587d30a.svg
Bruno Korbar
user
bjuncek
65825956117b524ef3ef37f6
Yongqin Xian
65825956117b524ef3ef37f7
Alessio Tonioni
admin_assigned
2023-12-20T09:42:01.851Z
63402b30e670ff9cf63d8caa
/avatars/0aee84d132a78d4ec71663836a57a245.svg
Alessio Tonioni
user
Alessiot
65825956117b524ef3ef37f8
Andrew Zisserman
65825956117b524ef3ef37f9
Federico Tombari
Text-Conditioned Resampler For Long Form Video Understanding
Videos are highly redundant data source and it is often enough to identify a
few key moments to solve any given task. In this paper, we present a
text-conditioned video resampler (TCR) module that uses a pre-trained and
frozen visual encoder and large language model (LLM) to process long video
sequences for a task. TCR localises relevant visual features from the video
given a text condition and provides them to a LLM to generate a text response.
Due to its lightweight design and use of cross-attention, TCR can process more
than 100 frames at a time allowing the model to use much longer chunks of video
than earlier works. We make the following contributions: (i) we design a
transformer-based sampling architecture that can process long videos
conditioned on a task, together with a training method that enables it to
bridge pre-trained visual and language models; (ii) we empirically validate its
efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art
on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks
which require longer video contexts and that can thus be used effectively for
further evaluation of long-range video models.
A text-conditioned video resampler uses a pre-trained visual encoder and large language model with cross-attention to process long video sequences efficiently, achieving state-of-the-art results on several tasks.
text-conditioned video resampler pre-trained visual encoder large language model cross-attention transformer-based sampling architecture NextQA EgoSchema EGO4D-LTA challenge long-range video models