Koala: Key frame-conditioned long video-LLM
Long video question answering is a challenging task that involves recognizing
short-term activities and reasoning about their fine-grained relationships.
State-of-the-art video Large Language Models (vLLMs) hold promise as a viable
solution due to their demonstrated emergent capabilities on new tasks. However,
despite being trained on millions of short seconds-long videos, vLLMs are
unable to understand minutes-long videos and accurately answer questions about
them. To address this limitation, we propose a lightweight and self-supervised
approach, Key frame-conditioned long video-LLM (Koala), that introduces
learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to
longer videos. Our approach introduces two new tokenizers that condition on
visual tokens computed from sparse video key frames for understanding short and
long video moments. We train our proposed approach on HowTo100M and demonstrate
its effectiveness on zero-shot long video understanding benchmarks, where it
outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across
all tasks. Surprisingly, we also empirically show that our approach not only
helps a pretrained vLLM to understand long videos but also improves its
accuracy on short-term action recognition.
6614bd09ad13fc591a5e3582
Reuben Tan
6614bd09ad13fc591a5e3583
Ximeng Sun
6614bd09ad13fc591a5e3584
Ping Hu
6614bd09ad13fc591a5e3585
Jui-hsien Wang
6614bd09ad13fc591a5e3586
Hanieh Deilamsalehy
6614bd09ad13fc591a5e3587
Bryan A. Plummer
6614bd09ad13fc591a5e3588
Bryan Russell
6614bd09ad13fc591a5e3589
Kate Saenko
Koala: Key frame-conditioned long video-LLM
Long video question answering is a challenging task that involves recognizing
short-term activities and reasoning about their fine-grained relationships.
State-of-the-art video Large Language Models (vLLMs) hold promise as a viable
solution due to their demonstrated emergent capabilities on new tasks. However,
despite being trained on millions of short seconds-long videos, vLLMs are
unable to understand minutes-long videos and accurately answer questions about
them. To address this limitation, we propose a lightweight and self-supervised
approach, Key frame-conditioned long video-LLM (Koala), that introduces
learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to
longer videos. Our approach introduces two new tokenizers that condition on
visual tokens computed from sparse video key frames for understanding short and
long video moments. We train our proposed approach on HowTo100M and demonstrate
its effectiveness on zero-shot long video understanding benchmarks, where it
outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across
all tasks. Surprisingly, we also empirically show that our approach not only
helps a pretrained vLLM to understand long videos but also improves its
accuracy on short-term action recognition.
A key frame-conditioned approach enhances pretrained video Large Language Models for long video understanding and short-term action recognition using learnable spatiotemporal queries and new tokenizers.
Large Language Models vLLMs Key frame-conditioned Koala spatiotemporal queries tokenizers HowTo100M long video understanding zero-shot short-term action recognition