title: Koala: Key frame-conditioned long video-LLM
summary: Long video question answering is a challenging task that involves recognizing
  short-term activities and reasoning about their fine-grained relationships.
  State-of-the-art video Large Language Models (vLLMs) hold promise as a viable
  solution due to their demonstrated emergent capabilities on new tasks. However,
  despite being trained on millions of short seconds-long videos, vLLMs are
  unable to understand minutes-long videos and accurately answer questions about
  them. To address this limitation, we propose a lightweight and self-supervised
  approach, Key frame-conditioned long video-LLM (Koala), that introduces
  learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to
  longer videos. Our approach introduces two new tokenizers that condition on
  visual tokens computed from sparse video key frames for understanding short and
  long video moments. We train our proposed approach on HowTo100M and demonstrate
  its effectiveness on zero-shot long video understanding benchmarks, where it
  outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across
  all tasks. Surprisingly, we also empirically show that our approach not only
  helps a pretrained vLLM to understand long videos but also improves its
  accuracy on short-term action recognition.
paper_authors._id: 6614bd09ad13fc591a5e3582
paper_authors.name: Reuben Tan
paper_authors._id: 6614bd09ad13fc591a5e3583
paper_authors.name: Ximeng Sun
paper_authors._id: 6614bd09ad13fc591a5e3584
paper_authors.name: Ping Hu
paper_authors._id: 6614bd09ad13fc591a5e3585
paper_authors.name: Jui-hsien Wang
paper_authors._id: 6614bd09ad13fc591a5e3586
paper_authors.name: Hanieh Deilamsalehy
paper_authors._id: 6614bd09ad13fc591a5e3587
paper_authors.name: Bryan A. Plummer
paper_authors._id: 6614bd09ad13fc591a5e3588
paper_authors.name: Bryan Russell
paper_authors._id: 6614bd09ad13fc591a5e3589
paper_authors.name: Kate Saenko
paper_title: Koala: Key frame-conditioned long video-LLM
paper_summary: Long video question answering is a challenging task that involves recognizing
  short-term activities and reasoning about their fine-grained relationships.
  State-of-the-art video Large Language Models (vLLMs) hold promise as a viable
  solution due to their demonstrated emergent capabilities on new tasks. However,
  despite being trained on millions of short seconds-long videos, vLLMs are
  unable to understand minutes-long videos and accurately answer questions about
  them. To address this limitation, we propose a lightweight and self-supervised
  approach, Key frame-conditioned long video-LLM (Koala), that introduces
  learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to
  longer videos. Our approach introduces two new tokenizers that condition on
  visual tokens computed from sparse video key frames for understanding short and
  long video moments. We train our proposed approach on HowTo100M and demonstrate
  its effectiveness on zero-shot long video understanding benchmarks, where it
  outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across
  all tasks. Surprisingly, we also empirically show that our approach not only
  helps a pretrained vLLM to understand long videos but also improves its
  accuracy on short-term action recognition.
paper_ai_keywords: Key frame-conditioned long video-LLM (Koala) learnable spatiotemporal queries visual tokens tokenizers short and long video moments zero-shot long video understanding benchmarks