title: Goldfish: Vision-Language Understanding of Arbitrarily Long Videos
summary: Most current LLM-based models for video understanding can process videos
  within minutes. However, they struggle with lengthy videos due to challenges
  such as "noise and redundancy", as well as "memory and computation"
  constraints. In this paper, we present Goldfish, a methodology tailored for
  comprehending videos of arbitrary lengths. We also introduce the TVQA-long
  benchmark, specifically designed to evaluate models' capabilities in
  understanding long videos with questions in both vision and text content.
  Goldfish approaches these challenges with an efficient retrieval mechanism that
  initially gathers the top-k video clips relevant to the instruction before
  proceeding to provide the desired response. This design of the retrieval
  mechanism enables the Goldfish to efficiently process arbitrarily long video
  sequences, facilitating its application in contexts such as movies or
  television series. To facilitate the retrieval process, we developed
  MiniGPT4-Video that generates detailed descriptions for the video clips. In
  addressing the scarcity of benchmarks for long video evaluation, we adapted the
  TVQA short video benchmark for extended content analysis by aggregating
  questions from entire episodes, thereby shifting the evaluation from partial to
  full episode comprehension. We attained a 41.78% accuracy rate on the TVQA-long
  benchmark, surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows
  exceptional performance in short video comprehension, exceeding existing
  state-of-the-art methods by 3.23%, 2.03%, 16.5% and 23.59% on the MSVD, MSRVTT,
  TGIF, and TVQA short video benchmarks, respectively. These results indicate
  that our models have significant improvements in both long and short-video
  understanding. Our models and code have been made publicly available at
  https://vision-cair.github.io/Goldfish_website/
paper_authors._id: 669881c42f9cd07f44a94334
paper_authors.name: Kirolos Ataallah
paper_authors._id: 669881c42f9cd07f44a94335
paper_authors.name: Xiaoqian Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:44.007Z
paper_authors._id: 620c5b20c33ca69c4c82145b
paper_authors.avatarUrl: /avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
paper_authors.fullname: Xiaoqian Shen
paper_authors.type: user
paper_authors.user: shenxq
paper_authors._id: 669881c42f9cd07f44a94336
paper_authors.name: Eslam Abdelrahman
paper_authors._id: 669881c42f9cd07f44a94337
paper_authors.name: Essam Sleiman
paper_authors._id: 669881c42f9cd07f44a94338
paper_authors.name: Mingchen Zhuge
paper_authors._id: 669881c42f9cd07f44a94339
paper_authors.name: Jian Ding
paper_authors._id: 669881c42f9cd07f44a9433a
paper_authors.name: Deyao Zhu
paper_authors._id: 669881c42f9cd07f44a9433b
paper_authors.name: Jürgen Schmidhuber
paper_authors._id: 669881c42f9cd07f44a9433c
paper_authors.name: Mohamed Elhoseiny
paper_title: Goldfish: Vision-Language Understanding of Arbitrarily Long Videos
paper_summary: Most current LLM-based models for video understanding can process videos
  within minutes. However, they struggle with lengthy videos due to challenges
  such as "noise and redundancy", as well as "memory and computation"
  constraints. In this paper, we present Goldfish, a methodology tailored for
  comprehending videos of arbitrary lengths. We also introduce the TVQA-long
  benchmark, specifically designed to evaluate models' capabilities in
  understanding long videos with questions in both vision and text content.
  Goldfish approaches these challenges with an efficient retrieval mechanism that
  initially gathers the top-k video clips relevant to the instruction before
  proceeding to provide the desired response. This design of the retrieval
  mechanism enables the Goldfish to efficiently process arbitrarily long video
  sequences, facilitating its application in contexts such as movies or
  television series. To facilitate the retrieval process, we developed
  MiniGPT4-Video that generates detailed descriptions for the video clips. In
  addressing the scarcity of benchmarks for long video evaluation, we adapted the
  TVQA short video benchmark for extended content analysis by aggregating
  questions from entire episodes, thereby shifting the evaluation from partial to
  full episode comprehension. We attained a 41.78% accuracy rate on the TVQA-long
  benchmark, surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows
  exceptional performance in short video comprehension, exceeding existing
  state-of-the-art methods by 3.23%, 2.03%, 16.5% and 23.59% on the MSVD, MSRVTT,
  TGIF, and TVQA short video benchmarks, respectively. These results indicate
  that our models have significant improvements in both long and short-video
  understanding. Our models and code have been made publicly available at
  https://vision-cair.github.io/Goldfish_website/
paper_ai_summary: Goldfish introduces an efficient retrieval mechanism and MiniGPT4-Video for accurate and efficient understanding of both long and short videos.
paper_ai_keywords: LLM-based models video understanding noisy and redundant videos memory and computation constraints retrieval mechanism MiniGPT4-Video TVQA-long benchmark TVQA short video benchmark full episode comprehension short video comprehension