title: MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with
    Interleaved Visual-Textual Tokens
summary: This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM)
  designed specifically for video understanding. The model is capable of
  processing both temporal visual and textual data, making it adept at
  understanding the complexities of videos. Building upon the success of
  MiniGPT-v2, which excelled in translating visual features into the LLM space
  for single images and achieved impressive results on various image-text
  benchmarks, this paper extends the model's capabilities to process a sequence
  of frames, enabling it to comprehend videos. MiniGPT4-video does not only
  consider visual content but also incorporates textual conversations, allowing
  the model to effectively answer queries involving both visual and text
  components. The proposed model outperforms existing state-of-the-art methods,
  registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF,
  and TVQA benchmarks respectively. Our models and code have been made publicly
  available here https://vision-cair.github.io/MiniGPT4-video/
paper_authors._id: 660f6878ccc61eea055c67ee
paper_authors.name: Kirolos Ataallah
paper_authors._id: 660f6878ccc61eea055c67ef
paper_authors.name: Xiaoqian Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:59.671Z
paper_authors._id: 620c5b20c33ca69c4c82145b
paper_authors.avatarUrl: /avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
paper_authors.fullname: Xiaoqian Shen
paper_authors.type: user
paper_authors.user: shenxq
paper_authors._id: 660f6878ccc61eea055c67f0
paper_authors.name: Eslam Abdelrahman
paper_authors._id: 660f6878ccc61eea055c67f1
paper_authors.name: Essam Sleiman
paper_authors._id: 660f6878ccc61eea055c67f2
paper_authors.name: Deyao Zhu
paper_authors._id: 660f6878ccc61eea055c67f3
paper_authors.name: Jian Ding
paper_authors._id: 660f6878ccc61eea055c67f4
paper_authors.name: Mohamed Elhoseiny
paper_title: MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with
    Interleaved Visual-Textual Tokens
paper_summary: This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM)
  designed specifically for video understanding. The model is capable of
  processing both temporal visual and textual data, making it adept at
  understanding the complexities of videos. Building upon the success of
  MiniGPT-v2, which excelled in translating visual features into the LLM space
  for single images and achieved impressive results on various image-text
  benchmarks, this paper extends the model's capabilities to process a sequence
  of frames, enabling it to comprehend videos. MiniGPT4-video does not only
  consider visual content but also incorporates textual conversations, allowing
  the model to effectively answer queries involving both visual and text
  components. The proposed model outperforms existing state-of-the-art methods,
  registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF,
  and TVQA benchmarks respectively. Our models and code have been made publicly
  available here https://vision-cair.github.io/MiniGPT4-video/
paper_ai_keywords: multimodal Large Language Model (LLM) video understanding temporal visual and textual data MiniGPT-v2 translating visual features sequence of frames visual content textual conversations MSVD MSRVTT TGIF TVQA