title: PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
summary: Extending image-based Large Multimodal Models (LMM) to videos is challenging
  due to the inherent complexity of video data. The recent approaches extending
  image-based LMM to videos either lack the grounding capabilities (e.g.,
  VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for
  better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we
  propose Video-LLaVA, the first LMM with pixel-level grounding capability,
  integrating audio cues by transcribing them into text to enrich video-context
  understanding. Our framework uses an off-the-shelf tracker and a novel
  grounding module, enabling it to spatially and temporally localize objects in
  videos following user instructions. We evaluate Video-LLaVA using video-based
  generative and question-answering benchmarks and introduce new benchmarks
  specifically designed to measure prompt-based object grounding performance in
  videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in
  Video-ChatGPT, for video-based conversation benchmarking, ensuring
  reproducibility of results which is a concern with the proprietary nature of
  GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its
  advantages to the video domain, delivering promising gains on video-based
  conversation and grounding tasks. Project Page:
  https://github.com/mbzuai-oryx/Video-LLaVA
paper_authors._id: 655edf70b9673a4249b65f7f
paper_authors.name: Shehan Munasinghe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:14:44.171Z
paper_authors._id: 641fecb25bccaa42483db102
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg
paper_authors.fullname: Shehan Munasinghe
paper_authors.type: user
paper_authors.user: shehan97
paper_authors._id: 655edf70b9673a4249b65f80
paper_authors.name: Rusiru Thushara
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:15:04.182Z
paper_authors._id: 64fda5aeda4429f25d2b0479
paper_authors.avatarUrl: /avatars/4d8d32189b0bd5b585a5ea7c79226e3f.svg
paper_authors.fullname: Rusiru Thushara
paper_authors.type: user
paper_authors.user: thusharakart
paper_authors._id: 655edf70b9673a4249b65f81
paper_authors.name: Muhammad Maaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-24T08:45:11.563Z
paper_authors._id: 64807585856901b0edb8d68b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
paper_authors.fullname: Muhammad Maaz
paper_authors.type: user
paper_authors.user: mmaaz60
paper_authors._id: 655edf70b9673a4249b65f82
paper_authors.name: Hanoona Abdul Rasheed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T16:15:36.566Z
paper_authors._id: 64636b2551fa6e6306046293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
paper_authors.fullname: Hanoona Rasheed
paper_authors.type: user
paper_authors.user: Hanoona
paper_authors._id: 655edf70b9673a4249b65f83
paper_authors.name: Salman Khan
paper_authors._id: 655edf70b9673a4249b65f84
paper_authors.name: Mubarak Shah
paper_authors._id: 655edf70b9673a4249b65f85
paper_authors.name: Fahad Khan
paper_title: PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
paper_summary: Extending image-based Large Multimodal Models (LMM) to videos is challenging
  due to the inherent complexity of video data. The recent approaches extending
  image-based LMM to videos either lack the grounding capabilities (e.g.,
  VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for
  better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we
  propose Video-LLaVA, the first LMM with pixel-level grounding capability,
  integrating audio cues by transcribing them into text to enrich video-context
  understanding. Our framework uses an off-the-shelf tracker and a novel
  grounding module, enabling it to spatially and temporally localize objects in
  videos following user instructions. We evaluate Video-LLaVA using video-based
  generative and question-answering benchmarks and introduce new benchmarks
  specifically designed to measure prompt-based object grounding performance in
  videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in
  Video-ChatGPT, for video-based conversation benchmarking, ensuring
  reproducibility of results which is a concern with the proprietary nature of
  GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its
  advantages to the video domain, delivering promising gains on video-based
  conversation and grounding tasks. Project Page:
  https://github.com/mbzuai-oryx/Video-LLaVA
paper_ai_summary: Video-LLaVA integrates audio transcriptions and uses a novel grounding module to enhance pixel-level grounding in large multimodal video models, leading to improved performance in generative and question-answering tasks.
paper_ai_keywords: Large Multimodal Models LMM video-based LMM Video-ChatGPT Video-LLaMA pixel-level grounding audio cues text transcription object localization user instructions video-based generative benchmarks video-based question-answering benchmarks Vicuna GPT-3.5 SoTA LLaVA model