title: VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
    Videos
summary: Fine-grained alignment between videos and text is challenging due to complex
  spatial and temporal dynamics in videos. Existing video-based Large Multimodal
  Models (LMMs) handle basic conversations but struggle with precise pixel-level
  grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
  for fine-grained pixel-level grounding in videos based on user-provided textual
  inputs. Our design seamlessly connects three key components: a Large Language
  Model, a dual vision encoder that emphasizes both spatial and temporal details,
  and a spatio-temporal decoder for accurate mask generation. This connection is
  facilitated via tunable V-L and L-V adapters that enable close Vision-Language
  (VL) alignment. The architecture is trained to synchronize both spatial and
  temporal elements of video content with textual instructions. To enable
  fine-grained grounding, we curate a multimodal dataset featuring detailed
  visually-grounded conversations using a semiautomatic annotation pipeline,
  resulting in a diverse set of 38k video-QA triplets along with 83k objects and
  671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
  Conversation Generation, Visual Grounding, and Referring Video Segmentation.
  Experimental results show that our model consistently outperforms existing
  approaches across all three tasks.
paper_authors._id: 672de757915fdfa0d046b1b3
paper_authors.name: Shehan Munasinghe
paper_authors._id: 672de757915fdfa0d046b1b4
paper_authors.name: Hanan Gani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:29:11.665Z
paper_authors._id: 64eddf17e630ae6d575f6231
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64eddf17e630ae6d575f6231/W8oofVqStLEAK1Xi4fQTh.png
paper_authors.fullname: Hanan Gani
paper_authors.type: user
paper_authors.user: hanangani
paper_authors._id: 672de757915fdfa0d046b1b5
paper_authors.name: Wenqi Zhu
paper_authors._id: 672de757915fdfa0d046b1b6
paper_authors.name: Jiale Cao
paper_authors._id: 672de757915fdfa0d046b1b7
paper_authors.name: Eric Xing
paper_authors._id: 672de757915fdfa0d046b1b8
paper_authors.name: Fahad Shahbaz Khan
paper_authors._id: 672de757915fdfa0d046b1b9
paper_authors.name: Salman Khan
paper_title: VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
    Videos
paper_summary: Fine-grained alignment between videos and text is challenging due to complex
  spatial and temporal dynamics in videos. Existing video-based Large Multimodal
  Models (LMMs) handle basic conversations but struggle with precise pixel-level
  grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
  for fine-grained pixel-level grounding in videos based on user-provided textual
  inputs. Our design seamlessly connects three key components: a Large Language
  Model, a dual vision encoder that emphasizes both spatial and temporal details,
  and a spatio-temporal decoder for accurate mask generation. This connection is
  facilitated via tunable V-L and L-V adapters that enable close Vision-Language
  (VL) alignment. The architecture is trained to synchronize both spatial and
  temporal elements of video content with textual instructions. To enable
  fine-grained grounding, we curate a multimodal dataset featuring detailed
  visually-grounded conversations using a semiautomatic annotation pipeline,
  resulting in a diverse set of 38k video-QA triplets along with 83k objects and
  671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
  Conversation Generation, Visual Grounding, and Referring Video Segmentation.
  Experimental results show that our model consistently outperforms existing
  approaches across all three tasks.
paper_ai_summary: VideoGLaMM, a Large Multimodal Model, achieves precise pixel-level grounding in videos using a dual vision encoder, spatio-temporal decoder, and tunable Visual-Language adapters, outperforming existing approaches on various multimodal tasks.
paper_ai_keywords: Large Multimodal Models Large Language Model dual vision encoder spatio-temporal decoder V-L and L-V adapters Vision-Language alignment semiautomatic annotation pipeline Grounded Conversation Generation Visual Grounding Referring Video Segmentation