title: Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of
    Images and Videos
summary: This work presents Sa2VA, the first unified model for dense grounded
  understanding of both images and videos. Unlike existing multi-modal large
  language models, which are often limited to specific modalities and tasks,
  Sa2VA supports a wide range of image and video tasks, including referring
  segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA
  combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced
  vision-language model, and unifies text, image, and video into a shared LLM
  token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2
  in producing precise masks, enabling a grounded, multi-modal understanding of
  both static and dynamic visual content. Additionally, we introduce Ref-SAV, an
  auto-labeled dataset containing over 72k object expressions in complex video
  scenes, designed to boost model performance. We also manually validate 2k video
  objects in the Ref-SAV datasets to benchmark referring video object
  segmentation in complex environments. Experiments show that Sa2VA achieves
  state-of-the-art across multiple tasks, particularly in referring video object
  segmentation, highlighting its potential for complex real-world applications.
paper_authors._id: 677e2052dbff7b495e85ec95
paper_authors.name: Haobo Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T09:44:27.046Z
paper_authors._id: 6391e41f2e73987364e6bcb2
paper_authors.avatarUrl: /avatars/d09a9ee329bb8c3a9e2929d67d24e97d.svg
paper_authors.fullname: Haobo Yuan
paper_authors.type: user
paper_authors.user: HarborYuan
paper_authors._id: 677e2052dbff7b495e85ec96
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:39:14.906Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 677e2052dbff7b495e85ec97
paper_authors.name: Tao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:39:23.333Z
paper_authors._id: 660d28db4215cc70372bc432
paper_authors.avatarUrl: /avatars/a515303c6e29725ef3698bb695ffa743.svg
paper_authors.fullname: Tao Zhang
paper_authors.type: user
paper_authors.user: TaoZhang
paper_authors._id: 677e2052dbff7b495e85ec98
paper_authors.name: Zilong Huang
paper_authors._id: 677e2052dbff7b495e85ec99
paper_authors.name: Shilin Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:40:16.694Z
paper_authors._id: 638598a138f4aec99c50750e
paper_authors.avatarUrl: /avatars/42a4aad213e04a0ded1ab7f81910e082.svg
paper_authors.fullname: Shilin Xu
paper_authors.type: user
paper_authors.user: shilinxu
paper_authors._id: 677e2052dbff7b495e85ec9a
paper_authors.name: Shunping Ji
paper_authors._id: 677e2052dbff7b495e85ec9b
paper_authors.name: Yunhai Tong
paper_authors._id: 677e2052dbff7b495e85ec9c
paper_authors.name: Lu Qi
paper_authors._id: 677e2052dbff7b495e85ec9d
paper_authors.name: Jiashi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-08T13:41:01.815Z
paper_authors._id: 67298e44017b96a1d0101dc4
paper_authors.avatarUrl: /avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
paper_authors.fullname: Jiashi Feng
paper_authors.type: user
paper_authors.user: jshfeng
paper_authors._id: 677e2052dbff7b495e85ec9e
paper_authors.name: Ming-Hsuan Yang
paper_title: Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of
    Images and Videos
paper_summary: This work presents Sa2VA, the first unified model for dense grounded
  understanding of both images and videos. Unlike existing multi-modal large
  language models, which are often limited to specific modalities and tasks,
  Sa2VA supports a wide range of image and video tasks, including referring
  segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA
  combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced
  vision-language model, and unifies text, image, and video into a shared LLM
  token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2
  in producing precise masks, enabling a grounded, multi-modal understanding of
  both static and dynamic visual content. Additionally, we introduce Ref-SAV, an
  auto-labeled dataset containing over 72k object expressions in complex video
  scenes, designed to boost model performance. We also manually validate 2k video
  objects in the Ref-SAV datasets to benchmark referring video object
  segmentation in complex environments. Experiments show that Sa2VA achieves
  state-of-the-art across multiple tasks, particularly in referring video object
  segmentation, highlighting its potential for complex real-world applications.
paper_ai_keywords: multi-modal large language models dense understanding image tasks video tasks referring segmentation conversation one-shot instruction tuning SAM-2 LLaAVE LLM (Large Language Model) instruction tokens precise masks grounded multi-modal understanding static visual content dynamic visual content auto-labeled dataset Ref-SAV object descriptions video scenes model performance referring video object segmentation state-of-the-art