Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of
  Images and Videos
This work presents Sa2VA, the first unified model for dense grounded
understanding of both images and videos. Unlike existing multi-modal large
language models, which are often limited to specific modalities and tasks,
Sa2VA supports a wide range of image and video tasks, including referring
segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA
combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced
vision-language model, and unifies text, image, and video into a shared LLM
token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2
in producing precise masks, enabling a grounded, multi-modal understanding of
both static and dynamic visual content. Additionally, we introduce Ref-SAV, an
auto-labeled dataset containing over 72k object expressions in complex video
scenes, designed to boost model performance. We also manually validate 2k video
objects in the Ref-SAV datasets to benchmark referring video object
segmentation in complex environments. Experiments show that Sa2VA achieves
state-of-the-art across multiple tasks, particularly in referring video object
segmentation, highlighting its potential for complex real-world applications.
677e2052dbff7b495e85ec95
Haobo Yuan
claimed_verified
2025-01-08T09:44:27.046Z
6391e41f2e73987364e6bcb2
/avatars/d09a9ee329bb8c3a9e2929d67d24e97d.svg
Haobo Yuan
user
HarborYuan
677e2052dbff7b495e85ec96
Xiangtai Li
admin_assigned
2025-01-08T13:39:14.906Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
677e2052dbff7b495e85ec97
Tao Zhang
admin_assigned
2025-01-08T13:39:23.333Z
660d28db4215cc70372bc432
/avatars/a515303c6e29725ef3698bb695ffa743.svg
Tao Zhang
user
TaoZhang
677e2052dbff7b495e85ec98
Zilong Huang
claimed_verified
2025-05-14T07:37:23.266Z
63a430ec84a6a25c65bff20c
/avatars/3ab51214c491f28d8fe6b70c373d72aa.svg
Zilong Huang
user
speedinghzl
677e2052dbff7b495e85ec99
Shilin Xu
admin_assigned
2025-01-08T13:40:16.694Z
638598a138f4aec99c50750e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lRf-wLwxLGXPI7SKbuwNy.png
Shilin Xu
user
shilinxu
677e2052dbff7b495e85ec9a
Shunping Ji
677e2052dbff7b495e85ec9b
Yunhai Tong
677e2052dbff7b495e85ec9c
Lu Qi
677e2052dbff7b495e85ec9d
Jiashi Feng
admin_assigned
2025-01-08T13:41:01.815Z
67298e44017b96a1d0101dc4
/avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
Jiashi Feng
user
jshfeng
677e2052dbff7b495e85ec9e
Ming-Hsuan Yang
Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of
  Images and Videos
This work presents Sa2VA, the first unified model for dense grounded
understanding of both images and videos. Unlike existing multi-modal large
language models, which are often limited to specific modalities and tasks,
Sa2VA supports a wide range of image and video tasks, including referring
segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA
combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced
vision-language model, and unifies text, image, and video into a shared LLM
token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2
in producing precise masks, enabling a grounded, multi-modal understanding of
both static and dynamic visual content. Additionally, we introduce Ref-SAV, an
auto-labeled dataset containing over 72k object expressions in complex video
scenes, designed to boost model performance. We also manually validate 2k video
objects in the Ref-SAV datasets to benchmark referring video object
segmentation in complex environments. Experiments show that Sa2VA achieves
state-of-the-art across multiple tasks, particularly in referring video object
segmentation, highlighting its potential for complex real-world applications.
Sa2VA unifies image and video understanding by integrating SAM-2 and LLaVA into a shared token space, achieving state-of-the-art performance across multiple tasks.
SA2VA SAM-2 LLaVA dense grounded understanding referring segmentation conversation referring video object segmentation auto-labeled dataset Ref-SAV