title: One Token to Seg Them All: Language Instructed Reasoning Segmentation in
    Videos
summary: We introduce VideoLISA, a video-based multimodal large language model
  designed to tackle the problem of language-instructed reasoning segmentation in
  videos. Leveraging the reasoning capabilities and world knowledge of large
  language models, and augmented by the Segment Anything Model, VideoLISA
  generates temporally consistent segmentation masks in videos based on language
  instructions. Existing image-based methods, such as LISA, struggle with video
  tasks due to the additional temporal dimension, which requires temporal dynamic
  understanding and consistent segmentation across frames. VideoLISA addresses
  these challenges by integrating a Sparse Dense Sampling strategy into the
  video-LLM, which balances temporal context and spatial detail within
  computational constraints. Additionally, we propose a One-Token-Seg-All
  approach using a specially designed <TRK> token, enabling the model to segment
  and track objects across multiple frames. Extensive evaluations on diverse
  benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate
  VideoLISA's superior performance in video object segmentation tasks involving
  complex reasoning, temporal understanding, and object tracking. While optimized
  for videos, VideoLISA also shows promising generalization to image
  segmentation, revealing its potential as a unified foundation model for
  language-instructed object segmentation. Code and model will be available at:
  https://github.com/showlab/VideoLISA.
paper_authors._id: 66fcf944c0a62736ec4530e6
paper_authors.name: Zechen Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:41:34.135Z
paper_authors._id: 64b7833aa5018e3c7c9b50d8
paper_authors.avatarUrl: /avatars/782415605ed786b73f484fcc86a6384f.svg
paper_authors.fullname: Zechen Bai
paper_authors.type: user
paper_authors.user: ZechenBai
paper_authors._id: 66fcf944c0a62736ec4530e7
paper_authors.name: Tong He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:41:41.962Z
paper_authors._id: 64478c64e2148488340229db
paper_authors.avatarUrl: /avatars/f5c23489a068e896381cdc25836ce3dd.svg
paper_authors.fullname: he
paper_authors.type: user
paper_authors.user: tonghe
paper_authors._id: 66fcf944c0a62736ec4530e8
paper_authors.name: Haiyang Mei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:41:48.251Z
paper_authors._id: 66fcfa6e05638227c44233a9
paper_authors.avatarUrl: /avatars/4a88765c7f5c5ca77da6d21eb01f73e0.svg
paper_authors.fullname: Haiyang Mei
paper_authors.type: user
paper_authors.user: meihaiyang
paper_authors._id: 66fcf944c0a62736ec4530e9
paper_authors.name: Pichao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:41:54.212Z
paper_authors._id: 65381380c104b8d4556301c3
paper_authors.avatarUrl: /avatars/3b6e1283cba0e6a9147f7cfcebc01a0b.svg
paper_authors.fullname: Pichao Wang
paper_authors.type: user
paper_authors.user: wangpichao
paper_authors._id: 66fcf944c0a62736ec4530ea
paper_authors.name: Ziteng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:42:01.234Z
paper_authors._id: 648ac3d53470b17ccc90deaf
paper_authors.avatarUrl: /avatars/35b678baa7a91959f40f704c5de2a3e1.svg
paper_authors.fullname: Ziteng Gao
paper_authors.type: user
paper_authors.user: sebgao
paper_authors._id: 66fcf944c0a62736ec4530eb
paper_authors.name: Joya Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:42:07.352Z
paper_authors._id: 642435a1a3adbc7142c3b0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/EBmQ7LnfdTdyuhSUti0-d.png
paper_authors.fullname: Joya Chen
paper_authors.type: user
paper_authors.user: chenjoya
paper_authors._id: 66fcf944c0a62736ec4530ec
paper_authors.name: Lei Liu
paper_authors._id: 66fcf944c0a62736ec4530ed
paper_authors.name: Zheng Zhang
paper_authors._id: 66fcf944c0a62736ec4530ee
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:43:03.311Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: One Token to Seg Them All: Language Instructed Reasoning Segmentation in
    Videos
paper_summary: We introduce VideoLISA, a video-based multimodal large language model
  designed to tackle the problem of language-instructed reasoning segmentation in
  videos. Leveraging the reasoning capabilities and world knowledge of large
  language models, and augmented by the Segment Anything Model, VideoLISA
  generates temporally consistent segmentation masks in videos based on language
  instructions. Existing image-based methods, such as LISA, struggle with video
  tasks due to the additional temporal dimension, which requires temporal dynamic
  understanding and consistent segmentation across frames. VideoLISA addresses
  these challenges by integrating a Sparse Dense Sampling strategy into the
  video-LLM, which balances temporal context and spatial detail within
  computational constraints. Additionally, we propose a One-Token-Seg-All
  approach using a specially designed <TRK> token, enabling the model to segment
  and track objects across multiple frames. Extensive evaluations on diverse
  benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate
  VideoLISA's superior performance in video object segmentation tasks involving
  complex reasoning, temporal understanding, and object tracking. While optimized
  for videos, VideoLISA also shows promising generalization to image
  segmentation, revealing its potential as a unified foundation model for
  language-instructed object segmentation. Code and model will be available at:
  https://github.com/showlab/VideoLISA.
paper_ai_keywords: VideoLISA language-instructed reasoning segmentation large language model Segment Anything Model temporally consistent segmentation masks image-based methods LISA video tasks temporal dimension temporal dynamic understanding Sparse Dense Sampling strategy video-LLM One-Token-Seg-All <TRK> token object segmentation tasks complex reasoning temporal understanding object tracking ReasonVOS benchmark