title: VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with
    Video LLM
summary: Video Large Language Models (Video LLMs) have recently exhibited remarkable
  capabilities in general video understanding. However, they mainly focus on
  holistic comprehension and struggle with capturing fine-grained spatial and
  temporal details. Besides, the lack of high-quality object-level video
  instruction data and a comprehensive benchmark further hinders their
  advancements. To tackle these challenges, we introduce the VideoRefer Suite to
  empower Video LLM for finer-level spatial-temporal video understanding, i.e.,
  enabling perception and reasoning on any objects throughout the video.
  Specially, we thoroughly develop VideoRefer Suite across three essential
  aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent
  data engine to meticulously curate a large-scale, high-quality object-level
  video instruction dataset, termed VideoRefer-700K. Next, we present the
  VideoRefer model, which equips a versatile spatial-temporal object encoder to
  capture precise regional and sequential representations. Finally, we
  meticulously create a VideoRefer-Bench to comprehensively assess the
  spatial-temporal understanding capability of a Video LLM, evaluating it across
  various aspects. Extensive experiments and analyses demonstrate that our
  VideoRefer model not only achieves promising performance on video referring
  benchmarks but also facilitates general video understanding capabilities.
paper_authors._id: 677761253c2cb54a3ac7918e
paper_authors.name: Yuqian Yuan
paper_authors._id: 677761253c2cb54a3ac7918f
paper_authors.name: Hang Zhang
paper_authors._id: 677761253c2cb54a3ac79190
paper_authors.name: Wentong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:13.776Z
paper_authors._id: 64c48a78d07620bdc99777d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c48a78d07620bdc99777d4/NJC4Ot0a7YSdU5RC6dgga.jpeg
paper_authors.fullname: LI WENTONG
paper_authors.type: user
paper_authors.user: sunshine-lwt
paper_authors._id: 677761253c2cb54a3ac79191
paper_authors.name: Zesen Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:02:16.139Z
paper_authors._id: 65b2529285b6c21448a10d65
paper_authors.avatarUrl: /avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
paper_authors.fullname: Zesen Cheng
paper_authors.type: user
paper_authors.user: ClownRat
paper_authors._id: 677761253c2cb54a3ac79192
paper_authors.name: Boqiang Zhang
paper_authors._id: 677761253c2cb54a3ac79193
paper_authors.name: Long Li
paper_authors._id: 677761253c2cb54a3ac79194
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:02:45.998Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 677761253c2cb54a3ac79195
paper_authors.name: Deli Zhao
paper_authors._id: 677761253c2cb54a3ac79196
paper_authors.name: Wenqiao Zhang
paper_authors._id: 677761253c2cb54a3ac79197
paper_authors.name: Yueting Zhuang
paper_authors._id: 677761253c2cb54a3ac79198
paper_authors.name: Jianke Zhu
paper_authors._id: 677761253c2cb54a3ac79199
paper_authors.name: Lidong Bing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:02:48.302Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with
    Video LLM
paper_summary: Video Large Language Models (Video LLMs) have recently exhibited remarkable
  capabilities in general video understanding. However, they mainly focus on
  holistic comprehension and struggle with capturing fine-grained spatial and
  temporal details. Besides, the lack of high-quality object-level video
  instruction data and a comprehensive benchmark further hinders their
  advancements. To tackle these challenges, we introduce the VideoRefer Suite to
  empower Video LLM for finer-level spatial-temporal video understanding, i.e.,
  enabling perception and reasoning on any objects throughout the video.
  Specially, we thoroughly develop VideoRefer Suite across three essential
  aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent
  data engine to meticulously curate a large-scale, high-quality object-level
  video instruction dataset, termed VideoRefer-700K. Next, we present the
  VideoRefer model, which equips a versatile spatial-temporal object encoder to
  capture precise regional and sequential representations. Finally, we
  meticulously create a VideoRefer-Bench to comprehensively assess the
  spatial-temporal understanding capability of a Video LLM, evaluating it across
  various aspects. Extensive experiments and analyses demonstrate that our
  VideoRefer model not only achieves promising performance on video referring
  benchmarks but also facilitates general video understanding capabilities.
paper_ai_keywords: Video Large Language Models (Video LLMs) VideoRefer Suite fine-grained spatial and temporal details object-level video instruction data benchmark VideoRefer-700K VideoRefer model versatile spatial-temporal object encoder regional and sequential representations VideoRefer-Bench spatial-temporal understanding capability video referring benchmarks