VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with
  Video LLM
Video Large Language Models (Video LLMs) have recently exhibited remarkable
capabilities in general video understanding. However, they mainly focus on
holistic comprehension and struggle with capturing fine-grained spatial and
temporal details. Besides, the lack of high-quality object-level video
instruction data and a comprehensive benchmark further hinders their
advancements. To tackle these challenges, we introduce the VideoRefer Suite to
empower Video LLM for finer-level spatial-temporal video understanding, i.e.,
enabling perception and reasoning on any objects throughout the video.
Specially, we thoroughly develop VideoRefer Suite across three essential
aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent
data engine to meticulously curate a large-scale, high-quality object-level
video instruction dataset, termed VideoRefer-700K. Next, we present the
VideoRefer model, which equips a versatile spatial-temporal object encoder to
capture precise regional and sequential representations. Finally, we
meticulously create a VideoRefer-Bench to comprehensively assess the
spatial-temporal understanding capability of a Video LLM, evaluating it across
various aspects. Extensive experiments and analyses demonstrate that our
VideoRefer model not only achieves promising performance on video referring
benchmarks but also facilitates general video understanding capabilities.
677761253c2cb54a3ac7918e
Yuqian Yuan
677761253c2cb54a3ac7918f
Hang Zhang
677761253c2cb54a3ac79190
Wentong Li
claimed_verified
2025-01-03T14:03:13.776Z
64c48a78d07620bdc99777d4
https://cdn-avatars.huggingface.co/v1/production/uploads/64c48a78d07620bdc99777d4/NJC4Ot0a7YSdU5RC6dgga.jpeg
LI WENTONG
user
sunshine-lwt
677761253c2cb54a3ac79191
Zesen Cheng
admin_assigned
2025-01-06T14:02:16.139Z
65b2529285b6c21448a10d65
/avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
Zesen Cheng
user
ClownRat
677761253c2cb54a3ac79192
Boqiang Zhang
677761253c2cb54a3ac79193
Long Li
677761253c2cb54a3ac79194
Xin Li
claimed_verified
2025-01-05T23:02:45.998Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
677761253c2cb54a3ac79195
Deli Zhao
677761253c2cb54a3ac79196
Wenqiao Zhang
677761253c2cb54a3ac79197
Yueting Zhuang
677761253c2cb54a3ac79198
Jianke Zhu
677761253c2cb54a3ac79199
Lidong Bing
claimed_verified
2025-01-05T23:02:48.302Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with
  Video LLM
Video Large Language Models (Video LLMs) have recently exhibited remarkable
capabilities in general video understanding. However, they mainly focus on
holistic comprehension and struggle with capturing fine-grained spatial and
temporal details. Besides, the lack of high-quality object-level video
instruction data and a comprehensive benchmark further hinders their
advancements. To tackle these challenges, we introduce the VideoRefer Suite to
empower Video LLM for finer-level spatial-temporal video understanding, i.e.,
enabling perception and reasoning on any objects throughout the video.
Specially, we thoroughly develop VideoRefer Suite across three essential
aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent
data engine to meticulously curate a large-scale, high-quality object-level
video instruction dataset, termed VideoRefer-700K. Next, we present the
VideoRefer model, which equips a versatile spatial-temporal object encoder to
capture precise regional and sequential representations. Finally, we
meticulously create a VideoRefer-Bench to comprehensively assess the
spatial-temporal understanding capability of a Video LLM, evaluating it across
various aspects. Extensive experiments and analyses demonstrate that our
VideoRefer model not only achieves promising performance on video referring
benchmarks but also facilitates general video understanding capabilities.
The VideoRefer Suite enhances Video LLMs for finer spatial-temporal understanding through a comprehensive dataset, model, and benchmark.
Video LLMs VideoRefer Suite VideoRefer-700K spatial-temporal object encoder VideoRefer-Bench video referring benchmarks