OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video
  Understanding?
Temporal Awareness, the ability to reason dynamically based on the timestamp
when a question is raised, is the key distinction between offline and online
video LLMs. Unlike offline models, which rely on complete videos for static,
post hoc analysis, online models process video streams incrementally and
dynamically adapt their responses based on the timestamp at which the question
is posed. Despite its significance, temporal awareness has not been adequately
evaluated in existing benchmarks. To fill this gap, we present OVO-Bench
(Online-VideO-Benchmark), a novel video benchmark that emphasizes the
importance of timestamps for advanced online video understanding capability
benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and
respond to events occurring at specific timestamps under three distinct
scenarios: (1) Backward tracing: trace back to past events to answer the
question. (2) Real-time understanding: understand and respond to events as they
unfold at the current timestamp. (3) Forward active responding: delay the
response until sufficient future information becomes available to answer the
question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos
and approximately human-curated 2,800 fine-grained meta-annotations with
precise timestamps. We combine automated generation pipelines with human
curation. With these high-quality samples, we further developed an evaluation
pipeline to systematically query video LLMs along the video timeline.
Evaluations of nine Video-LLMs reveal that, despite advancements on traditional
benchmarks, current models struggle with online video understanding, showing a
significant gap compared to human agents. We hope OVO-Bench will drive progress
in video LLMs and inspire future research in online video reasoning. Our
benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.
6784bf46dd05c40289463293
Yifei Li
claimed_verified
2025-01-13T08:54:23.749Z
64adfeac4beffa272dfaef21
/avatars/883f6ba38b993476115dfafcef9ce3c1.svg
Yifei Li
user
JoeLeelyf
6784bf46dd05c40289463294
Junbo Niu
admin_assigned
2025-01-13T09:18:32.157Z
650abbb71aece923f21d87fc
/avatars/f09ff031c278bc42bfd7a563853e142c.svg
Junbo Niu
user
Niujunbo2002
6784bf46dd05c40289463295
Ziyang Miao
6784bf46dd05c40289463296
Chunjiang Ge
admin_assigned
2025-01-13T09:19:10.355Z
645e054ff7a55f0d780a8ff7
/avatars/9614510443bee3bd5d6266efd1c39fc1.svg
Chunjiang Ge
user
HelloJiang
6784bf46dd05c40289463297
Yuanhang Zhou
admin_assigned
2025-01-13T09:19:15.974Z
5f0fb4827a6784092812b6eb
https://cdn-avatars.huggingface.co/v1/production/uploads/1594864762501-noauth.jpeg
Yuanhang Zhou
user
zyh
6784bf46dd05c40289463298
Qihao He
admin_assigned
2025-01-13T09:19:22.762Z
6784c961dd05c40289493dee
/avatars/41b94d1b3f51e4bb070f291f80fcafd8.svg
Qihao He
user
qiaozc
6784bf46dd05c40289463299
Xiaoyi Dong
6784bf46dd05c4028946329a
Haodong Duan
admin_assigned
2025-01-13T09:19:40.474Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
6784bf46dd05c4028946329b
Shuangrui Ding
admin_assigned
2025-01-13T09:19:46.093Z
65a7c0335e79abfa2ec30c52
/avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
Shuangrui Ding
user
Mar2Ding
6784bf46dd05c4028946329c
Rui Qian
6784bf46dd05c4028946329d
Pan Zhang
6784bf46dd05c4028946329e
Yuhang Zang
claimed_verified
2025-01-13T08:54:27.776Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
6784bf46dd05c4028946329f
Yuhang Cao
admin_assigned
2025-01-13T09:20:37.666Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
6784bf46dd05c402894632a0
Conghui He
admin_assigned
2025-01-13T09:20:44.739Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
6784bf46dd05c402894632a1
Jiaqi Wang
claimed_verified
2025-01-13T08:54:25.642Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video
  Understanding?
Temporal Awareness, the ability to reason dynamically based on the timestamp
when a question is raised, is the key distinction between offline and online
video LLMs. Unlike offline models, which rely on complete videos for static,
post hoc analysis, online models process video streams incrementally and
dynamically adapt their responses based on the timestamp at which the question
is posed. Despite its significance, temporal awareness has not been adequately
evaluated in existing benchmarks. To fill this gap, we present OVO-Bench
(Online-VideO-Benchmark), a novel video benchmark that emphasizes the
importance of timestamps for advanced online video understanding capability
benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and
respond to events occurring at specific timestamps under three distinct
scenarios: (1) Backward tracing: trace back to past events to answer the
question. (2) Real-time understanding: understand and respond to events as they
unfold at the current timestamp. (3) Forward active responding: delay the
response until sufficient future information becomes available to answer the
question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos
and approximately human-curated 2,800 fine-grained meta-annotations with
precise timestamps. We combine automated generation pipelines with human
curation. With these high-quality samples, we further developed an evaluation
pipeline to systematically query video LLMs along the video timeline.
Evaluations of nine Video-LLMs reveal that, despite advancements on traditional
benchmarks, current models struggle with online video understanding, showing a
significant gap compared to human agents. We hope OVO-Bench will drive progress
in video LLMs and inspire future research in online video reasoning. Our
benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.
OVO-Bench is a new benchmark for evaluating the temporal awareness of online video language models across three scenarios: backward tracing, real-time understanding, and forward active responding.
temporal awareness online video LLMs offline models video streams timestamps OVO-Bench Online-VideO-Benchmark backward tracing real-time understanding forward active responding video LLMs automated generation pipelines human curation evaluation pipeline video timeline