title: OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video
    Understanding?
summary: Temporal Awareness, the ability to reason dynamically based on the timestamp
  when a question is raised, is the key distinction between offline and online
  video LLMs. Unlike offline models, which rely on complete videos for static,
  post hoc analysis, online models process video streams incrementally and
  dynamically adapt their responses based on the timestamp at which the question
  is posed. Despite its significance, temporal awareness has not been adequately
  evaluated in existing benchmarks. To fill this gap, we present OVO-Bench
  (Online-VideO-Benchmark), a novel video benchmark that emphasizes the
  importance of timestamps for advanced online video understanding capability
  benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and
  respond to events occurring at specific timestamps under three distinct
  scenarios: (1) Backward tracing: trace back to past events to answer the
  question. (2) Real-time understanding: understand and respond to events as they
  unfold at the current timestamp. (3) Forward active responding: delay the
  response until sufficient future information becomes available to answer the
  question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos
  and approximately human-curated 2,800 fine-grained meta-annotations with
  precise timestamps. We combine automated generation pipelines with human
  curation. With these high-quality samples, we further developed an evaluation
  pipeline to systematically query video LLMs along the video timeline.
  Evaluations of nine Video-LLMs reveal that, despite advancements on traditional
  benchmarks, current models struggle with online video understanding, showing a
  significant gap compared to human agents. We hope OVO-Bench will drive progress
  in video LLMs and inspire future research in online video reasoning. Our
  benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.
paper_authors._id: 6784bf46dd05c40289463293
paper_authors.name: Yifei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:23.749Z
paper_authors._id: 64adfeac4beffa272dfaef21
paper_authors.avatarUrl: /avatars/883f6ba38b993476115dfafcef9ce3c1.svg
paper_authors.fullname: Yifei Li
paper_authors.type: user
paper_authors.user: JoeLeelyf
paper_authors._id: 6784bf46dd05c40289463294
paper_authors.name: Junbo Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:18:32.157Z
paper_authors._id: 650abbb71aece923f21d87fc
paper_authors.avatarUrl: /avatars/f09ff031c278bc42bfd7a563853e142c.svg
paper_authors.fullname: Junbo Niu
paper_authors.type: user
paper_authors.user: Niujunbo2002
paper_authors._id: 6784bf46dd05c40289463295
paper_authors.name: Ziyang Miao
paper_authors._id: 6784bf46dd05c40289463296
paper_authors.name: Chunjiang Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:19:10.355Z
paper_authors._id: 645e054ff7a55f0d780a8ff7
paper_authors.avatarUrl: /avatars/9614510443bee3bd5d6266efd1c39fc1.svg
paper_authors.fullname: Chunjiang Ge
paper_authors.type: user
paper_authors.user: HelloJiang
paper_authors._id: 6784bf46dd05c40289463297
paper_authors.name: Yuanhang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:19:15.974Z
paper_authors._id: 5f0fb4827a6784092812b6eb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594864762501-noauth.jpeg
paper_authors.fullname: Yuanhang Zhou
paper_authors.type: user
paper_authors.user: zyh
paper_authors._id: 6784bf46dd05c40289463298
paper_authors.name: Qihao He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:19:22.762Z
paper_authors._id: 6784c961dd05c40289493dee
paper_authors.avatarUrl: /avatars/41b94d1b3f51e4bb070f291f80fcafd8.svg
paper_authors.fullname: Qihao He
paper_authors.type: user
paper_authors.user: qiaozc
paper_authors._id: 6784bf46dd05c40289463299
paper_authors.name: Xiaoyi Dong
paper_authors._id: 6784bf46dd05c4028946329a
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:19:40.474Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 6784bf46dd05c4028946329b
paper_authors.name: Shuangrui Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:19:46.093Z
paper_authors._id: 65a7c0335e79abfa2ec30c52
paper_authors.avatarUrl: /avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
paper_authors.fullname: Shuangrui Ding
paper_authors.type: user
paper_authors.user: Mar2Ding
paper_authors._id: 6784bf46dd05c4028946329c
paper_authors.name: Rui Qian
paper_authors._id: 6784bf46dd05c4028946329d
paper_authors.name: Pan Zhang
paper_authors._id: 6784bf46dd05c4028946329e
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:27.776Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 6784bf46dd05c4028946329f
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:20:37.666Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 6784bf46dd05c402894632a0
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:20:44.739Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 6784bf46dd05c402894632a1
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:25.642Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video
    Understanding?
paper_summary: Temporal Awareness, the ability to reason dynamically based on the timestamp
  when a question is raised, is the key distinction between offline and online
  video LLMs. Unlike offline models, which rely on complete videos for static,
  post hoc analysis, online models process video streams incrementally and
  dynamically adapt their responses based on the timestamp at which the question
  is posed. Despite its significance, temporal awareness has not been adequately
  evaluated in existing benchmarks. To fill this gap, we present OVO-Bench
  (Online-VideO-Benchmark), a novel video benchmark that emphasizes the
  importance of timestamps for advanced online video understanding capability
  benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and
  respond to events occurring at specific timestamps under three distinct
  scenarios: (1) Backward tracing: trace back to past events to answer the
  question. (2) Real-time understanding: understand and respond to events as they
  unfold at the current timestamp. (3) Forward active responding: delay the
  response until sufficient future information becomes available to answer the
  question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos
  and approximately human-curated 2,800 fine-grained meta-annotations with
  precise timestamps. We combine automated generation pipelines with human
  curation. With these high-quality samples, we further developed an evaluation
  pipeline to systematically query video LLMs along the video timeline.
  Evaluations of nine Video-LLMs reveal that, despite advancements on traditional
  benchmarks, current models struggle with online video understanding, showing a
  significant gap compared to human agents. We hope OVO-Bench will drive progress
  in video LLMs and inspire future research in online video reasoning. Our
  benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.
paper_ai_keywords: temporal awareness offline video LLMs online video LLMs video streams dynamic adaptation timestamps OVO-Bench Online-VideO-Benchmark backward tracing real-time understanding forward active responding video LLMs video timeline online video understanding human-curated fine-grained meta-annotations evaluation pipeline