title: Fostering Video Reasoning via Next-Event Prediction
summary: Next-token prediction serves as the foundational learning task enabling
  reasoning in LLMs. But what should the learning task be when aiming to equip
  MLLMs with temporal reasoning capabilities over video inputs? Existing tasks
  such as video question answering often rely on annotations from humans or much
  stronger MLLMs, while video captioning tends to entangle temporal reasoning
  with spatial information. To address this gap, we propose next-event prediction
  (NEP), a learning task that harnesses future video segments as a rich,
  self-supervised signal to foster temporal reasoning. We segment each video into
  past and future frames: the MLLM takes the past frames as input and predicts a
  summary of events derived from the future frames, thereby encouraging the model
  to reason temporally in order to complete the task. To support this task, we
  curate V1-33K, a dataset comprising 33,000 automatically extracted video
  segments spanning diverse real-world scenarios. We further explore a range of
  video instruction-tuning strategies to study their effects on temporal
  reasoning. To evaluate progress, we introduce FutureBench to assess coherence
  in predicting unseen future events. Experiments validate that NEP offers a
  scalable and effective training paradigm for fostering temporal reasoning in
  MLLMs.
paper_authors._id: 68380912e9c1608de91e23f3
paper_authors.name: Haonan Wang
paper_authors._id: 68380912e9c1608de91e23f4
paper_authors.name: Hongfu Liu
paper_authors._id: 68380912e9c1608de91e23f5
paper_authors.name: Xiangyan Liu
paper_authors._id: 68380912e9c1608de91e23f6
paper_authors.name: Chao Du
paper_authors._id: 68380912e9c1608de91e23f7
paper_authors.name: Kenji Kawaguchi
paper_authors._id: 68380912e9c1608de91e23f8
paper_authors.name: Ye Wang
paper_authors._id: 68380912e9c1608de91e23f9
paper_authors.name: Tianyu Pang
paper_title: Fostering Video Reasoning via Next-Event Prediction
paper_summary: Next-token prediction serves as the foundational learning task enabling
  reasoning in LLMs. But what should the learning task be when aiming to equip
  MLLMs with temporal reasoning capabilities over video inputs? Existing tasks
  such as video question answering often rely on annotations from humans or much
  stronger MLLMs, while video captioning tends to entangle temporal reasoning
  with spatial information. To address this gap, we propose next-event prediction
  (NEP), a learning task that harnesses future video segments as a rich,
  self-supervised signal to foster temporal reasoning. We segment each video into
  past and future frames: the MLLM takes the past frames as input and predicts a
  summary of events derived from the future frames, thereby encouraging the model
  to reason temporally in order to complete the task. To support this task, we
  curate V1-33K, a dataset comprising 33,000 automatically extracted video
  segments spanning diverse real-world scenarios. We further explore a range of
  video instruction-tuning strategies to study their effects on temporal
  reasoning. To evaluate progress, we introduce FutureBench to assess coherence
  in predicting unseen future events. Experiments validate that NEP offers a
  scalable and effective training paradigm for fostering temporal reasoning in
  MLLMs.
paper_ai_summary: Next-event prediction (NEP) is proposed as a learning task to enable MLLMs to reason temporally over video inputs, using future video segments as a self-supervised signal.
paper_ai_keywords: next-token prediction next-event prediction (NEP) LLMs MLLMs video question answering video captioning temporal reasoning future video segments past frames video segments V1-33K video instruction-tuning strategies FutureBench