title: Temporal Reasoning Transfer from Text to Video
summary: Video Large Language Models (Video LLMs) have shown promising capabilities in
  video comprehension, yet they struggle with tracking temporal changes and
  reasoning about temporal relationships. While previous research attributed this
  limitation to the ineffective temporal encoding of visual inputs, our
  diagnostic study reveals that video representations contain sufficient
  information for even small probing classifiers to achieve perfect accuracy.
  Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning
  capability stems from the underlying LLM's inherent difficulty with temporal
  concepts, as evidenced by poor performance on textual temporal
  question-answering tasks. Building on this discovery, we introduce the Textual
  Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning
  tasks in pure text format from existing image-text datasets, addressing the
  scarcity of video samples with complex temporal scenarios. Remarkably, without
  using any video data, T3 enhances LongVA-7B's temporal understanding, yielding
  a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark,
  which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video
  samples. Additionally, the enhanced LongVA-7B model achieves competitive
  performance on comprehensive video benchmarks. For example, it achieves a 49.7
  accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful
  large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further
  analysis reveals a strong correlation between textual and video temporal task
  performance, validating the efficacy of transferring temporal reasoning
  abilities from text to video domains.
paper_authors._id: 6707388ec818ba2f6579eb78
paper_authors.name: Lei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:11:14.571Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 6707388ec818ba2f6579eb79
paper_authors.name: Yuanxin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:01:34.232Z
paper_authors._id: 6489761dcaea79f577897f98
paper_authors.avatarUrl: /avatars/8f56dc9c08dc2b672555602d68509a03.svg
paper_authors.fullname: Yuanxin Liu
paper_authors.type: user
paper_authors.user: lyx97
paper_authors._id: 6707388ec818ba2f6579eb7a
paper_authors.name: Linli Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:11:21.755Z
paper_authors._id: 655ca347f426a304c6b393a1
paper_authors.avatarUrl: /avatars/67f0310d59c5912d38c2ad8e6448614d.svg
paper_authors.fullname: Linli Yao
paper_authors.type: user
paper_authors.user: yaolily
paper_authors._id: 6707388ec818ba2f6579eb7b
paper_authors.name: Peiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:11:32.678Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 6707388ec818ba2f6579eb7c
paper_authors.name: Chenxin An
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:11:37.979Z
paper_authors._id: 64acb321264bbbf171a2b040
paper_authors.avatarUrl: /avatars/0ad344c0e9b1e3fda469932f91d117dc.svg
paper_authors.fullname: Chenxin An
paper_authors.type: user
paper_authors.user: Chancy
paper_authors._id: 6707388ec818ba2f6579eb7d
paper_authors.name: Lean Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:11:43.776Z
paper_authors._id: 650c509472afb1e60e6151ae
paper_authors.avatarUrl: /avatars/c16ab5053a586819dc2b965303215ff7.svg
paper_authors.fullname: Lean Wang
paper_authors.type: user
paper_authors.user: AdaHousman
paper_authors._id: 6707388ec818ba2f6579eb7e
paper_authors.name: Xu Sun
paper_authors._id: 6707388ec818ba2f6579eb7f
paper_authors.name: Lingpeng Kong
paper_authors._id: 6707388ec818ba2f6579eb80
paper_authors.name: Qi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:12:34.540Z
paper_authors._id: 63ad3de96ee60ca58a409280
paper_authors.avatarUrl: /avatars/7461f4fda3692f042e556d2a7c339bc0.svg
paper_authors.fullname: Qi Liu
paper_authors.type: user
paper_authors.user: QiLiuHKU
paper_title: Temporal Reasoning Transfer from Text to Video
paper_summary: Video Large Language Models (Video LLMs) have shown promising capabilities in
  video comprehension, yet they struggle with tracking temporal changes and
  reasoning about temporal relationships. While previous research attributed this
  limitation to the ineffective temporal encoding of visual inputs, our
  diagnostic study reveals that video representations contain sufficient
  information for even small probing classifiers to achieve perfect accuracy.
  Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning
  capability stems from the underlying LLM's inherent difficulty with temporal
  concepts, as evidenced by poor performance on textual temporal
  question-answering tasks. Building on this discovery, we introduce the Textual
  Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning
  tasks in pure text format from existing image-text datasets, addressing the
  scarcity of video samples with complex temporal scenarios. Remarkably, without
  using any video data, T3 enhances LongVA-7B's temporal understanding, yielding
  a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark,
  which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video
  samples. Additionally, the enhanced LongVA-7B model achieves competitive
  performance on comprehensive video benchmarks. For example, it achieves a 49.7
  accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful
  large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further
  analysis reveals a strong correlation between textual and video temporal task
  performance, validating the efficacy of transferring temporal reasoning
  abilities from text to video domains.
paper_ai_keywords: Video Large Language Models (Video LLMs) video comprehension temporal changes reasoning about applications temporal temporal encoding probing classifiers temporal reasoning capability textual temporal textual temporal reasoning transfer (T3) TempCompass benchmark LongVA-7B ShareGPT4Video-8B video benchmarks Temporal Reasoning task of Video-MME InternVL-1-V1.5-10B VILA1.5-40B textual and video temporal task performance