Temporal Reasoning Transfer from Text to Video
Video Large Language Models (Video LLMs) have shown promising capabilities in
video comprehension, yet they struggle with tracking temporal changes and
reasoning about temporal relationships. While previous research attributed this
limitation to the ineffective temporal encoding of visual inputs, our
diagnostic study reveals that video representations contain sufficient
information for even small probing classifiers to achieve perfect accuracy.
Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning
capability stems from the underlying LLM's inherent difficulty with temporal
concepts, as evidenced by poor performance on textual temporal
question-answering tasks. Building on this discovery, we introduce the Textual
Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning
tasks in pure text format from existing image-text datasets, addressing the
scarcity of video samples with complex temporal scenarios. Remarkably, without
using any video data, T3 enhances LongVA-7B's temporal understanding, yielding
a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark,
which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video
samples. Additionally, the enhanced LongVA-7B model achieves competitive
performance on comprehensive video benchmarks. For example, it achieves a 49.7
accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful
large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further
analysis reveals a strong correlation between textual and video temporal task
performance, validating the efficacy of transferring temporal reasoning
abilities from text to video domains.
6707388ec818ba2f6579eb78
Lei Li
admin_assigned
2024-10-10T09:11:14.571Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
6707388ec818ba2f6579eb79
Yuanxin Liu
claimed_verified
2024-10-10T08:01:34.232Z
6489761dcaea79f577897f98
/avatars/8f56dc9c08dc2b672555602d68509a03.svg
Yuanxin Liu
user
lyx97
6707388ec818ba2f6579eb7a
Linli Yao
admin_assigned
2024-10-10T09:11:21.755Z
655ca347f426a304c6b393a1
/avatars/67f0310d59c5912d38c2ad8e6448614d.svg
Linli Yao
user
yaolily
6707388ec818ba2f6579eb7b
Peiyuan Zhang
admin_assigned
2024-10-10T09:11:32.678Z
63565cc56d7fcf1bedb7d347
https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
Zhang Peiyuan
user
PY007
6707388ec818ba2f6579eb7c
Chenxin An
admin_assigned
2024-10-10T09:11:37.979Z
64acb321264bbbf171a2b040
/avatars/0ad344c0e9b1e3fda469932f91d117dc.svg
Chenxin An
user
Chancy
6707388ec818ba2f6579eb7d
Lean Wang
admin_assigned
2024-10-10T09:11:43.776Z
650c509472afb1e60e6151ae
/avatars/c16ab5053a586819dc2b965303215ff7.svg
Lean Wang
user
AdaHousman
6707388ec818ba2f6579eb7e
Xu Sun
6707388ec818ba2f6579eb7f
Lingpeng Kong
6707388ec818ba2f6579eb80
Qi Liu
admin_assigned
2024-10-10T09:12:34.540Z
63ad3de96ee60ca58a409280
/avatars/7461f4fda3692f042e556d2a7c339bc0.svg
Qi Liu
user
QiLiuHKU
Temporal Reasoning Transfer from Text to Video
Video Large Language Models (Video LLMs) have shown promising capabilities in
video comprehension, yet they struggle with tracking temporal changes and
reasoning about temporal relationships. While previous research attributed this
limitation to the ineffective temporal encoding of visual inputs, our
diagnostic study reveals that video representations contain sufficient
information for even small probing classifiers to achieve perfect accuracy.
Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning
capability stems from the underlying LLM's inherent difficulty with temporal
concepts, as evidenced by poor performance on textual temporal
question-answering tasks. Building on this discovery, we introduce the Textual
Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning
tasks in pure text format from existing image-text datasets, addressing the
scarcity of video samples with complex temporal scenarios. Remarkably, without
using any video data, T3 enhances LongVA-7B's temporal understanding, yielding
a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark,
which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video
samples. Additionally, the enhanced LongVA-7B model achieves competitive
performance on comprehensive video benchmarks. For example, it achieves a 49.7
accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful
large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further
analysis reveals a strong correlation between textual and video temporal task
performance, validating the efficacy of transferring temporal reasoning
abilities from text to video domains.
Textual Temporal reasoning Transfer (T3) enhances Video LLMs' temporal understanding by leveraging text-based temporal tasks, leading to superior performance on video benchmarks without additional video data.
Video LLMs temporal changes temporal relationships temporal encoding temporal reasoning temporal question-answering Textual Temporal reasoning Transfer (T3) TempCompass benchmark LongVA-7B ShareGPT4Video-8B Video-MME InternVL-Chat-V1.5-20B VILA1.5-40B