Multimodal Long Video Modeling Based on Temporal Dynamic Context
Recent advances in Large Language Models (LLMs) have led to significant
breakthroughs in video understanding. However, existing models still struggle
with long video processing due to the context length constraint of LLMs and the
vast amount of information within the video. Although some recent methods are
designed for long video understanding, they often lose crucial information
during token compression and struggle with additional modality like audio. In
this work, we propose a dynamic long video encoding method utilizing the
temporal relationship between frames, named Temporal Dynamic Context (TDC).
Firstly, we segment the video into semantically consistent scenes based on
inter-frame similarities, then encode each frame into tokens using visual-audio
encoders. Secondly, we propose a novel temporal context compressor to reduce
the number of tokens within each segment. Specifically, we employ a query-based
Transformer to aggregate video, audio, and instruction text tokens into a
limited set of temporal context tokens. Finally, we feed the static frame
tokens and the temporal context tokens into the LLM for video understanding.
Furthermore, to handle extremely long videos, we propose a training-free
chain-of-thought strategy that progressively extracts answers from multiple
video segments. These intermediate answers serve as part of the reasoning
process and contribute to the final answer. We conduct extensive experiments on
general video understanding and audio-video understanding benchmarks, where our
method demonstrates strong performance. The code and models are available at
https://github.com/Hoar012/TDC-Video.
67ff92d142dd8c5df9fcb97a
Haoran Hao
claimed_verified
2025-04-16T11:57:49.451Z
669e221425f2081c6e3d8b61
https://cdn-avatars.huggingface.co/v1/production/uploads/669e221425f2081c6e3d8b61/OearXzKrvoIZtDKTH4kje.jpeg
Haoran Hao
user
Hoar012
67ff92d142dd8c5df9fcb97b
Jiaming Han
claimed_verified
2025-04-17T11:10:32.279Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
67ff92d142dd8c5df9fcb97c
Yiyuan Zhang
67ff92d142dd8c5df9fcb97d
Xiangyu Yue
Multimodal Long Video Modeling Based on Temporal Dynamic Context
Recent advances in Large Language Models (LLMs) have led to significant
breakthroughs in video understanding. However, existing models still struggle
with long video processing due to the context length constraint of LLMs and the
vast amount of information within the video. Although some recent methods are
designed for long video understanding, they often lose crucial information
during token compression and struggle with additional modality like audio. In
this work, we propose a dynamic long video encoding method utilizing the
temporal relationship between frames, named Temporal Dynamic Context (TDC).
Firstly, we segment the video into semantically consistent scenes based on
inter-frame similarities, then encode each frame into tokens using visual-audio
encoders. Secondly, we propose a novel temporal context compressor to reduce
the number of tokens within each segment. Specifically, we employ a query-based
Transformer to aggregate video, audio, and instruction text tokens into a
limited set of temporal context tokens. Finally, we feed the static frame
tokens and the temporal context tokens into the LLM for video understanding.
Furthermore, to handle extremely long videos, we propose a training-free
chain-of-thought strategy that progressively extracts answers from multiple
video segments. These intermediate answers serve as part of the reasoning
process and contribute to the final answer. We conduct extensive experiments on
general video understanding and audio-video understanding benchmarks, where our
method demonstrates strong performance. The code and models are available at
https://github.com/Hoar012/TDC-Video.
A method for dynamic long video encoding uses Temporal Dynamic Context to compress tokens while retaining information, and uses a chain-of-thought strategy for extremely long videos.
Large Language Models long video understanding context length constraint token compression Temporal Dynamic Context inter-frame similarities visual-audio encoders temporal context compressor query-based Transformer static frame tokens temporal context tokens general video understanding audio-video understanding benchmarks chain-of-thought strategy