title: Multimodal Long Video Modeling Based on Temporal Dynamic Context
summary: Recent advances in Large Language Models (LLMs) have led to significant
  breakthroughs in video understanding. However, existing models still struggle
  with long video processing due to the context length constraint of LLMs and the
  vast amount of information within the video. Although some recent methods are
  designed for long video understanding, they often lose crucial information
  during token compression and struggle with additional modality like audio. In
  this work, we propose a dynamic long video encoding method utilizing the
  temporal relationship between frames, named Temporal Dynamic Context (TDC).
  Firstly, we segment the video into semantically consistent scenes based on
  inter-frame similarities, then encode each frame into tokens using visual-audio
  encoders. Secondly, we propose a novel temporal context compressor to reduce
  the number of tokens within each segment. Specifically, we employ a query-based
  Transformer to aggregate video, audio, and instruction text tokens into a
  limited set of temporal context tokens. Finally, we feed the static frame
  tokens and the temporal context tokens into the LLM for video understanding.
  Furthermore, to handle extremely long videos, we propose a training-free
  chain-of-thought strategy that progressively extracts answers from multiple
  video segments. These intermediate answers serve as part of the reasoning
  process and contribute to the final answer. We conduct extensive experiments on
  general video understanding and audio-video understanding benchmarks, where our
  method demonstrates strong performance. The code and models are available at
  https://github.com/Hoar012/TDC-Video.
paper_authors._id: 67ff92d142dd8c5df9fcb97a
paper_authors.name: Haoran Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T11:57:49.451Z
paper_authors._id: 669e221425f2081c6e3d8b61
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669e221425f2081c6e3d8b61/OearXzKrvoIZtDKTH4kje.jpeg
paper_authors.fullname: Haoran Hao
paper_authors.type: user
paper_authors.user: Hoar012
paper_authors._id: 67ff92d142dd8c5df9fcb97b
paper_authors.name: Jiaming Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-17T11:10:32.279Z
paper_authors._id: 62318c0386753f5f41d0e261
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
paper_authors.fullname: Jiaming Han
paper_authors.type: user
paper_authors.user: csuhan
paper_authors._id: 67ff92d142dd8c5df9fcb97c
paper_authors.name: Yiyuan Zhang
paper_authors._id: 67ff92d142dd8c5df9fcb97d
paper_authors.name: Xiangyu Yue
paper_title: Multimodal Long Video Modeling Based on Temporal Dynamic Context
paper_summary: Recent advances in Large Language Models (LLMs) have led to significant
  breakthroughs in video understanding. However, existing models still struggle
  with long video processing due to the context length constraint of LLMs and the
  vast amount of information within the video. Although some recent methods are
  designed for long video understanding, they often lose crucial information
  during token compression and struggle with additional modality like audio. In
  this work, we propose a dynamic long video encoding method utilizing the
  temporal relationship between frames, named Temporal Dynamic Context (TDC).
  Firstly, we segment the video into semantically consistent scenes based on
  inter-frame similarities, then encode each frame into tokens using visual-audio
  encoders. Secondly, we propose a novel temporal context compressor to reduce
  the number of tokens within each segment. Specifically, we employ a query-based
  Transformer to aggregate video, audio, and instruction text tokens into a
  limited set of temporal context tokens. Finally, we feed the static frame
  tokens and the temporal context tokens into the LLM for video understanding.
  Furthermore, to handle extremely long videos, we propose a training-free
  chain-of-thought strategy that progressively extracts answers from multiple
  video segments. These intermediate answers serve as part of the reasoning
  process and contribute to the final answer. We conduct extensive experiments on
  general video understanding and audio-video understanding benchmarks, where our
  method demonstrates strong performance. The code and models are available at
  https://github.com/Hoar012/TDC-Video.
paper_ai_summary: A method for dynamic long video encoding uses Temporal Dynamic Context to compress tokens while retaining information, and uses a chain-of-thought strategy for extremely long videos.
paper_ai_keywords: Large Language Models long video understanding context length constraint token compression Temporal Dynamic Context inter-frame similarities visual-audio encoders temporal context compressor query-based Transformer static frame tokens temporal context tokens general video understanding audio-video understanding benchmarks chain-of-thought strategy