title: ST-LLM: Large Language Models Are Effective Temporal Learners
summary: Large Language Models (LLMs) have showcased impressive capabilities in text
  comprehension and generation, prompting research efforts towards video LLMs to
  facilitate human-AI interaction at the video level. However, how to effectively
  encode and understand videos in video-based dialogue systems remains to be
  solved. In this paper, we investigate a straightforward yet unexplored
  question: Can we feed all spatial-temporal tokens into the LLM, thus delegating
  the task of video sequence modeling to the LLMs? Surprisingly, this simple
  approach yields significant improvements in video understanding. Based upon
  this, we propose ST-LLM, an effective video-LLM baseline with Spatial-Temporal
  sequence modeling inside LLM. Furthermore, to address the overhead and
  stability issues introduced by uncompressed video tokens within LLMs, we
  develop a dynamic masking strategy with tailor-made training objectives. For
  particularly long videos, we have also designed a global-local input module to
  balance efficiency and effectiveness. Consequently, we harness LLM for
  proficient spatial-temporal modeling, while upholding efficiency and stability.
  Extensive experimental results attest to the effectiveness of our method.
  Through a more concise model and training pipeline, ST-LLM establishes a new
  state-of-the-art result on VideoChatGPT-Bench and MVBench. Codes have been
  available at https://github.com/TencentARC/ST-LLM.
paper_authors._id: 660c1ff5c1174469000169fa
paper_authors.name: Ruyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T13:36:37.512Z
paper_authors._id: 650f0e3988cdfe73a864b9c5
paper_authors.avatarUrl: /avatars/ce74b96743a99819a762c23b4f2204fd.svg
paper_authors.fullname: Ruyang Liu
paper_authors.type: user
paper_authors.user: farewellthree
paper_authors._id: 660c1ff5c1174469000169fb
paper_authors.name: Chen Li
paper_authors._id: 660c1ff5c1174469000169fc
paper_authors.name: Haoran Tang
paper_authors._id: 660c1ff5c1174469000169fd
paper_authors.name: Yixiao Ge
paper_authors._id: 660c1ff5c1174469000169fe
paper_authors.name: Ying Shan
paper_authors._id: 660c1ff5c1174469000169ff
paper_authors.name: Ge Li
paper_title: ST-LLM: Large Language Models Are Effective Temporal Learners
paper_summary: Large Language Models (LLMs) have showcased impressive capabilities in text
  comprehension and generation, prompting research efforts towards video LLMs to
  facilitate human-AI interaction at the video level. However, how to effectively
  encode and understand videos in video-based dialogue systems remains to be
  solved. In this paper, we investigate a straightforward yet unexplored
  question: Can we feed all spatial-temporal tokens into the LLM, thus delegating
  the task of video sequence modeling to the LLMs? Surprisingly, this simple
  approach yields significant improvements in video understanding. Based upon
  this, we propose ST-LLM, an effective video-LLM baseline with Spatial-Temporal
  sequence modeling inside LLM. Furthermore, to address the overhead and
  stability issues introduced by uncompressed video tokens within LLMs, we
  develop a dynamic masking strategy with tailor-made training objectives. For
  particularly long videos, we have also designed a global-local input module to
  balance efficiency and effectiveness. Consequently, we harness LLM for
  proficient spatial-temporal modeling, while upholding efficiency and stability.
  Extensive experimental results attest to the effectiveness of our method.
  Through a more concise model and training pipeline, ST-LLM establishes a new
  state-of-the-art result on VideoChatGPT-Bench and MVBench. Codes have been
  available at https://github.com/TencentARC/ST-LLM.
paper_ai_summary: ST-LLM leverages large language models to effectively encode and understand videos for dialogue systems by integrating spatial-temporal modeling with dynamic masking and global-local input strategies, achieving state-of-the-art performance on VideoChatGPT-Bench and MVBench.
paper_ai_keywords: Spatial-Temporal sequence modeling dynamic masking strategy global-local input module ST-LLM