title: MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video
    Understanding
summary: With the success of large language models (LLMs), integrating the vision
  model into LLMs to build vision-language foundation models has gained much more
  interest recently. However, existing LLM-based large multimodal models (e.g.,
  Video-LLaMA, VideoChat) can only take in a limited number of frames for short
  video understanding. In this study, we mainly focus on designing an efficient
  and effective model for long-term video understanding. Instead of trying to
  process more frames simultaneously like most existing work, we propose to
  process videos in an online manner and store past video information in a memory
  bank. This allows our model to reference historical video content for long-term
  analysis without exceeding LLMs' context length constraints or GPU memory
  limits. Our memory bank can be seamlessly integrated into current multimodal
  LLMs in an off-the-shelf manner. We conduct extensive experiments on various
  video understanding tasks, such as long-video understanding, video question
  answering, and video captioning, and our model can achieve state-of-the-art
  performances across multiple datasets. Code available at
  https://boheumd.github.io/MA-LMM/.
paper_authors._id: 6614b63e49d371dcc819c58d
paper_authors.name: Bo He
paper_authors._id: 6614b63e49d371dcc819c58e
paper_authors.name: Hengduo Li
paper_authors._id: 6614b63e49d371dcc819c58f
paper_authors.name: Young Kyun Jang
paper_authors._id: 6614b63e49d371dcc819c590
paper_authors.name: Menglin Jia
paper_authors._id: 6614b63e49d371dcc819c591
paper_authors.name: Xuefei Cao
paper_authors._id: 6614b63e49d371dcc819c592
paper_authors.name: Ashish Shah
paper_authors._id: 6614b63e49d371dcc819c593
paper_authors.name: Abhinav Shrivastava
paper_authors._id: 6614b63e49d371dcc819c594
paper_authors.name: Ser-Nam Lim
paper_title: MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video
    Understanding
paper_summary: With the success of large language models (LLMs), integrating the vision
  model into LLMs to build vision-language foundation models has gained much more
  interest recently. However, existing LLM-based large multimodal models (e.g.,
  Video-LLaMA, VideoChat) can only take in a limited number of frames for short
  video understanding. In this study, we mainly focus on designing an efficient
  and effective model for long-term video understanding. Instead of trying to
  process more frames simultaneously like most existing work, we propose to
  process videos in an online manner and store past video information in a memory
  bank. This allows our model to reference historical video content for long-term
  analysis without exceeding LLMs' context length constraints or GPU memory
  limits. Our memory bank can be seamlessly integrated into current multimodal
  LLMs in an off-the-shelf manner. We conduct extensive experiments on various
  video understanding tasks, such as long-video understanding, video question
  answering, and video captioning, and our model can achieve state-of-the-art
  performances across multiple datasets. Code available at
  https://boheumd.github.io/MA-LMM/.
paper_ai_summary: A memory bank is integrated into large multimodal models for efficient long-term video understanding, achieving state-of-the-art performance across various video tasks.
paper_ai_keywords: large language models vision-language foundation models multimodal models Video-LLaMA VideoChat long-term video understanding online processing memory bank long-video understanding video question answering video captioning