title: PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance
summary: The past year has witnessed the significant advancement of video-based large
  language models. However, the challenge of developing a unified model for both
  short and long video understanding remains unresolved. Most existing video LLMs
  cannot handle hour-long videos, while methods custom for long videos tend to be
  ineffective for shorter videos and images. In this paper, we identify the key
  issue as the redundant content in videos. To address this, we propose a novel
  pooling strategy that simultaneously achieves token compression and
  instruction-aware visual feature aggregation. Our model is termed Prompt-guided
  Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three
  core components: the CLIP-based visual-prompt alignment that extracts visual
  information relevant to the user's instructions, the prompt-guided pooling that
  compresses the visual sequence to arbitrary scales using convolution-style
  pooling, and the clip context extension designed for lengthy prompt common in
  visual dialogue. Moreover, our codebase also integrates the most advanced video
  Direct Preference Optimization (DPO) and visual interleave training. Extensive
  experiments have validated the performance of our model. With superior
  throughput and only 1024 visual context, PPLLaVA achieves better results on
  image benchmarks as a video LLM, while achieving state-of-the-art performance
  across various video benchmarks, excelling in tasks ranging from caption
  generation to multiple-choice questions, and handling video lengths from
  seconds to hours. Codes have been available at
  https://github.com/farewellthree/PPLLaVA.
paper_authors._id: 672a17a8341f15009010af9d
paper_authors.name: Ruyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T13:36:20.999Z
paper_authors._id: 650f0e3988cdfe73a864b9c5
paper_authors.avatarUrl: /avatars/ce74b96743a99819a762c23b4f2204fd.svg
paper_authors.fullname: Ruyang Liu
paper_authors.type: user
paper_authors.user: farewellthree
paper_authors._id: 672a17a8341f15009010af9e
paper_authors.name: Haoran Tang
paper_authors._id: 672a17a8341f15009010af9f
paper_authors.name: Haibo Liu
paper_authors._id: 672a17a8341f15009010afa0
paper_authors.name: Yixiao Ge
paper_authors._id: 672a17a8341f15009010afa1
paper_authors.name: Ying Shan
paper_authors._id: 672a17a8341f15009010afa2
paper_authors.name: Chen Li
paper_authors._id: 672a17a8341f15009010afa3
paper_authors.name: Jiankun Yang
paper_title: PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance
paper_summary: The past year has witnessed the significant advancement of video-based large
  language models. However, the challenge of developing a unified model for both
  short and long video understanding remains unresolved. Most existing video LLMs
  cannot handle hour-long videos, while methods custom for long videos tend to be
  ineffective for shorter videos and images. In this paper, we identify the key
  issue as the redundant content in videos. To address this, we propose a novel
  pooling strategy that simultaneously achieves token compression and
  instruction-aware visual feature aggregation. Our model is termed Prompt-guided
  Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three
  core components: the CLIP-based visual-prompt alignment that extracts visual
  information relevant to the user's instructions, the prompt-guided pooling that
  compresses the visual sequence to arbitrary scales using convolution-style
  pooling, and the clip context extension designed for lengthy prompt common in
  visual dialogue. Moreover, our codebase also integrates the most advanced video
  Direct Preference Optimization (DPO) and visual interleave training. Extensive
  experiments have validated the performance of our model. With superior
  throughput and only 1024 visual context, PPLLaVA achieves better results on
  image benchmarks as a video LLM, while achieving state-of-the-art performance
  across various video benchmarks, excelling in tasks ranging from caption
  generation to multiple-choice questions, and handling video lengths from
  seconds to hours. Codes have been available at
  https://github.com/farewellthree/PPLLaVA.
paper_ai_summary: PPLLaVA is a video-based large language model that addresses challenges in handling both short and long videos by using a prompt-guided pooling strategy to efficiently compress and aggregate visual features.
paper_ai_keywords: video-based large language models token compression instruction-aware visual feature aggregation CLIP-based visual-prompt alignment prompt-guided pooling convolution-style pooling clip context extension Direct Preference Optimization (DPO) visual interleave training caption generation multiple-choice questions