title: PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video
    Dense Captioning
summary: Vision-language pre-training has significantly elevated performance across a
  wide range of image-language applications. Yet, the pre-training process for
  video-related tasks demands exceptionally large computational and data
  resources, which hinders the progress of video-language models. This paper
  investigates a straightforward, highly efficient, and resource-light approach
  to adapting an existing image-language pre-trained model for dense video
  understanding. Our preliminary experiments reveal that directly fine-tuning
  pre-trained image-language models with multiple frames as inputs on video
  datasets leads to performance saturation or even a drop. Our further
  investigation reveals that it is largely attributed to the bias of learned
  high-norm visual features. Motivated by this finding, we propose a simple but
  effective pooling strategy to smooth the feature distribution along the
  temporal dimension and thus reduce the dominant impacts from the extreme
  features. The new model is termed Pooling LLaVA, or  in short.
   achieves new state-of-the-art performance on modern benchmark
  datasets for both video question-answer and captioning tasks. Notably, on the
  recent popular Video ChatGPT benchmark, PLLaVA achieves a score of 3.48 out of
  5 on average of five evaluated dimensions, exceeding the previous SOTA results
  from GPT4V (IG-VLM) by 9\%. On the latest multi-choice benchmark MVBench,
  PLLaVA achieves 58.1\% accuracy on average across 20 sub-tasks, 14.5\% higher
  than GPT4V (IG-VLM). Code is available at
  https://github.com/magic-research/PLLaVA.
paper_authors._id: 662ef3e7b10f82aa4fabd7c9
paper_authors.name: Lin Xu
paper_authors._id: 662ef3e7b10f82aa4fabd7ca
paper_authors.name: Yilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T11:48:54.834Z
paper_authors._id: 6385d86181fe8c678a345ff0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6385d86181fe8c678a345ff0/ETvVOtBAQ-RpwfT4TCWsv.jpeg
paper_authors.fullname: Yilin Zhao
paper_authors.type: user
paper_authors.user: ermu2001
paper_authors._id: 662ef3e7b10f82aa4fabd7cb
paper_authors.name: Daquan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T11:49:16.844Z
paper_authors._id: 63fe0b160c1bbe8e29d2dd32
paper_authors.avatarUrl: /avatars/bc574036287170a77057893efaa48e2d.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: DaQuan21
paper_authors._id: 662ef3e7b10f82aa4fabd7cc
paper_authors.name: Zhijie Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T11:57:19.161Z
paper_authors._id: 64415957bd0c9726529802f6
paper_authors.avatarUrl: /avatars/1132d1ee68fb58ec635d57c8175caacd.svg
paper_authors.fullname: Zhijie Lin
paper_authors.type: user
paper_authors.user: Ikuinen
paper_authors._id: 662ef3e7b10f82aa4fabd7cd
paper_authors.name: See Kiong Ng
paper_authors._id: 662ef3e7b10f82aa4fabd7ce
paper_authors.name: Jiashi Feng
paper_title: PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video
    Dense Captioning
paper_summary: Vision-language pre-training has significantly elevated performance across a
  wide range of image-language applications. Yet, the pre-training process for
  video-related tasks demands exceptionally large computational and data
  resources, which hinders the progress of video-language models. This paper
  investigates a straightforward, highly efficient, and resource-light approach
  to adapting an existing image-language pre-trained model for dense video
  understanding. Our preliminary experiments reveal that directly fine-tuning
  pre-trained image-language models with multiple frames as inputs on video
  datasets leads to performance saturation or even a drop. Our further
  investigation reveals that it is largely attributed to the bias of learned
  high-norm visual features. Motivated by this finding, we propose a simple but
  effective pooling strategy to smooth the feature distribution along the
  temporal dimension and thus reduce the dominant impacts from the extreme
  features. The new model is termed Pooling LLaVA, or  in short.
   achieves new state-of-the-art performance on modern benchmark
  datasets for both video question-answer and captioning tasks. Notably, on the
  recent popular Video ChatGPT benchmark, PLLaVA achieves a score of 3.48 out of
  5 on average of five evaluated dimensions, exceeding the previous SOTA results
  from GPT4V (IG-VLM) by 9\%. On the latest multi-choice benchmark MVBench,
  PLLaVA achieves 58.1\% accuracy on average across 20 sub-tasks, 14.5\% higher
  than GPT4V (IG-VLM). Code is available at
  https://github.com/magic-research/PLLaVA.
paper_ai_summary: Pooling LLaVA improves video-language pre-trained models by smoothing high-norm visual features, achieving state-of-the-art performance in video question-answering, captioning, and benchmark tasks.
paper_ai_keywords: vision-language pre-training dense video understanding fine-tuning image-language models video datasets high-norm visual features pooling strategy temporal dimension extreme features video question-answer captioning Video ChatGPT benchmark MVBench benchmark accuracy