title: Unifying Specialized Visual Encoders for Video Language Models
summary: The recent advent of Large Language Models (LLMs) has ushered sophisticated
  reasoning capabilities into the realm of video through Video Large Language
  Models (VideoLLMs). However, VideoLLMs currently rely on a single vision
  encoder for all of their visual processing, which limits the amount and type of
  visual information that can be conveyed to the LLM. Our method, MERV,
  Multi-Encoder Representation of Videos, instead leverages multiple frozen
  visual encoders to create a unified representation of a video, providing the
  VideoLLM with a comprehensive set of specialized visual knowledge.
  Spatio-temporally aligning the features from each encoder allows us to tackle a
  wider range of open-ended and multiple-choice video understanding questions and
  outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy
  than Video-LLaVA across the standard suite video understanding benchmarks,
  while also having a better Video-ChatGPT score. We also improve upon SeViLA,
  the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV
  introduces minimal extra parameters and trains faster than equivalent
  single-encoder methods while parallelizing the visual processing. Finally, we
  provide qualitative evidence that MERV successfully captures domain knowledge
  from each of its encoders. Our results offer promising directions in utilizing
  multiple vision encoders for comprehensive video understanding.
paper_authors._id: 677773a8820f96f0211ae9fe
paper_authors.name: Jihoon Chung
paper_authors._id: 677773a8820f96f0211ae9ff
paper_authors.name: Tyler Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:08.420Z
paper_authors._id: 6301af7259ab5d9dc0a15060
paper_authors.avatarUrl: /avatars/0bcda98801877332893a1de169000737.svg
paper_authors.fullname: Tyler Zhu
paper_authors.type: user
paper_authors.user: tyleryzhu
paper_authors._id: 677773a8820f96f0211aea00
paper_authors.name: Max Gonzalez Saez-Diez
paper_authors._id: 677773a8820f96f0211aea01
paper_authors.name: Juan Carlos Niebles
paper_authors._id: 677773a8820f96f0211aea02
paper_authors.name: Honglu Zhou
paper_authors._id: 677773a8820f96f0211aea03
paper_authors.name: Olga Russakovsky
paper_title: Unifying Specialized Visual Encoders for Video Language Models
paper_summary: The recent advent of Large Language Models (LLMs) has ushered sophisticated
  reasoning capabilities into the realm of video through Video Large Language
  Models (VideoLLMs). However, VideoLLMs currently rely on a single vision
  encoder for all of their visual processing, which limits the amount and type of
  visual information that can be conveyed to the LLM. Our method, MERV,
  Multi-Encoder Representation of Videos, instead leverages multiple frozen
  visual encoders to create a unified representation of a video, providing the
  VideoLLM with a comprehensive set of specialized visual knowledge.
  Spatio-temporally aligning the features from each encoder allows us to tackle a
  wider range of open-ended and multiple-choice video understanding questions and
  outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy
  than Video-LLaVA across the standard suite video understanding benchmarks,
  while also having a better Video-ChatGPT score. We also improve upon SeViLA,
  the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV
  introduces minimal extra parameters and trains faster than equivalent
  single-encoder methods while parallelizing the visual processing. Finally, we
  provide qualitative evidence that MERV successfully captures domain knowledge
  from each of its encoders. Our results offer promising directions in utilizing
  multiple vision encoders for comprehensive video understanding.
paper_ai_keywords: Large Language Models (LLMs) Video Large Language Models (VideoLLMs) Multi-Encoder Representation of Videos (MERV) visual encoders unified representation specialized visual knowledge Spatio-temporally aligning open-ended video understanding questions multiple-choice video understanding questions prior state-of-the-art works Video-LLaVA Video-ChatGPT SeViLA zero-shot Perception Test parallelizing the visual processing domain knowledge