Unifying Specialized Visual Encoders for Video Language Models
The recent advent of Large Language Models (LLMs) has ushered sophisticated
reasoning capabilities into the realm of video through Video Large Language
Models (VideoLLMs). However, VideoLLMs currently rely on a single vision
encoder for all of their visual processing, which limits the amount and type of
visual information that can be conveyed to the LLM. Our method, MERV,
Multi-Encoder Representation of Videos, instead leverages multiple frozen
visual encoders to create a unified representation of a video, providing the
VideoLLM with a comprehensive set of specialized visual knowledge.
Spatio-temporally aligning the features from each encoder allows us to tackle a
wider range of open-ended and multiple-choice video understanding questions and
outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy
than Video-LLaVA across the standard suite video understanding benchmarks,
while also having a better Video-ChatGPT score. We also improve upon SeViLA,
the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV
introduces minimal extra parameters and trains faster than equivalent
single-encoder methods while parallelizing the visual processing. Finally, we
provide qualitative evidence that MERV successfully captures domain knowledge
from each of its encoders. Our results offer promising directions in utilizing
multiple vision encoders for comprehensive video understanding.
677773a8820f96f0211ae9fe
Jihoon Chung
677773a8820f96f0211ae9ff
Tyler Zhu
claimed_verified
2025-01-03T14:03:08.420Z
6301af7259ab5d9dc0a15060
/avatars/0bcda98801877332893a1de169000737.svg
Tyler Zhu
user
tyleryzhu
677773a8820f96f0211aea00
Max Gonzalez Saez-Diez
677773a8820f96f0211aea01
Juan Carlos Niebles
677773a8820f96f0211aea02
Honglu Zhou
677773a8820f96f0211aea03
Olga Russakovsky
Unifying Specialized Visual Encoders for Video Language Models
The recent advent of Large Language Models (LLMs) has ushered sophisticated
reasoning capabilities into the realm of video through Video Large Language
Models (VideoLLMs). However, VideoLLMs currently rely on a single vision
encoder for all of their visual processing, which limits the amount and type of
visual information that can be conveyed to the LLM. Our method, MERV,
Multi-Encoder Representation of Videos, instead leverages multiple frozen
visual encoders to create a unified representation of a video, providing the
VideoLLM with a comprehensive set of specialized visual knowledge.
Spatio-temporally aligning the features from each encoder allows us to tackle a
wider range of open-ended and multiple-choice video understanding questions and
outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy
than Video-LLaVA across the standard suite video understanding benchmarks,
while also having a better Video-ChatGPT score. We also improve upon SeViLA,
the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV
introduces minimal extra parameters and trains faster than equivalent
single-encoder methods while parallelizing the visual processing. Finally, we
provide qualitative evidence that MERV successfully captures domain knowledge
from each of its encoders. Our results offer promising directions in utilizing
multiple vision encoders for comprehensive video understanding.
MERV, a Multi-Encoder Representation of Videos, enhances VideoLLMs by using multiple visual encoders for better visual information processing, accuracy, and faster training.
Large Language Models Video Large Language Models multi-encoder visual encoders unified representation spatio-temporally aligning video understanding accuracy Video-LLaVA Video-ChatGPT SeViLA zero-shot Perception Test domain knowledge