PAVE: Patching and Adapting Video Large Language Models
Pre-trained video large language models (Video LLMs) exhibit remarkable
reasoning capabilities, yet adapting these models to new tasks involving
additional modalities or data types (e.g., audio or 3D information) remains
challenging. In this paper, we present PAVE, a flexible framework for adapting
pre-trained Video LLMs to downstream tasks with side-channel signals, such as
audio, 3D cues, or multi-view videos. PAVE introduces lightweight adapters,
referred to as "patches," which add a small number of parameters and operations
to a base model without modifying its architecture or pre-trained weights. In
doing so, PAVE can effectively adapt the pre-trained base model to support
diverse downstream tasks, including audio-visual question answering, 3D
reasoning, multi-view video recognition, and high frame rate video
understanding. Across these tasks, PAVE significantly enhances the performance
of the base model, surpassing state-of-the-art task-specific models while
incurring a minor cost of ~0.1% additional FLOPs and parameters. Further, PAVE
supports multi-task learning and generalizes well across different Video LLMs.
Our code is available at https://github.com/dragonlzm/PAVE.
67e372f9988d6c18d8204f48
Zhuoming Liu
claimed_verified
2025-03-26T20:44:26.546Z
666f2534a119281ee0e39783
/avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg
Zhuoming Liu
user
zhuomingliu
67e372f9988d6c18d8204f49
Yiquan Li
67e372f9988d6c18d8204f4a
Khoi Duc Nguyen
67e372f9988d6c18d8204f4b
Yiwu Zhong
67e372f9988d6c18d8204f4c
Yin Li
PAVE: Patching and Adapting Video Large Language Models
Pre-trained video large language models (Video LLMs) exhibit remarkable
reasoning capabilities, yet adapting these models to new tasks involving
additional modalities or data types (e.g., audio or 3D information) remains
challenging. In this paper, we present PAVE, a flexible framework for adapting
pre-trained Video LLMs to downstream tasks with side-channel signals, such as
audio, 3D cues, or multi-view videos. PAVE introduces lightweight adapters,
referred to as "patches," which add a small number of parameters and operations
to a base model without modifying its architecture or pre-trained weights. In
doing so, PAVE can effectively adapt the pre-trained base model to support
diverse downstream tasks, including audio-visual question answering, 3D
reasoning, multi-view video recognition, and high frame rate video
understanding. Across these tasks, PAVE significantly enhances the performance
of the base model, surpassing state-of-the-art task-specific models while
incurring a minor cost of ~0.1% additional FLOPs and parameters. Further, PAVE
supports multi-task learning and generalizes well across different Video LLMs.
Our code is available at https://github.com/dragonlzm/PAVE.
PAVE is a flexible framework that adapts pre-trained Video LLMs with lightweight patches for diverse downstream tasks, enhancing performance while requiring minimal additional resources.
Video LLMs PAVE adapters patches side-channel signals audio-visual question answering 3D reasoning multi-view video recognition high frame rate video understanding multi-task learning