title: PAVE: Patching and Adapting Video Large Language Models
summary: Pre-trained video large language models (Video LLMs) exhibit remarkable
  reasoning capabilities, yet adapting these models to new tasks involving
  additional modalities or data types (e.g., audio or 3D information) remains
  challenging. In this paper, we present PAVE, a flexible framework for adapting
  pre-trained Video LLMs to downstream tasks with side-channel signals, such as
  audio, 3D cues, or multi-view videos. PAVE introduces lightweight adapters,
  referred to as "patches," which add a small number of parameters and operations
  to a base model without modifying its architecture or pre-trained weights. In
  doing so, PAVE can effectively adapt the pre-trained base model to support
  diverse downstream tasks, including audio-visual question answering, 3D
  reasoning, multi-view video recognition, and high frame rate video
  understanding. Across these tasks, PAVE significantly enhances the performance
  of the base model, surpassing state-of-the-art task-specific models while
  incurring a minor cost of ~0.1% additional FLOPs and parameters. Further, PAVE
  supports multi-task learning and generalizes well across different Video LLMs.
  Our code is available at https://github.com/dragonlzm/PAVE.
paper_authors._id: 67e372f9988d6c18d8204f48
paper_authors.name: Zhuoming Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:44:26.546Z
paper_authors._id: 666f2534a119281ee0e39783
paper_authors.avatarUrl: /avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg
paper_authors.fullname: Zhuoming Liu
paper_authors.type: user
paper_authors.user: zhuomingliu
paper_authors._id: 67e372f9988d6c18d8204f49
paper_authors.name: Yiquan Li
paper_authors._id: 67e372f9988d6c18d8204f4a
paper_authors.name: Khoi Duc Nguyen
paper_authors._id: 67e372f9988d6c18d8204f4b
paper_authors.name: Yiwu Zhong
paper_authors._id: 67e372f9988d6c18d8204f4c
paper_authors.name: Yin Li
paper_title: PAVE: Patching and Adapting Video Large Language Models
paper_summary: Pre-trained video large language models (Video LLMs) exhibit remarkable
  reasoning capabilities, yet adapting these models to new tasks involving
  additional modalities or data types (e.g., audio or 3D information) remains
  challenging. In this paper, we present PAVE, a flexible framework for adapting
  pre-trained Video LLMs to downstream tasks with side-channel signals, such as
  audio, 3D cues, or multi-view videos. PAVE introduces lightweight adapters,
  referred to as "patches," which add a small number of parameters and operations
  to a base model without modifying its architecture or pre-trained weights. In
  doing so, PAVE can effectively adapt the pre-trained base model to support
  diverse downstream tasks, including audio-visual question answering, 3D
  reasoning, multi-view video recognition, and high frame rate video
  understanding. Across these tasks, PAVE significantly enhances the performance
  of the base model, surpassing state-of-the-art task-specific models while
  incurring a minor cost of ~0.1% additional FLOPs and parameters. Further, PAVE
  supports multi-task learning and generalizes well across different Video LLMs.
  Our code is available at https://github.com/dragonlzm/PAVE.
paper_ai_summary: PAVE is a flexible framework that adapts pre-trained Video LLMs with lightweight patches for diverse downstream tasks, enhancing performance while requiring minimal additional resources.
paper_ai_keywords: Video LLMs PAVE adapters patches side-channel signals audio-visual question answering 3D reasoning multi-view video recognition high frame rate video understanding multi-task learning