title: video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
summary: While recent advancements in reasoning optimization have significantly
  enhanced the capabilities of large language models (LLMs), existing efforts to
  improve reasoning have been limited to solving mathematical problems and
  focusing on visual graphical inputs, neglecting broader applications in general
  video understanding.This paper proposes video-SALMONN-o1, the first open-source
  reasoning-enhanced audio-visual LLM designed for general video understanding
  tasks. To enhance its reasoning abilities, we develop a reasoning-intensive
  dataset featuring challenging audio-visual questions with step-by-step
  solutions. We also propose process direct preference optimization (pDPO), which
  leverages contrastive step selection to achieve efficient step-level reward
  modelling tailored for multimodal inputs. Additionally, we introduce RivaBench,
  the first reasoning-intensive video understanding benchmark, featuring over
  4,000 high-quality, expert-curated question-answer pairs across scenarios such
  as standup comedy, academic presentations, and synthetic video detection.
  video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision
  baseline across different video reasoning benchmarks. Besides, pDPO achieves
  6-8% improvements compared to the supervised fine-tuning model on RivaBench.
  Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection
  capabilities.
paper_authors._id: 67b4147f7721b4fe4d2bd466
paper_authors.name: Guangzhi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:50:36.976Z
paper_authors._id: 64ab544489aa67e4a2505eeb
paper_authors.avatarUrl: /avatars/f1a9def3afbec2f8b89ef4450770d67e.svg
paper_authors.fullname: Guangzhi Sun
paper_authors.type: user
paper_authors.user: BrianatCambridge
paper_authors._id: 67b4147f7721b4fe4d2bd467
paper_authors.name: Yudong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:50:43.942Z
paper_authors._id: 66ea99f751c3bcd917ae1265
paper_authors.avatarUrl: /avatars/17c4206c39d4da49d2516422c392e0ee.svg
paper_authors.fullname: Yudong Yang
paper_authors.type: user
paper_authors.user: yangyudong2020
paper_authors._id: 67b4147f7721b4fe4d2bd468
paper_authors.name: Jimin Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:50:50.168Z
paper_authors._id: 66825d0e971dff9d3a24d912
paper_authors.avatarUrl: /avatars/024b704a1f8c3aec20474e162ac326ad.svg
paper_authors.fullname: Jimin Zhuang
paper_authors.type: user
paper_authors.user: OctaAcid
paper_authors._id: 67b4147f7721b4fe4d2bd469
paper_authors.name: Changli Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:50:57.247Z
paper_authors._id: 63770389cdcc1bf630870758
paper_authors.avatarUrl: /avatars/6bc7c602e79688bc42a4c79ecf5b6d2d.svg
paper_authors.fullname: Changli Tang
paper_authors.type: user
paper_authors.user: Changli
paper_authors._id: 67b4147f7721b4fe4d2bd46a
paper_authors.name: Yixuan Li
paper_authors._id: 67b4147f7721b4fe4d2bd46b
paper_authors.name: Wei Li
paper_authors._id: 67b4147f7721b4fe4d2bd46c
paper_authors.name: Zejun MA
paper_authors._id: 67b4147f7721b4fe4d2bd46d
paper_authors.name: Chao Zhang
paper_title: video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
paper_summary: While recent advancements in reasoning optimization have significantly
  enhanced the capabilities of large language models (LLMs), existing efforts to
  improve reasoning have been limited to solving mathematical problems and
  focusing on visual graphical inputs, neglecting broader applications in general
  video understanding.This paper proposes video-SALMONN-o1, the first open-source
  reasoning-enhanced audio-visual LLM designed for general video understanding
  tasks. To enhance its reasoning abilities, we develop a reasoning-intensive
  dataset featuring challenging audio-visual questions with step-by-step
  solutions. We also propose process direct preference optimization (pDPO), which
  leverages contrastive step selection to achieve efficient step-level reward
  modelling tailored for multimodal inputs. Additionally, we introduce RivaBench,
  the first reasoning-intensive video understanding benchmark, featuring over
  4,000 high-quality, expert-curated question-answer pairs across scenarios such
  as standup comedy, academic presentations, and synthetic video detection.
  video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision
  baseline across different video reasoning benchmarks. Besides, pDPO achieves
  6-8% improvements compared to the supervised fine-tuning model on RivaBench.
  Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection
  capabilities.
paper_ai_keywords: video-SALMONN-o1 audio-visual LLM reasoning-intensive step-by-step process direct preference optimization (pDPO) contrastive step selection multimodal inputs RivaBench high-quality, expert-curated question-answer pairs video reasoning benchmarks LLaVA-OneVision zero-shot synthetic video detection