video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
While recent advancements in reasoning optimization have significantly
enhanced the capabilities of large language models (LLMs), existing efforts to
improve reasoning have been limited to solving mathematical problems and
focusing on visual graphical inputs, neglecting broader applications in general
video understanding.This paper proposes video-SALMONN-o1, the first open-source
reasoning-enhanced audio-visual LLM designed for general video understanding
tasks. To enhance its reasoning abilities, we develop a reasoning-intensive
dataset featuring challenging audio-visual questions with step-by-step
solutions. We also propose process direct preference optimization (pDPO), which
leverages contrastive step selection to achieve efficient step-level reward
modelling tailored for multimodal inputs. Additionally, we introduce RivaBench,
the first reasoning-intensive video understanding benchmark, featuring over
4,000 high-quality, expert-curated question-answer pairs across scenarios such
as standup comedy, academic presentations, and synthetic video detection.
video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision
baseline across different video reasoning benchmarks. Besides, pDPO achieves
6-8% improvements compared to the supervised fine-tuning model on RivaBench.
Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection
capabilities.
67b4147f7721b4fe4d2bd466
Guangzhi Sun
admin_assigned
2025-02-19T15:50:36.976Z
64ab544489aa67e4a2505eeb
/avatars/f1a9def3afbec2f8b89ef4450770d67e.svg
Guangzhi Sun
user
BrianatCambridge
67b4147f7721b4fe4d2bd467
Yudong Yang
admin_assigned
2025-02-19T15:50:43.942Z
66ea99f751c3bcd917ae1265
/avatars/17c4206c39d4da49d2516422c392e0ee.svg
Yudong Yang
user
yangyudong2020
67b4147f7721b4fe4d2bd468
Jimin Zhuang
admin_assigned
2025-02-19T15:50:50.168Z
66825d0e971dff9d3a24d912
/avatars/024b704a1f8c3aec20474e162ac326ad.svg
Jimin Zhuang
user
OctaAcid
67b4147f7721b4fe4d2bd469
Changli Tang
admin_assigned
2025-02-19T15:50:57.247Z
63770389cdcc1bf630870758
/avatars/6bc7c602e79688bc42a4c79ecf5b6d2d.svg
Changli Tang
user
Changli
67b4147f7721b4fe4d2bd46a
Yixuan Li
claimed_verified
2025-07-17T13:56:45.053Z
658404683925d68d7c604f6d
/avatars/0174a8d7b8ef7cd37b4a919a83884ed8.svg
Yixuan Li
user
DragonAura
67b4147f7721b4fe4d2bd46b
Wei Li
67b4147f7721b4fe4d2bd46c
Zejun MA
67b4147f7721b4fe4d2bd46d
Chao Zhang
video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model
While recent advancements in reasoning optimization have significantly
enhanced the capabilities of large language models (LLMs), existing efforts to
improve reasoning have been limited to solving mathematical problems and
focusing on visual graphical inputs, neglecting broader applications in general
video understanding.This paper proposes video-SALMONN-o1, the first open-source
reasoning-enhanced audio-visual LLM designed for general video understanding
tasks. To enhance its reasoning abilities, we develop a reasoning-intensive
dataset featuring challenging audio-visual questions with step-by-step
solutions. We also propose process direct preference optimization (pDPO), which
leverages contrastive step selection to achieve efficient step-level reward
modelling tailored for multimodal inputs. Additionally, we introduce RivaBench,
the first reasoning-intensive video understanding benchmark, featuring over
4,000 high-quality, expert-curated question-answer pairs across scenarios such
as standup comedy, academic presentations, and synthetic video detection.
video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision
baseline across different video reasoning benchmarks. Besides, pDPO achieves
6-8% improvements compared to the supervised fine-tuning model on RivaBench.
Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection
capabilities.
video-SALMONN-o1 is a reasoning-enhanced audio-visual LLM for general video understanding, utilizing a new dataset and optimization method, pDPO, to achieve improved accuracy and zero-shot capabilities.
reasoning optimization large language models LLMs audio-visual LLM video-SALMONN-o1 reasoning-intensive dataset process direct preference optimization pDPO contrastive step selection RivaBench video reasoning benchmarks LLaVA-OneVision zero-shot synthetic video detection