SiLVR: A Simple Language-based Video Reasoning Framework
Recent advances in test-time optimization have led to remarkable reasoning
capabilities in Large Language Models (LLMs), enabling them to solve highly
complex problems in math and coding. However, the reasoning capabilities of
multimodal LLMs (MLLMs) still significantly lag, especially for complex
video-language tasks. To address this issue, we present SiLVR, a Simple
Language-based Video Reasoning framework that decomposes complex video
understanding into two stages. In the first stage, SiLVR transforms raw video
into language-based representations using multisensory inputs, such as short
clip captions and audio/speech subtitles. In the second stage, language
descriptions are fed into a powerful reasoning LLM to solve complex
video-language understanding tasks. To handle long-context multisensory inputs,
we use an adaptive token reduction scheme, which dynamically determines the
temporal granularity with which to sample the tokens. Our simple, modular, and
training-free video reasoning framework achieves the best-reported results on
Video-MME (long), Video-MMMU (comprehension), Video-MMLU, CGBench, and EgoLife.
Furthermore, our empirical study focused on video reasoning capabilities shows
that, despite not being explicitly trained on video, strong reasoning LLMs can
effectively aggregate multisensory input information from video, speech, and
audio for complex temporal, causal, long-context, and knowledge acquisition
reasoning tasks in video. Code is available at https://github.com/CeeZh/SILVR.
683dfea7a6815c77ac987005
Ce Zhang
claimed_verified
2025-06-05T08:06:53.492Z
6519fd9416f0c073c5c551c3
/avatars/597ae2f69e1fa2b4bd32a69d1d999b55.svg
Ce Zhang
user
ceezh
683dfea7a6815c77ac987006
Yan-Bo Lin
claimed_verified
2025-06-03T08:47:27.971Z
63a715c38729ce5b5640b80b
/avatars/5dc0a167c5dae7756a77d79cfd6334b2.svg
yb
user
genjib
683dfea7a6815c77ac987007
Ziyang Wang
683dfea7a6815c77ac987008
Mohit Bansal
683dfea7a6815c77ac987009
Gedas Bertasius
SiLVR: A Simple Language-based Video Reasoning Framework
Recent advances in test-time optimization have led to remarkable reasoning
capabilities in Large Language Models (LLMs), enabling them to solve highly
complex problems in math and coding. However, the reasoning capabilities of
multimodal LLMs (MLLMs) still significantly lag, especially for complex
video-language tasks. To address this issue, we present SiLVR, a Simple
Language-based Video Reasoning framework that decomposes complex video
understanding into two stages. In the first stage, SiLVR transforms raw video
into language-based representations using multisensory inputs, such as short
clip captions and audio/speech subtitles. In the second stage, language
descriptions are fed into a powerful reasoning LLM to solve complex
video-language understanding tasks. To handle long-context multisensory inputs,
we use an adaptive token reduction scheme, which dynamically determines the
temporal granularity with which to sample the tokens. Our simple, modular, and
training-free video reasoning framework achieves the best-reported results on
Video-MME (long), Video-MMMU (comprehension), Video-MMLU, CGBench, and EgoLife.
Furthermore, our empirical study focused on video reasoning capabilities shows
that, despite not being explicitly trained on video, strong reasoning LLMs can
effectively aggregate multisensory input information from video, speech, and
audio for complex temporal, causal, long-context, and knowledge acquisition
reasoning tasks in video. Code is available at https://github.com/CeeZh/SILVR.
SiLVR, a language-based framework, enhances multimodal LLMs' video reasoning by leveraging adaptive token reduction, achieving top results on several benchmarks.
Large Language Models multimodal LLMs video-language tasks Language-based Video Reasoning multisensory inputs adaptive token reduction temporal granularity Video-MME Video-MMMU Video-MMLU CGBench EgoLife