MORSE-500: A Programmatically Controllable Video Benchmark to
  Stress-Test Multimodal Reasoning
Despite rapid advances in vision-language models (VLMs), current benchmarks
for multimodal reasoning fall short in three key dimensions. First, they
overwhelmingly rely on static images, failing to capture the temporal
complexity of real-world environments. Second, they narrowly focus on
mathematical problem-solving, neglecting the broader spectrum of reasoning
skills -- including abstract, physical, planning, spatial, and temporal
capabilities -- required for robust multimodal intelligence. Third, many
benchmarks quickly saturate, offering limited headroom for diagnosing failure
modes or measuring continued progress. We introduce MORSE-500 (Multimodal
Reasoning Stress-test Environment), a video benchmark composed of 500 fully
scripted clips with embedded questions spanning six complementary reasoning
categories. Each instance is programmatically generated using deterministic
Python scripts (via Manim, Matplotlib, MoviePy), generative video models, and
curated real footage. This script-driven design allows fine-grained control
over visual complexity, distractor density, and temporal dynamics -- enabling
difficulty to be scaled systematically as models improve. Unlike static
benchmarks that become obsolete once saturated, MORSE-500 is built to evolve:
its controllable generation pipeline supports the creation of arbitrarily
challenging new instances, making it ideally suited for stress-testing
next-generation models. Initial experiments with state-of-the-art systems --
including various Gemini 2.5 Pro and OpenAI o3 which represent the strongest
available at the time, alongside strong open-source models -- reveal
substantial performance gaps across all categories, with particularly large
deficits in abstract and planning tasks. We release the full dataset,
generation scripts, and evaluation harness to support transparent,
reproducible, and forward-looking multimodal reasoning research.
6846657d3ec10bdd8ab4daca
Zikui Cai
claimed_verified
2025-06-09T10:11:17.551Z
630bc5ae86b8b9904c33e94b
/avatars/b176d9b1691c05cc941409dd6c2b2228.svg
Zikui Cai
user
Zikui
6846657d3ec10bdd8ab4dacb
Andrew Wang
6846657d3ec10bdd8ab4dacc
Anirudh Satheesh
claimed_verified
2025-06-11T08:36:45.813Z
63e01e7ff351dc074570af2c
/avatars/dc2c5027cab3d93d215856904ea18fd3.svg
Anirudh Satheesh
user
asatheesh
6846657d3ec10bdd8ab4dacd
Ankit Nakhawa
6846657d3ec10bdd8ab4dace
Hyunwoo Jae
6846657d3ec10bdd8ab4dacf
Keenan Powell
6846657d3ec10bdd8ab4dad0
Minghui Liu
6846657d3ec10bdd8ab4dad1
Neel Jay
claimed_verified
2025-06-12T12:43:05.567Z
64dec56bd90f5eee7a417f88
/avatars/590948de7d3c80f613f9adf4dfea92af.svg
Neel Jay
user
neeljaycs
6846657d3ec10bdd8ab4dad2
Sungbin Oh
6846657d3ec10bdd8ab4dad3
Xiyao Wang
claimed_verified
2025-06-10T09:22:06.068Z
655fed9fdef5905d38b84af3
/avatars/2cda4182dfd11a1e94743639e62328ea.svg
Xiyao Wang
user
russwang
6846657d3ec10bdd8ab4dad4
Yongyuan Liang
6846657d3ec10bdd8ab4dad5
Tom Goldstein
6846657d3ec10bdd8ab4dad6
Furong Huang
MORSE-500: A Programmatically Controllable Video Benchmark to
  Stress-Test Multimodal Reasoning
Despite rapid advances in vision-language models (VLMs), current benchmarks
for multimodal reasoning fall short in three key dimensions. First, they
overwhelmingly rely on static images, failing to capture the temporal
complexity of real-world environments. Second, they narrowly focus on
mathematical problem-solving, neglecting the broader spectrum of reasoning
skills -- including abstract, physical, planning, spatial, and temporal
capabilities -- required for robust multimodal intelligence. Third, many
benchmarks quickly saturate, offering limited headroom for diagnosing failure
modes or measuring continued progress. We introduce MORSE-500 (Multimodal
Reasoning Stress-test Environment), a video benchmark composed of 500 fully
scripted clips with embedded questions spanning six complementary reasoning
categories. Each instance is programmatically generated using deterministic
Python scripts (via Manim, Matplotlib, MoviePy), generative video models, and
curated real footage. This script-driven design allows fine-grained control
over visual complexity, distractor density, and temporal dynamics -- enabling
difficulty to be scaled systematically as models improve. Unlike static
benchmarks that become obsolete once saturated, MORSE-500 is built to evolve:
its controllable generation pipeline supports the creation of arbitrarily
challenging new instances, making it ideally suited for stress-testing
next-generation models. Initial experiments with state-of-the-art systems --
including various Gemini 2.5 Pro and OpenAI o3 which represent the strongest
available at the time, alongside strong open-source models -- reveal
substantial performance gaps across all categories, with particularly large
deficits in abstract and planning tasks. We release the full dataset,
generation scripts, and evaluation harness to support transparent,
reproducible, and forward-looking multimodal reasoning research.
MORSE-500, a video benchmark with 500 scripted clips, evaluates multimodal reasoning across six categories, highlighting performance gaps in abstract and planning tasks.
Vision-language models MORSE-500 multimodal reasoning video benchmark scripted clips reasoning categories Manim Matplotlib MoviePy generative video models controllable generation spatial capabilities temporal capabilities abstract reasoning planning tasks