VS-Bench: Evaluating VLMs for Strategic Reasoning and Decision-Making in
  Multi-Agent Environments
Recent advancements in Vision Language Models (VLMs) have expanded their
capabilities to interactive agent tasks, yet existing benchmarks remain limited
to single-agent or text-only environments. In contrast, real-world scenarios
often involve multiple agents interacting within rich visual and linguistic
contexts, posing challenges with both multimodal observations and strategic
interactions. To bridge this gap, we introduce Visual Strategic Bench
(VS-Bench), a multimodal benchmark that evaluates VLMs for strategic reasoning
and decision-making in multi-agent environments. VS-Bench comprises eight
vision-grounded environments spanning cooperative, competitive, and
mixed-motive interactions, designed to assess agents' ability to predict
others' future moves and optimize for long-term objectives. We consider two
complementary evaluation dimensions, including offline evaluation of strategic
reasoning by next-action prediction accuracy and online evaluation of
decision-making by normalized episode return. Extensive experiments of fourteen
leading VLMs reveal a significant gap between current models and optimal
performance, with the best models attaining 47.8% prediction accuracy and 24.3%
normalized return. We further conduct in-depth analyses on multimodal
observations, test-time scaling, social behaviors, and failure cases of VLM
agents. By standardizing the evaluation and highlighting the limitations of
existing models, we envision VS-Bench as a foundation for future research on
strategic multimodal agents. Code and data are available at
https://vs-bench.github.io.
683fa95ea0770843560c7ae3
Zelai Xu
extracted_confirmed
2025-06-05T12:16:05.641Z
653a5b0f7c01c693a16dd184
/avatars/4b43d88709dc8037250404452e81adcf.svg
Zelai Xu
user
zelaix
683fa95ea0770843560c7ae4
Zhexuan Xu
claimed_verified
2025-06-05T12:42:09.925Z
683984ae71d8234f74f2166d
/avatars/fd8b37f6a12ef24ae37c0d127fc3f695.svg
xzxuan
user
xzxuan
683fa95ea0770843560c7ae5
Xiangmin Yi
claimed_verified
2025-06-05T17:08:16.562Z
67c6fd7dc46aff331d05e49a
/avatars/a1876ec436f9d15a51c9f5b5ca333f25.svg
Xiangmin Yi
user
lazyyxm
683fa95ea0770843560c7ae6
Huining Yuan
admin_assigned
2025-06-04T10:26:15.430Z
683fb7bb7c8d720be437948d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pWekQrjYM6lw9bDjvLHg7.png
Huining Yuan
user
HuiningYuan
683fa95ea0770843560c7ae7
Xinlei Chen
admin_assigned
2025-06-05T07:41:40.125Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
683fa95ea0770843560c7ae8
Yi Wu
683fa95ea0770843560c7ae9
Chao Yu
683fa95ea0770843560c7aea
Yu Wang
VS-Bench: Evaluating VLMs for Strategic Reasoning and Decision-Making in
  Multi-Agent Environments
Recent advancements in Vision Language Models (VLMs) have expanded their
capabilities to interactive agent tasks, yet existing benchmarks remain limited
to single-agent or text-only environments. In contrast, real-world scenarios
often involve multiple agents interacting within rich visual and linguistic
contexts, posing challenges with both multimodal observations and strategic
interactions. To bridge this gap, we introduce Visual Strategic Bench
(VS-Bench), a multimodal benchmark that evaluates VLMs for strategic reasoning
and decision-making in multi-agent environments. VS-Bench comprises eight
vision-grounded environments spanning cooperative, competitive, and
mixed-motive interactions, designed to assess agents' ability to predict
others' future moves and optimize for long-term objectives. We consider two
complementary evaluation dimensions, including offline evaluation of strategic
reasoning by next-action prediction accuracy and online evaluation of
decision-making by normalized episode return. Extensive experiments of fourteen
leading VLMs reveal a significant gap between current models and optimal
performance, with the best models attaining 47.8% prediction accuracy and 24.3%
normalized return. We further conduct in-depth analyses on multimodal
observations, test-time scaling, social behaviors, and failure cases of VLM
agents. By standardizing the evaluation and highlighting the limitations of
existing models, we envision VS-Bench as a foundation for future research on
strategic multimodal agents. Code and data are available at
https://vs-bench.github.io.
VS-Bench is a multimodal benchmark designed to evaluate Vision Language Models' strategic reasoning and decision-making in complex multi-agent environments.
Vision Language Models VS-Bench multimodal benchmark strategic reasoning decision-making multi-agent environments vision-grounded environments cooperative competitive mixed-motive interactions next-action prediction normalized episode return multimodal observations test-time scaling social behaviors failure cases