VisualAgentBench: Towards Large Multimodal Models as Visual Foundation
  Agents
Large Multimodal Models (LMMs) have ushered in a new era in artificial
intelligence, merging capabilities in both language and vision to form highly
capable Visual Foundation Agents. These agents are postulated to excel across a
myriad of tasks, potentially approaching general artificial intelligence.
However, existing benchmarks fail to sufficiently challenge or showcase the
full potential of LMMs in complex, real-world environments. To address this
gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering
benchmark specifically designed to train and evaluate LMMs as visual foundation
agents across diverse scenarios, including Embodied, Graphical User Interface,
and Visual Design, with tasks formulated to probe the depth of LMMs'
understanding and interaction capabilities. Through rigorous testing across
nine proprietary LMM APIs and eight open models, we demonstrate the
considerable yet still developing agent capabilities of these models.
Additionally, VAB constructs a trajectory training set constructed through
hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and
Human Demonstrations, promoting substantial performance improvements in LMMs
through behavior cloning. Our work not only aims to benchmark existing models
but also provides a solid foundation for future development into visual
foundation agents. Code, train \& test data, and part of fine-tuned open LMMs
are available at https://github.com/THUDM/VisualAgentBench.
66bad6638a3c4b82368bdea8
Xiao Liu
admin_assigned
2024-08-13T09:33:02.305Z
62f098602ca4d32a7cd87aba
https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
Xiao Liu
user
ShawLiu
66bad6638a3c4b82368bdea9
Tianjie Zhang
claimed_verified
2024-08-13T08:27:03.812Z
64dceda0f44d2407e60179f8
/avatars/726d5ed318b40ae886b7387a532217e4.svg
Tianjie Zhang
user
tianjiezhang
66bad6638a3c4b82368bdeaa
Yu Gu
claimed_verified
2024-10-14T14:08:32.411Z
5e7e595230dc073f817a2bb5
/avatars/d5ff36e45555d9e169cf56c845736444.svg
Yu Gu
user
entslscheia
66bad6638a3c4b82368bdeab
Iat Long Iong
66bad6638a3c4b82368bdeac
Yifan Xu
66bad6638a3c4b82368bdead
Xixuan Song
admin_assigned
2024-08-13T08:56:52.734Z
63f36b504745321de3510823
https://cdn-avatars.huggingface.co/v1/production/uploads/1679311624745-63f36b504745321de3510823.jpeg
Song XiXuan
user
DrSong
66bad6638a3c4b82368bdeae
Shudan Zhang
admin_assigned
2024-08-13T08:57:02.469Z
637883f3736e2989331da910
/avatars/f72b579724d2d43889907e8b732e79e6.svg
Shudan Zhang
user
zdaniel0222
66bad6638a3c4b82368bdeaf
Hanyu Lai
admin_assigned
2024-08-13T08:57:08.769Z
62ea2d48b493272c269e8f34
/avatars/89b215dafa503b51ab212a9b63c82aca.svg
Hanyu Lai
user
hanyullai
66bad6638a3c4b82368bdeb0
Xinyi Liu
66bad6638a3c4b82368bdeb1
Hanlin Zhao
admin_assigned
2024-08-13T08:58:12.909Z
647aa6121eff17116fa61213
/avatars/0735b857db37e59018fb5aa1dd231000.svg
zhaohanlin
user
ultrazhl
66bad6638a3c4b82368bdeb2
Jiadai Sun
admin_assigned
2024-08-13T08:58:20.293Z
64e82f2c9a3cd93b370e9936
/avatars/942e6008e8e78583c2750d678fcaadb1.svg
jiadaisun
user
jiadaisun
66bad6638a3c4b82368bdeb3
Xinyue Yang
66bad6638a3c4b82368bdeb4
Yu Yang
admin_assigned
2024-08-13T08:58:49.143Z
60a53adbf9b53404e7806277
https://cdn-avatars.huggingface.co/v1/production/uploads/1621441193991-noauth.jpeg
Yu Yang
user
yuyang
66bad6638a3c4b82368bdeb5
Zehan Qi
admin_assigned
2024-08-13T08:58:56.766Z
650ace9fa80a948447812fc5
/avatars/e01fe0d973008f5e3cd3c99f895b5946.svg
zehanqi
user
zehanqi
66bad6638a3c4b82368bdeb6
Shuntian Yao
admin_assigned
2024-08-13T08:59:04.146Z
6464c91ef4f3102bb6c7418a
/avatars/1e6ce129d8cd51e81d517e23ad9aee43.svg
Shuntian Yao
user
TMIP
66bad6638a3c4b82368bdeb7
Xueqiao Sun
66bad6638a3c4b82368bdeb8
Siyi Cheng
admin_assigned
2024-08-13T09:06:20.918Z
654cf05258c9d74272181118
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Dfx-hp3RLkP95gQtOM7T0.png
Siyi Cheng
user
LeakyKayO
66bad6638a3c4b82368bdeb9
Qinkai Zheng
admin_assigned
2024-08-13T09:06:27.960Z
6231576e92e83fd1179ac3f0
https://cdn-avatars.huggingface.co/v1/production/uploads/1664543160657-6231576e92e83fd1179ac3f0.jpeg
Qinkai Zheng
user
Stanislas
66bad6638a3c4b82368bdeba
Hao Yu
66bad6638a3c4b82368bdebb
Hanchen Zhang
66bad6638a3c4b82368bdebc
Wenyi Hong
admin_assigned
2024-08-13T09:09:28.794Z
62ecd24cb8764c7738ef2793
/avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
Wenyi Hong
user
wenyi
66bad6638a3c4b82368bdebd
Ming Ding
66bad6638a3c4b82368bdebe
Lihang Pan
admin_assigned
2024-08-13T09:09:38.155Z
6401838752fb66b80d209207
/avatars/c4da78b1fa53769ca9132528924b88b0.svg
Li hang pan
user
k789456789040
66bad6638a3c4b82368bdebf
Xiaotao Gu
claimed_verified
2025-07-02T09:25:09.903Z
6438bca4a5e10f6d58694b47
/avatars/3aeb25fbc73c5cab1265e13d11adfb76.svg
XG
user
xgeric
66bad6638a3c4b82368bdec0
Aohan Zeng
admin_assigned
2024-08-13T09:10:03.039Z
62dc173789b4cf157d36ebee
https://cdn-avatars.huggingface.co/v1/production/uploads/1659772051636-62dc173789b4cf157d36ebee.jpeg
Zeng Aohan
user
Sengxian
66bad6638a3c4b82368bdec1
Zhengxiao Du
admin_assigned
2024-08-13T09:10:10.310Z
63033dc4e1e7f0e03a5e1a31
https://cdn-avatars.huggingface.co/v1/production/uploads/1661157784937-63033dc4e1e7f0e03a5e1a31.jpeg
Zhengxiao Du
user
zxdu20
66bad6638a3c4b82368bdec2
Chan Hee Song
admin_assigned
2024-08-13T09:10:17.970Z
63d19365b30415240fd6515b
https://cdn-avatars.huggingface.co/v1/production/uploads/63d19365b30415240fd6515b/eOEYSsyDTfPTDrR6Cm5Jn.jpeg
Chan Hee Song
user
chanhee-luke
66bad6638a3c4b82368bdec3
Yu Su
claimed_verified
2024-10-14T19:21:57.847Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
66bad6638a3c4b82368bdec4
Yuxiao Dong
admin_assigned
2024-08-13T09:10:26.615Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
66bad6638a3c4b82368bdec5
Jie Tang
admin_assigned
2024-08-13T09:10:34.600Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation
  Agents
Large Multimodal Models (LMMs) have ushered in a new era in artificial
intelligence, merging capabilities in both language and vision to form highly
capable Visual Foundation Agents. These agents are postulated to excel across a
myriad of tasks, potentially approaching general artificial intelligence.
However, existing benchmarks fail to sufficiently challenge or showcase the
full potential of LMMs in complex, real-world environments. To address this
gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering
benchmark specifically designed to train and evaluate LMMs as visual foundation
agents across diverse scenarios, including Embodied, Graphical User Interface,
and Visual Design, with tasks formulated to probe the depth of LMMs'
understanding and interaction capabilities. Through rigorous testing across
nine proprietary LMM APIs and eight open models, we demonstrate the
considerable yet still developing agent capabilities of these models.
Additionally, VAB constructs a trajectory training set constructed through
hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and
Human Demonstrations, promoting substantial performance improvements in LMMs
through behavior cloning. Our work not only aims to benchmark existing models
but also provides a solid foundation for future development into visual
foundation agents. Code, train \& test data, and part of fine-tuned open LMMs
are available at https://github.com/THUDM/VisualAgentBench.
VisualAgentBench (VAB) is a benchmark designed to evaluate Large Multimodal Models (LMMs) across various real-world scenarios, including Embodied, Graphical User Interface, and Visual Design tasks, to assess their understanding and interaction capabilities.
Large Multimodal Models Visual Foundation Agents VisualAgentBench Embodied Graphical User Interface Visual Design Program-based Solvers LMM Agent Bootstrapping Human Demonstrations behavior cloning