title: VisualAgentBench: Towards Large Multimodal Models as Visual Foundation
    Agents
summary: Large Multimodal Models (LMMs) have ushered in a new era in artificial
  intelligence, merging capabilities in both language and vision to form highly
  capable Visual Foundation Agents. These agents are postulated to excel across a
  myriad of tasks, potentially approaching general artificial intelligence.
  However, existing benchmarks fail to sufficiently challenge or showcase the
  full potential of LMMs in complex, real-world environments. To address this
  gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering
  benchmark specifically designed to train and evaluate LMMs as visual foundation
  agents across diverse scenarios, including Embodied, Graphical User Interface,
  and Visual Design, with tasks formulated to probe the depth of LMMs'
  understanding and interaction capabilities. Through rigorous testing across
  nine proprietary LMM APIs and eight open models, we demonstrate the
  considerable yet still developing agent capabilities of these models.
  Additionally, VAB constructs a trajectory training set constructed through
  hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and
  Human Demonstrations, promoting substantial performance improvements in LMMs
  through behavior cloning. Our work not only aims to benchmark existing models
  but also provides a solid foundation for future development into visual
  foundation agents. Code, train \& test data, and part of fine-tuned open LMMs
  are available at https://github.com/THUDM/VisualAgentBench.
paper_authors._id: 66bad6638a3c4b82368bdea8
paper_authors.name: Xiao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:33:02.305Z
paper_authors._id: 62f098602ca4d32a7cd87aba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: ShawLiu
paper_authors._id: 66bad6638a3c4b82368bdea9
paper_authors.name: Tianjie Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T08:27:03.812Z
paper_authors._id: 64dceda0f44d2407e60179f8
paper_authors.avatarUrl: /avatars/726d5ed318b40ae886b7387a532217e4.svg
paper_authors.fullname: Tianjie Zhang
paper_authors.type: user
paper_authors.user: tianjiezhang
paper_authors._id: 66bad6638a3c4b82368bdeaa
paper_authors.name: Yu Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T14:08:32.411Z
paper_authors._id: 5e7e595230dc073f817a2bb5
paper_authors.avatarUrl: /avatars/d5ff36e45555d9e169cf56c845736444.svg
paper_authors.fullname: Yu Gu
paper_authors.type: user
paper_authors.user: entslscheia
paper_authors._id: 66bad6638a3c4b82368bdeab
paper_authors.name: Iat Long Iong
paper_authors._id: 66bad6638a3c4b82368bdeac
paper_authors.name: Yifan Xu
paper_authors._id: 66bad6638a3c4b82368bdead
paper_authors.name: Xixuan Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:56:52.734Z
paper_authors._id: 63f36b504745321de3510823
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679311624745-63f36b504745321de3510823.jpeg
paper_authors.fullname: Song XiXuan
paper_authors.type: user
paper_authors.user: DrSong
paper_authors._id: 66bad6638a3c4b82368bdeae
paper_authors.name: Shudan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:57:02.469Z
paper_authors._id: 637883f3736e2989331da910
paper_authors.avatarUrl: /avatars/f72b579724d2d43889907e8b732e79e6.svg
paper_authors.fullname: Shudan Zhang
paper_authors.type: user
paper_authors.user: zdaniel0222
paper_authors._id: 66bad6638a3c4b82368bdeaf
paper_authors.name: Hanyu Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:57:08.769Z
paper_authors._id: 62ea2d48b493272c269e8f34
paper_authors.avatarUrl: /avatars/89b215dafa503b51ab212a9b63c82aca.svg
paper_authors.fullname: Hanyu Lai
paper_authors.type: user
paper_authors.user: hanyullai
paper_authors._id: 66bad6638a3c4b82368bdeb0
paper_authors.name: Xinyi Liu
paper_authors._id: 66bad6638a3c4b82368bdeb1
paper_authors.name: Hanlin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:58:12.909Z
paper_authors._id: 647aa6121eff17116fa61213
paper_authors.avatarUrl: /avatars/0735b857db37e59018fb5aa1dd231000.svg
paper_authors.fullname: zhaohanlin
paper_authors.type: user
paper_authors.user: ultrazhl
paper_authors._id: 66bad6638a3c4b82368bdeb2
paper_authors.name: Jiadai Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:58:20.293Z
paper_authors._id: 64e82f2c9a3cd93b370e9936
paper_authors.avatarUrl: /avatars/942e6008e8e78583c2750d678fcaadb1.svg
paper_authors.fullname: jiadaisun
paper_authors.type: user
paper_authors.user: jiadaisun
paper_authors._id: 66bad6638a3c4b82368bdeb3
paper_authors.name: Xinyue Yang
paper_authors._id: 66bad6638a3c4b82368bdeb4
paper_authors.name: Yu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:58:49.143Z
paper_authors._id: 60a53adbf9b53404e7806277
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1621441193991-noauth.jpeg
paper_authors.fullname: Yu Yang
paper_authors.type: user
paper_authors.user: yuyang
paper_authors._id: 66bad6638a3c4b82368bdeb5
paper_authors.name: Zehan Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:58:56.766Z
paper_authors._id: 650ace9fa80a948447812fc5
paper_authors.avatarUrl: /avatars/e01fe0d973008f5e3cd3c99f895b5946.svg
paper_authors.fullname: zehanqi
paper_authors.type: user
paper_authors.user: zehanqi
paper_authors._id: 66bad6638a3c4b82368bdeb6
paper_authors.name: Shuntian Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T08:59:04.146Z
paper_authors._id: 6464c91ef4f3102bb6c7418a
paper_authors.avatarUrl: /avatars/1e6ce129d8cd51e81d517e23ad9aee43.svg
paper_authors.fullname: Shuntian Yao
paper_authors.type: user
paper_authors.user: TMIP
paper_authors._id: 66bad6638a3c4b82368bdeb7
paper_authors.name: Xueqiao Sun
paper_authors._id: 66bad6638a3c4b82368bdeb8
paper_authors.name: Siyi Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:06:20.918Z
paper_authors._id: 654cf05258c9d74272181118
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Dfx-hp3RLkP95gQtOM7T0.png
paper_authors.fullname: Siyi Cheng
paper_authors.type: user
paper_authors.user: LeakyKayO
paper_authors._id: 66bad6638a3c4b82368bdeb9
paper_authors.name: Qinkai Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:06:27.960Z
paper_authors._id: 6231576e92e83fd1179ac3f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664543160657-6231576e92e83fd1179ac3f0.jpeg
paper_authors.fullname: Qinkai Zheng
paper_authors.type: user
paper_authors.user: Stanislas
paper_authors._id: 66bad6638a3c4b82368bdeba
paper_authors.name: Hao Yu
paper_authors._id: 66bad6638a3c4b82368bdebb
paper_authors.name: Hanchen Zhang
paper_authors._id: 66bad6638a3c4b82368bdebc
paper_authors.name: Wenyi Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:09:28.794Z
paper_authors._id: 62ecd24cb8764c7738ef2793
paper_authors.avatarUrl: /avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
paper_authors.fullname: Wenyi Hong
paper_authors.type: user
paper_authors.user: wenyi
paper_authors._id: 66bad6638a3c4b82368bdebd
paper_authors.name: Ming Ding
paper_authors._id: 66bad6638a3c4b82368bdebe
paper_authors.name: Lihang Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:09:38.155Z
paper_authors._id: 6401838752fb66b80d209207
paper_authors.avatarUrl: /avatars/c4da78b1fa53769ca9132528924b88b0.svg
paper_authors.fullname: Li hang pan
paper_authors.type: user
paper_authors.user: k789456789040
paper_authors._id: 66bad6638a3c4b82368bdebf
paper_authors.name: Xiaotao Gu
paper_authors._id: 66bad6638a3c4b82368bdec0
paper_authors.name: Aohan Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:10:03.039Z
paper_authors._id: 62dc173789b4cf157d36ebee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659772051636-62dc173789b4cf157d36ebee.jpeg
paper_authors.fullname: Zeng Aohan
paper_authors.type: user
paper_authors.user: Sengxian
paper_authors._id: 66bad6638a3c4b82368bdec1
paper_authors.name: Zhengxiao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:10:10.310Z
paper_authors._id: 63033dc4e1e7f0e03a5e1a31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661157784937-63033dc4e1e7f0e03a5e1a31.jpeg
paper_authors.fullname: Zhengxiao Du
paper_authors.type: user
paper_authors.user: zxdu20
paper_authors._id: 66bad6638a3c4b82368bdec2
paper_authors.name: Chan Hee Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:10:17.970Z
paper_authors._id: 63d19365b30415240fd6515b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d19365b30415240fd6515b/eOEYSsyDTfPTDrR6Cm5Jn.jpeg
paper_authors.fullname: Chan Hee Song
paper_authors.type: user
paper_authors.user: chanhee-luke
paper_authors._id: 66bad6638a3c4b82368bdec3
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:21:57.847Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_authors._id: 66bad6638a3c4b82368bdec4
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:10:26.615Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 66bad6638a3c4b82368bdec5
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:10:34.600Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_title: VisualAgentBench: Towards Large Multimodal Models as Visual Foundation
    Agents
paper_summary: Large Multimodal Models (LMMs) have ushered in a new era in artificial
  intelligence, merging capabilities in both language and vision to form highly
  capable Visual Foundation Agents. These agents are postulated to excel across a
  myriad of tasks, potentially approaching general artificial intelligence.
  However, existing benchmarks fail to sufficiently challenge or showcase the
  full potential of LMMs in complex, real-world environments. To address this
  gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering
  benchmark specifically designed to train and evaluate LMMs as visual foundation
  agents across diverse scenarios, including Embodied, Graphical User Interface,
  and Visual Design, with tasks formulated to probe the depth of LMMs'
  understanding and interaction capabilities. Through rigorous testing across
  nine proprietary LMM APIs and eight open models, we demonstrate the
  considerable yet still developing agent capabilities of these models.
  Additionally, VAB constructs a trajectory training set constructed through
  hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and
  Human Demonstrations, promoting substantial performance improvements in LMMs
  through behavior cloning. Our work not only aims to benchmark existing models
  but also provides a solid foundation for future development into visual
  foundation agents. Code, train \& test data, and part of fine-tuned open LMMs
  are available at https://github.com/THUDM/VisualAgentBench.
paper_ai_summary: VisualAgentBench (VAB) is a benchmark designed to evaluate Large Multimodal Models (LMMs) across various real-world scenarios, including Embodied, Graphical User Interface, and Visual Design tasks, to assess their understanding and interaction capabilities.
paper_ai_keywords: Large Multimodal Models Visual Foundation Agents VisualAgentBench Embodied Graphical User Interface Visual Design Program-based Solvers LMM Agent Bootstrapping Human Demonstrations behavior cloning