title: AgentBench: Evaluating LLMs as Agents
summary: Large Language Models (LLMs) are becoming increasingly smart and autonomous,
  targeting real-world pragmatic missions beyond traditional NLP tasks. As a
  result, there has been an urgent need to evaluate LLMs as agents on challenging
  tasks in interactive environments. We present AgentBench, a multi-dimensional
  evolving benchmark that currently consists of 8 distinct environments to assess
  LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
  open-ended generation setting. Our extensive test over 25 LLMs (including APIs
  and open-sourced models) shows that, while top commercial LLMs present a strong
  ability of acting as agents in complex environments, there is a significant
  disparity in performance between them and open-sourced competitors. It also
  serves as a component of an ongoing project with wider coverage and deeper
  consideration towards systematic LLM evaluation. Datasets, environments, and an
  integrated evaluation package for AgentBench are released at
  https://github.com/THUDM/AgentBench
paper_authors._id: 64d1af361ed6649d70c8b57f
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-09T00:42:59.125Z
paper_authors._id: 62f098602ca4d32a7cd87aba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: ShawLiu
paper_authors._id: 64d1af361ed6649d70c8b580
paper_authors.name: Hao Yu
paper_authors._id: 64d1af361ed6649d70c8b581
paper_authors.name: Hanchen Zhang
paper_authors._id: 64d1af361ed6649d70c8b582
paper_authors.name: Yifan Xu
paper_authors._id: 64d1af361ed6649d70c8b583
paper_authors.name: Xuanyu Lei
paper_authors._id: 64d1af361ed6649d70c8b584
paper_authors.name: Hanyu Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:53:49.378Z
paper_authors._id: 62ea2d48b493272c269e8f34
paper_authors.avatarUrl: /avatars/89b215dafa503b51ab212a9b63c82aca.svg
paper_authors.fullname: Hanyu Lai
paper_authors.type: user
paper_authors.user: hanyullai
paper_authors._id: 64d1af361ed6649d70c8b585
paper_authors.name: Yu Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-09T10:52:09.435Z
paper_authors._id: 5e7e595230dc073f817a2bb5
paper_authors.avatarUrl: /avatars/d5ff36e45555d9e169cf56c845736444.svg
paper_authors.fullname: Yu Gu
paper_authors.type: user
paper_authors.user: entslscheia
paper_authors._id: 64d1af361ed6649d70c8b586
paper_authors.name: Hangliang Ding
paper_authors._id: 64d1af361ed6649d70c8b587
paper_authors.name: Kaiwen Men
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:54:22.565Z
paper_authors._id: 5f37e3a97e5835433862179b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1597497791282-noauth.jpeg
paper_authors.fullname: Kaiwen Men
paper_authors.type: user
paper_authors.user: mkw18
paper_authors._id: 64d1af361ed6649d70c8b588
paper_authors.name: Kejuan Yang
paper_authors._id: 64d1af361ed6649d70c8b589
paper_authors.name: Shudan Zhang
paper_authors._id: 64d1af361ed6649d70c8b58a
paper_authors.name: Xiang Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:55:14.300Z
paper_authors._id: 60bd427529800c34660339f6
paper_authors.avatarUrl: /avatars/1dbc89402c7e38eb131d43dbc28022ab.svg
paper_authors.fullname: Xiang Deng
paper_authors.type: user
paper_authors.user: XiangD-OSU
paper_authors._id: 64d1af361ed6649d70c8b58b
paper_authors.name: Aohan Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:55:30.534Z
paper_authors._id: 62dc173789b4cf157d36ebee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659772051636-62dc173789b4cf157d36ebee.jpeg
paper_authors.fullname: Zeng Aohan
paper_authors.type: user
paper_authors.user: Sengxian
paper_authors._id: 64d1af361ed6649d70c8b58c
paper_authors.name: Zhengxiao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:55:43.918Z
paper_authors._id: 63033dc4e1e7f0e03a5e1a31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661157784937-63033dc4e1e7f0e03a5e1a31.jpeg
paper_authors.fullname: Zhengxiao Du
paper_authors.type: user
paper_authors.user: zxdu20
paper_authors._id: 64d1af361ed6649d70c8b58d
paper_authors.name: Chenhui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:56:31.453Z
paper_authors._id: 64522233ea94bf023430dd95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CVDqDeJ_fLTULhCTTSogb.png
paper_authors.fullname: Chenhui Zhang
paper_authors.type: user
paper_authors.user: danielz01
paper_authors._id: 64d1af361ed6649d70c8b58e
paper_authors.name: Sheng Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-09T10:52:13.279Z
paper_authors._id: 614b532936c340687d05a585
paper_authors.avatarUrl: /avatars/23914c474bb737bbdf2f49acacad6ef7.svg
paper_authors.fullname: Sheng Shen
paper_authors.type: user
paper_authors.user: shengs
paper_authors._id: 64d1af361ed6649d70c8b58f
paper_authors.name: Tianjun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:56:55.127Z
paper_authors._id: 6374cd6b6ea8da14f8fef8dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
paper_authors.fullname: Tianjun Zhang
paper_authors.type: user
paper_authors.user: tianjunz
paper_authors._id: 64d1af361ed6649d70c8b590
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:22:19.635Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/grnmnxO1foz6aRRJ_A2NU.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_authors._id: 64d1af361ed6649d70c8b591
paper_authors.name: Huan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:57:16.851Z
paper_authors._id: 6127b327699a5d2cde0b44dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629991702069-noauth.png
paper_authors.fullname: Huan Sun
paper_authors.type: user
paper_authors.user: huansun
paper_authors._id: 64d1af361ed6649d70c8b592
paper_authors.name: Minlie Huang
paper_authors._id: 64d1af361ed6649d70c8b593
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:57:41.866Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 64d1af361ed6649d70c8b594
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:57:58.148Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_title: AgentBench: Evaluating LLMs as Agents
paper_summary: Large Language Models (LLMs) are becoming increasingly smart and autonomous,
  targeting real-world pragmatic missions beyond traditional NLP tasks. As a
  result, there has been an urgent need to evaluate LLMs as agents on challenging
  tasks in interactive environments. We present AgentBench, a multi-dimensional
  evolving benchmark that currently consists of 8 distinct environments to assess
  LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
  open-ended generation setting. Our extensive test over 25 LLMs (including APIs
  and open-sourced models) shows that, while top commercial LLMs present a strong
  ability of acting as agents in complex environments, there is a significant
  disparity in performance between them and open-sourced competitors. It also
  serves as a component of an ongoing project with wider coverage and deeper
  consideration towards systematic LLM evaluation. Datasets, environments, and an
  integrated evaluation package for AgentBench are released at
  https://github.com/THUDM/AgentBench
paper_ai_keywords: Large Language Models (LLMs) AgentBench neighborhood benchmark multi-turn open-ended generation LLM-as-Agent reasoning decision-making interactive environments commercial LLMs open-sourced models systematic LLM evaluation