AgentBench: Evaluating LLMs as Agents
Large Language Models (LLMs) are becoming increasingly smart and autonomous,
targeting real-world pragmatic missions beyond traditional NLP tasks. As a
result, there has been an urgent need to evaluate LLMs as agents on challenging
tasks in interactive environments. We present AgentBench, a multi-dimensional
evolving benchmark that currently consists of 8 distinct environments to assess
LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
open-ended generation setting. Our extensive test over 25 LLMs (including APIs
and open-sourced models) shows that, while top commercial LLMs present a strong
ability of acting as agents in complex environments, there is a significant
disparity in performance between them and open-sourced competitors. It also
serves as a component of an ongoing project with wider coverage and deeper
consideration towards systematic LLM evaluation. Datasets, environments, and an
integrated evaluation package for AgentBench are released at
https://github.com/THUDM/AgentBench
64d1af361ed6649d70c8b57f
Xiao Liu
claimed_verified
2023-08-09T00:42:59.125Z
62f098602ca4d32a7cd87aba
https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
Xiao Liu
user
ShawLiu
64d1af361ed6649d70c8b580
Hao Yu
64d1af361ed6649d70c8b581
Hanchen Zhang
64d1af361ed6649d70c8b582
Yifan Xu
64d1af361ed6649d70c8b583
Xuanyu Lei
64d1af361ed6649d70c8b584
Hanyu Lai
admin_assigned
2023-08-08T14:53:49.378Z
62ea2d48b493272c269e8f34
/avatars/89b215dafa503b51ab212a9b63c82aca.svg
Hanyu Lai
user
hanyullai
64d1af361ed6649d70c8b585
Yu Gu
claimed_verified
2023-08-09T10:52:09.435Z
5e7e595230dc073f817a2bb5
/avatars/d5ff36e45555d9e169cf56c845736444.svg
Yu Gu
user
entslscheia
64d1af361ed6649d70c8b586
Hangliang Ding
64d1af361ed6649d70c8b587
Kaiwen Men
admin_assigned
2023-08-08T14:54:22.565Z
5f37e3a97e5835433862179b
https://cdn-avatars.huggingface.co/v1/production/uploads/1597497791282-noauth.jpeg
Kaiwen Men
user
mkw18
64d1af361ed6649d70c8b588
Kejuan Yang
64d1af361ed6649d70c8b589
Shudan Zhang
64d1af361ed6649d70c8b58a
Xiang Deng
admin_assigned
2023-08-08T14:55:14.300Z
60bd427529800c34660339f6
/avatars/1dbc89402c7e38eb131d43dbc28022ab.svg
Xiang Deng
user
XiangD-OSU
64d1af361ed6649d70c8b58b
Aohan Zeng
admin_assigned
2023-08-08T14:55:30.534Z
62dc173789b4cf157d36ebee
https://cdn-avatars.huggingface.co/v1/production/uploads/1659772051636-62dc173789b4cf157d36ebee.jpeg
Zeng Aohan
user
Sengxian
64d1af361ed6649d70c8b58c
Zhengxiao Du
admin_assigned
2023-08-08T14:55:43.918Z
63033dc4e1e7f0e03a5e1a31
https://cdn-avatars.huggingface.co/v1/production/uploads/1661157784937-63033dc4e1e7f0e03a5e1a31.jpeg
Zhengxiao Du
user
zxdu20
64d1af361ed6649d70c8b58d
Chenhui Zhang
admin_assigned
2023-08-08T14:56:31.453Z
64522233ea94bf023430dd95
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CVDqDeJ_fLTULhCTTSogb.png
Chenhui Zhang
user
danielz01
64d1af361ed6649d70c8b58e
Sheng Shen
claimed_verified
2023-08-09T10:52:13.279Z
614b532936c340687d05a585
/avatars/23914c474bb737bbdf2f49acacad6ef7.svg
Sheng Shen
user
shengs
64d1af361ed6649d70c8b58f
Tianjun Zhang
admin_assigned
2023-08-08T14:56:55.127Z
6374cd6b6ea8da14f8fef8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
Tianjun Zhang
user
tianjunz
64d1af361ed6649d70c8b590
Yu Su
claimed_verified
2024-10-14T19:22:19.635Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
64d1af361ed6649d70c8b591
Huan Sun
admin_assigned
2023-08-08T14:57:16.851Z
6127b327699a5d2cde0b44dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1629991702069-noauth.png
Huan Sun
user
huansun
64d1af361ed6649d70c8b592
Minlie Huang
64d1af361ed6649d70c8b593
Yuxiao Dong
admin_assigned
2023-08-08T14:57:41.866Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
64d1af361ed6649d70c8b594
Jie Tang
admin_assigned
2023-08-08T14:57:58.148Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
AgentBench: Evaluating LLMs as Agents
Large Language Models (LLMs) are becoming increasingly smart and autonomous,
targeting real-world pragmatic missions beyond traditional NLP tasks. As a
result, there has been an urgent need to evaluate LLMs as agents on challenging
tasks in interactive environments. We present AgentBench, a multi-dimensional
evolving benchmark that currently consists of 8 distinct environments to assess
LLM-as-Agent's reasoning and decision-making abilities in a multi-turn
open-ended generation setting. Our extensive test over 25 LLMs (including APIs
and open-sourced models) shows that, while top commercial LLMs present a strong
ability of acting as agents in complex environments, there is a significant
disparity in performance between them and open-sourced competitors. It also
serves as a component of an ongoing project with wider coverage and deeper
consideration towards systematic LLM evaluation. Datasets, environments, and an
integrated evaluation package for AgentBench are released at
https://github.com/THUDM/AgentBench
AgentBench is a multi-dimensional benchmark for evaluating LLMs as autonomous agents across various interactive environments, highlighting performance differences between commercial and open-source models.
Large Language Models (LLMs) AgentBench multi-turn open-ended generation evaluation benchmark complex environments systematic LLM evaluation datasets interactive environments