MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Recent advances in large language models (LLMs) have increased the demand for
comprehensive benchmarks to evaluate their capabilities as human-like agents.
Existing benchmarks, while useful, often focus on specific application
scenarios, emphasizing task completion but failing to dissect the underlying
skills that drive these outcomes. This lack of granularity makes it difficult
to deeply discern where failures stem from. Additionally, setting up these
environments requires considerable effort, and issues of unreliability and
reproducibility sometimes arise, especially in interactive tasks. To address
these limitations, we introduce the Massive Multitask Agent Understanding
(MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need
for complex environment setups. It evaluates models across five domains,
including teal{Tool-use}, teal{Directed Acyclic Graph
(DAG) QA}, teal{Data Science and Machine Learning coding},
teal{Contest-level programming} and teal{Mathematics},
and covers five essential capabilities: orange{Understanding},
orange{Reasoning}, orange{Planning},
orange{Problem-solving}, and orange{Self-correction}.
With a total of 20 meticulously designed tasks encompassing over 3K distinct
prompts, MMAU provides a comprehensive framework for evaluating the strengths
and limitations of LLM agents. By testing 18 representative models on MMAU, we
provide deep and insightful analyses. Ultimately, MMAU not only sheds light on
the capabilities and limitations of LLM agents but also enhances the
interpretability of their performance. Datasets and evaluation scripts of MMAU
are released at https://github.com/apple/axlearn/docs/research/mmau.
66a85220d96a5adbcca664e0
Guoli Yin
admin_assigned
2024-07-30T08:44:13.295Z
605b7f42935268bc086131ba
/avatars/a55109f714b33f9d59d69011ddeb0b9f.svg
Guoli Yin
user
gyin94
66a85220d96a5adbcca664e1
Haoping Bai
claimed_verified
2024-08-03T08:26:00.210Z
633251c649a95639154e4f27
https://cdn-avatars.huggingface.co/v1/production/uploads/1664242105182-noauth.png
Haoping (Felix) Bai
user
bhpfelix
66a85220d96a5adbcca664e2
Shuang Ma
66a85220d96a5adbcca664e3
Feng Nan
66a85220d96a5adbcca664e4
Yanchao Sun
admin_assigned
2024-07-30T08:44:48.889Z
6488a8e65815252b02b948c8
/avatars/bb1ff64841c75b8d331986709b0a0a93.svg
Yanchao Sun
user
yanchao-sun
66a85220d96a5adbcca664e5
Zhaoyang Xu
admin_assigned
2024-07-30T08:44:55.724Z
65f968801780bc37117841c0
/avatars/b95d393d3301fff5c1c12b7de6ea04e7.svg
Zhaoyang Xu
user
kuwernv
66a85220d96a5adbcca664e6
Shen Ma
66a85220d96a5adbcca664e7
Jiarui Lu
admin_assigned
2024-07-30T08:45:19.282Z
64bad07938953777fe3ed5e9
/avatars/91c07a176d8d51111873751e82a4e0ea.svg
Jiarui Lu
user
jiarui-lu
66a85220d96a5adbcca664e8
Xiang Kong
admin_assigned
2024-07-30T08:45:32.157Z
65cfddb9844e4d8531cb6edc
/avatars/51ad35c439fa68ad5a1b4e33bc179907.svg
Xiang Kong
user
shawnkx
66a85220d96a5adbcca664e9
Aonan Zhang
admin_assigned
2024-07-30T08:45:45.585Z
65f3e68a138c6ab771434e2d
/avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
Aonan Zhang
user
AonanZhang
66a85220d96a5adbcca664ea
Dian Ang Yap
admin_assigned
2024-07-30T08:45:53.095Z
63cdd059ff7cd335f0e994f7
/avatars/5b7786883a8c503fbb954a74614aa524.svg
Dian Ang Yap
user
dayap
66a85220d96a5adbcca664eb
Yizhe zhang
admin_assigned
2024-07-30T08:46:21.996Z
5e64858c87403103f9f1055d
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
Yizhe Zhang
user
yizhezhang
66a85220d96a5adbcca664ec
Karsten Ahnert
admin_assigned
2024-07-30T08:46:28.014Z
643c19935fcffe09fb685f39
/avatars/9e837d32affad0843ea90c1b30cb67aa.svg
Karsten Ahnert
user
headmyshoulder
66a85220d96a5adbcca664ed
Vik Kamath
66a85220d96a5adbcca664ee
Mathias Berglund
66a85220d96a5adbcca664ef
Dominic Walsh
admin_assigned
2024-07-30T08:46:47.327Z
6696d534d79ce5b27da945db
/avatars/33229348a480ccd66ba63349a6f7489b.svg
Dominic Walsh
user
Dominic3661
66a85220d96a5adbcca664f0
Tobias Gindele
66a85220d96a5adbcca664f1
Juergen Wiest
66a85220d96a5adbcca664f2
Zhengfeng Lai
claimed_verified
2024-08-09T07:48:27.037Z
66b5295f83425904fa7a1a6a
/avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
Zhengfeng Lai
user
jefflai
66a85220d96a5adbcca664f3
Xiaoming Wang
66a85220d96a5adbcca664f4
Jiulong Shan
66a85220d96a5adbcca664f5
Meng Cao
66a85220d96a5adbcca664f6
Ruoming Pang
admin_assigned
2024-07-30T08:56:16.520Z
654ef8ad3fe6c0b1f871942f
/avatars/8d3689b9bf57c7c8060ac510a1839158.svg
Ruoming Pang
user
ruoming
66a85220d96a5adbcca664f7
Zirui Wang
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Recent advances in large language models (LLMs) have increased the demand for
comprehensive benchmarks to evaluate their capabilities as human-like agents.
Existing benchmarks, while useful, often focus on specific application
scenarios, emphasizing task completion but failing to dissect the underlying
skills that drive these outcomes. This lack of granularity makes it difficult
to deeply discern where failures stem from. Additionally, setting up these
environments requires considerable effort, and issues of unreliability and
reproducibility sometimes arise, especially in interactive tasks. To address
these limitations, we introduce the Massive Multitask Agent Understanding
(MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need
for complex environment setups. It evaluates models across five domains,
including teal{Tool-use}, teal{Directed Acyclic Graph
(DAG) QA}, teal{Data Science and Machine Learning coding},
teal{Contest-level programming} and teal{Mathematics},
and covers five essential capabilities: orange{Understanding},
orange{Reasoning}, orange{Planning},
orange{Problem-solving}, and orange{Self-correction}.
With a total of 20 meticulously designed tasks encompassing over 3K distinct
prompts, MMAU provides a comprehensive framework for evaluating the strengths
and limitations of LLM agents. By testing 18 representative models on MMAU, we
provide deep and insightful analyses. Ultimately, MMAU not only sheds light on
the capabilities and limitations of LLM agents but also enhances the
interpretability of their performance. Datasets and evaluation scripts of MMAU
are released at https://github.com/apple/axlearn/docs/research/mmau.
MMAU benchmark evaluates large language models' capabilities across multiple domains and essential skills using comprehensive offline tasks to enhance interpretability and reliability.
Large language models MMAU benchmark Tool-use Directed Acyclic Graph (DAG) QA Data Science and Machine Learning coding Contest-level programming Mathematics Understanding Reasoning Planning Problem-solving Self-correction