title: MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
summary: Recent advances in large language models (LLMs) have increased the demand for
  comprehensive benchmarks to evaluate their capabilities as human-like agents.
  Existing benchmarks, while useful, often focus on specific application
  scenarios, emphasizing task completion but failing to dissect the underlying
  skills that drive these outcomes. This lack of granularity makes it difficult
  to deeply discern where failures stem from. Additionally, setting up these
  environments requires considerable effort, and issues of unreliability and
  reproducibility sometimes arise, especially in interactive tasks. To address
  these limitations, we introduce the Massive Multitask Agent Understanding
  (MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need
  for complex environment setups. It evaluates models across five domains,
  including teal{Tool-use}, teal{Directed Acyclic Graph
  (DAG) QA}, teal{Data Science and Machine Learning coding},
  teal{Contest-level programming} and teal{Mathematics},
  and covers five essential capabilities: orange{Understanding},
  orange{Reasoning}, orange{Planning},
  orange{Problem-solving}, and orange{Self-correction}.
  With a total of 20 meticulously designed tasks encompassing over 3K distinct
  prompts, MMAU provides a comprehensive framework for evaluating the strengths
  and limitations of LLM agents. By testing 18 representative models on MMAU, we
  provide deep and insightful analyses. Ultimately, MMAU not only sheds light on
  the capabilities and limitations of LLM agents but also enhances the
  interpretability of their performance. Datasets and evaluation scripts of MMAU
  are released at https://github.com/apple/axlearn/docs/research/mmau.
paper_authors._id: 66a85220d96a5adbcca664e0
paper_authors.name: Guoli Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:44:13.295Z
paper_authors._id: 605b7f42935268bc086131ba
paper_authors.avatarUrl: /avatars/a55109f714b33f9d59d69011ddeb0b9f.svg
paper_authors.fullname: Guoli Yin
paper_authors.type: user
paper_authors.user: gyin94
paper_authors._id: 66a85220d96a5adbcca664e1
paper_authors.name: Haoping Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-03T08:26:00.210Z
paper_authors._id: 633251c649a95639154e4f27
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664242105182-noauth.png
paper_authors.fullname: Haoping (Felix) Bai
paper_authors.type: user
paper_authors.user: bhpfelix
paper_authors._id: 66a85220d96a5adbcca664e2
paper_authors.name: Shuang Ma
paper_authors._id: 66a85220d96a5adbcca664e3
paper_authors.name: Feng Nan
paper_authors._id: 66a85220d96a5adbcca664e4
paper_authors.name: Yanchao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:44:48.889Z
paper_authors._id: 6488a8e65815252b02b948c8
paper_authors.avatarUrl: /avatars/bb1ff64841c75b8d331986709b0a0a93.svg
paper_authors.fullname: Yanchao Sun
paper_authors.type: user
paper_authors.user: yanchao-sun
paper_authors._id: 66a85220d96a5adbcca664e5
paper_authors.name: Zhaoyang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:44:55.724Z
paper_authors._id: 65f968801780bc37117841c0
paper_authors.avatarUrl: /avatars/b95d393d3301fff5c1c12b7de6ea04e7.svg
paper_authors.fullname: Zhaoyang Xu
paper_authors.type: user
paper_authors.user: kuwernv
paper_authors._id: 66a85220d96a5adbcca664e6
paper_authors.name: Shen Ma
paper_authors._id: 66a85220d96a5adbcca664e7
paper_authors.name: Jiarui Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:45:19.282Z
paper_authors._id: 64bad07938953777fe3ed5e9
paper_authors.avatarUrl: /avatars/91c07a176d8d51111873751e82a4e0ea.svg
paper_authors.fullname: Jiarui Lu
paper_authors.type: user
paper_authors.user: jiarui-lu
paper_authors._id: 66a85220d96a5adbcca664e8
paper_authors.name: Xiang Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:45:32.157Z
paper_authors._id: 65cfddb9844e4d8531cb6edc
paper_authors.avatarUrl: /avatars/51ad35c439fa68ad5a1b4e33bc179907.svg
paper_authors.fullname: Xiang Kong
paper_authors.type: user
paper_authors.user: shawnkx
paper_authors._id: 66a85220d96a5adbcca664e9
paper_authors.name: Aonan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:45:45.585Z
paper_authors._id: 65f3e68a138c6ab771434e2d
paper_authors.avatarUrl: /avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
paper_authors.fullname: Aonan Zhang
paper_authors.type: user
paper_authors.user: AonanZhang
paper_authors._id: 66a85220d96a5adbcca664ea
paper_authors.name: Dian Ang Yap
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:45:53.095Z
paper_authors._id: 63cdd059ff7cd335f0e994f7
paper_authors.avatarUrl: /avatars/5b7786883a8c503fbb954a74614aa524.svg
paper_authors.fullname: Dian Ang Yap
paper_authors.type: user
paper_authors.user: dayap
paper_authors._id: 66a85220d96a5adbcca664eb
paper_authors.name: Yizhe zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:46:21.996Z
paper_authors._id: 5e64858c87403103f9f1055d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizhezhang
paper_authors._id: 66a85220d96a5adbcca664ec
paper_authors.name: Karsten Ahnert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:46:28.014Z
paper_authors._id: 643c19935fcffe09fb685f39
paper_authors.avatarUrl: /avatars/9e837d32affad0843ea90c1b30cb67aa.svg
paper_authors.fullname: Karsten Ahnert
paper_authors.type: user
paper_authors.user: headmyshoulder
paper_authors._id: 66a85220d96a5adbcca664ed
paper_authors.name: Vik Kamath
paper_authors._id: 66a85220d96a5adbcca664ee
paper_authors.name: Mathias Berglund
paper_authors._id: 66a85220d96a5adbcca664ef
paper_authors.name: Dominic Walsh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:46:47.327Z
paper_authors._id: 6696d534d79ce5b27da945db
paper_authors.avatarUrl: /avatars/33229348a480ccd66ba63349a6f7489b.svg
paper_authors.fullname: Dominic Walsh
paper_authors.type: user
paper_authors.user: Dominic3661
paper_authors._id: 66a85220d96a5adbcca664f0
paper_authors.name: Tobias Gindele
paper_authors._id: 66a85220d96a5adbcca664f1
paper_authors.name: Juergen Wiest
paper_authors._id: 66a85220d96a5adbcca664f2
paper_authors.name: Zhengfeng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:27.037Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 66a85220d96a5adbcca664f3
paper_authors.name: Xiaoming Wang
paper_authors._id: 66a85220d96a5adbcca664f4
paper_authors.name: Jiulong Shan
paper_authors._id: 66a85220d96a5adbcca664f5
paper_authors.name: Meng Cao
paper_authors._id: 66a85220d96a5adbcca664f6
paper_authors.name: Ruoming Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T08:56:16.520Z
paper_authors._id: 654ef8ad3fe6c0b1f871942f
paper_authors.avatarUrl: /avatars/8d3689b9bf57c7c8060ac510a1839158.svg
paper_authors.fullname: Ruoming Pang
paper_authors.type: user
paper_authors.user: ruoming
paper_authors._id: 66a85220d96a5adbcca664f7
paper_authors.name: Zirui Wang
paper_title: MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
paper_summary: Recent advances in large language models (LLMs) have increased the demand for
  comprehensive benchmarks to evaluate their capabilities as human-like agents.
  Existing benchmarks, while useful, often focus on specific application
  scenarios, emphasizing task completion but failing to dissect the underlying
  skills that drive these outcomes. This lack of granularity makes it difficult
  to deeply discern where failures stem from. Additionally, setting up these
  environments requires considerable effort, and issues of unreliability and
  reproducibility sometimes arise, especially in interactive tasks. To address
  these limitations, we introduce the Massive Multitask Agent Understanding
  (MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need
  for complex environment setups. It evaluates models across five domains,
  including teal{Tool-use}, teal{Directed Acyclic Graph
  (DAG) QA}, teal{Data Science and Machine Learning coding},
  teal{Contest-level programming} and teal{Mathematics},
  and covers five essential capabilities: orange{Understanding},
  orange{Reasoning}, orange{Planning},
  orange{Problem-solving}, and orange{Self-correction}.
  With a total of 20 meticulously designed tasks encompassing over 3K distinct
  prompts, MMAU provides a comprehensive framework for evaluating the strengths
  and limitations of LLM agents. By testing 18 representative models on MMAU, we
  provide deep and insightful analyses. Ultimately, MMAU not only sheds light on
  the capabilities and limitations of LLM agents but also enhances the
  interpretability of their performance. Datasets and evaluation scripts of MMAU
  are released at https://github.com/apple/axlearn/docs/research/mmau.
paper_ai_summary: MMAU benchmark evaluates large language models' capabilities across multiple domains and essential skills using comprehensive offline tasks to enhance interpretability and reliability.
paper_ai_keywords: Large language models MMAU benchmark Tool-use Directed Acyclic Graph (DAG) QA Data Science and Machine Learning coding Contest-level programming Mathematics Understanding Reasoning Planning Problem-solving Self-correction