title: A Survey on Evaluation of Large Language Models
summary: Large language models (LLMs) are gaining increasing popularity in both
  academia and industry, owing to their unprecedented performance in various
  applications. As LLMs continue to play a vital role in both research and daily
  use, their evaluation becomes increasingly critical, not only at the task
  level, but also at the society level for better understanding of their
  potential risks. Over the past years, significant efforts have been made to
  examine LLMs from various perspectives. This paper presents a comprehensive
  review of these evaluation methods for LLMs, focusing on three key dimensions:
  what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide
  an overview from the perspective of evaluation tasks, encompassing general
  natural language processing tasks, reasoning, medical usage, ethics,
  educations, natural and social sciences, agent applications, and other areas.
  Secondly, we answer the `where' and `how' questions by diving into the
  evaluation methods and benchmarks, which serve as crucial components in
  assessing performance of LLMs. Then, we summarize the success and failure cases
  of LLMs in different tasks. Finally, we shed light on several future challenges
  that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to
  researchers in the realm of LLMs evaluation, thereby aiding the development of
  more proficient LLMs. Our key point is that evaluation should be treated as an
  essential discipline to better assist the development of LLMs. We consistently
  maintain the related open-source materials at:
  https://github.com/MLGroupJLU/LLM-eval-survey.
paper_authors._id: 64a787b2fc1d1ebc8a84867f
paper_authors.name: Yupeng Chang
paper_authors._id: 64a787b2fc1d1ebc8a848680
paper_authors.name: Xu Wang
paper_authors._id: 64a787b2fc1d1ebc8a848681
paper_authors.name: Jindong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:24:21.647Z
paper_authors._id: 6204cc0d522e40b4a18d86e2
paper_authors.avatarUrl: /avatars/18daf2de5671e711dc745388dd60569d.svg
paper_authors.fullname: Jindong Wang
paper_authors.type: user
paper_authors.user: jindongwang
paper_authors._id: 64a787b2fc1d1ebc8a848682
paper_authors.name: Yuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:31:34.945Z
paper_authors._id: 670e57b3391f1a7021182bff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N0tuHZVz8KFPCv8G1qUX2.png
paper_authors.fullname: Yuan Wu
paper_authors.type: user
paper_authors.user: WhiteCatY
paper_authors._id: 64a787b2fc1d1ebc8a848683
paper_authors.name: Kaijie Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-06T07:06:00.081Z
paper_authors._id: 641fb3733c49fc33cedb4429
paper_authors.avatarUrl: /avatars/3577bca726282ed4b26009f3a2d5e87d.svg
paper_authors.fullname: Kaijie Zhu
paper_authors.type: user
paper_authors.user: March07
paper_authors._id: 64a787b2fc1d1ebc8a848684
paper_authors.name: Hao Chen
paper_authors._id: 64a787b2fc1d1ebc8a848685
paper_authors.name: Linyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T13:31:06.181Z
paper_authors._id: 64895683f534abe18eec264b
paper_authors.avatarUrl: /avatars/73cc9e6db6db86793787750776b57c63.svg
paper_authors.fullname: Linyi Yang
paper_authors.type: user
paper_authors.user: linyiyang2023
paper_authors._id: 64a787b2fc1d1ebc8a848686
paper_authors.name: Xiaoyuan Yi
paper_authors._id: 64a787b2fc1d1ebc8a848687
paper_authors.name: Cunxiang Wang
paper_authors._id: 64a787b2fc1d1ebc8a848688
paper_authors.name: Yidong Wang
paper_authors._id: 64a787b2fc1d1ebc8a848689
paper_authors.name: Wei Ye
paper_authors._id: 64a787b2fc1d1ebc8a84868a
paper_authors.name: Yue Zhang
paper_authors._id: 64a787b2fc1d1ebc8a84868b
paper_authors.name: Yi Chang
paper_authors._id: 64a787b2fc1d1ebc8a84868c
paper_authors.name: Philip S. Yu
paper_authors._id: 64a787b2fc1d1ebc8a84868d
paper_authors.name: Qiang Yang
paper_authors._id: 64a787b2fc1d1ebc8a84868e
paper_authors.name: Xing Xie
paper_title: A Survey on Evaluation of Large Language Models
paper_summary: Large language models (LLMs) are gaining increasing popularity in both
  academia and industry, owing to their unprecedented performance in various
  applications. As LLMs continue to play a vital role in both research and daily
  use, their evaluation becomes increasingly critical, not only at the task
  level, but also at the society level for better understanding of their
  potential risks. Over the past years, significant efforts have been made to
  examine LLMs from various perspectives. This paper presents a comprehensive
  review of these evaluation methods for LLMs, focusing on three key dimensions:
  what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide
  an overview from the perspective of evaluation tasks, encompassing general
  natural language processing tasks, reasoning, medical usage, ethics,
  educations, natural and social sciences, agent applications, and other areas.
  Secondly, we answer the `where' and `how' questions by diving into the
  evaluation methods and benchmarks, which serve as crucial components in
  assessing performance of LLMs. Then, we summarize the success and failure cases
  of LLMs in different tasks. Finally, we shed light on several future challenges
  that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to
  researchers in the realm of LLMs evaluation, thereby aiding the development of
  more proficient LLMs. Our key point is that evaluation should be treated as an
  essential discipline to better assist the development of LLMs. We consistently
  maintain the related open-source materials at:
  https://github.com/MLGroupJLU/LLM-eval-survey.
paper_ai_keywords: large language models (LLMs) evaluation tasks natural language processing (NLP) reasoning medical usage ethics education natural sciences social sciences agent applications evaluation methods benchmarks future challenges LLMs evaluation