title: KoLA: Carefully Benchmarking World Knowledge of Large Language Models
summary: The unprecedented performance of large language models (LLMs) necessitates
  improvements in evaluations. Rather than merely exploring the breadth of LLM
  abilities, we believe meticulous and thoughtful designs are essential to
  thorough, unbiased, and applicable evaluations. Given the importance of world
  knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark
  (KoLA), in which we carefully design three crucial factors: (1) For ability
  modeling, we mimic human cognition to form a four-level taxonomy of
  knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair
  comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs,
  along with continuously collected emerging corpora, aiming to evaluate the
  capacity to handle unseen data and evolving knowledge. (3) For evaluation
  criteria, we adopt a contrastive system, including overall standard scores for
  better numerical comparability across tasks and models and a unique
  self-contrast metric for automatically evaluating knowledge hallucination. We
  evaluate 21 open-source and commercial LLMs and obtain some intriguing
  findings. The KoLA dataset and open-participation leaderboard are publicly
  released at https://kola.xlore.cn and will be continuously updated to provide
  references for developing LLMs and knowledge-related systems.
paper_authors._id: 648be739152fc5a80d2c7c88
paper_authors.name: Jifan Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T12:04:33.769Z
paper_authors._id: 632d6e8d2f9c892dcf0ed4bb
paper_authors.avatarUrl: /avatars/fba1fb87bcac340a8eee3b9f4fc35bc5.svg
paper_authors.fullname: Jifan Yu
paper_authors.type: user
paper_authors.user: JovanYu
paper_authors._id: 648be739152fc5a80d2c7c89
paper_authors.name: Xiaozhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T07:15:58.863Z
paper_authors._id: 648c48e7a47a0850f9458b67
paper_authors.avatarUrl: /avatars/47d71d80f9901313feb0199c37296389.svg
paper_authors.fullname: Xiaozhi Wang
paper_authors.type: user
paper_authors.user: wangxz098
paper_authors._id: 648be739152fc5a80d2c7c8a
paper_authors.name: Shangqing Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:40:43.817Z
paper_authors._id: 648c48d8c0ddeee6df5b6d22
paper_authors.avatarUrl: /avatars/8706b0b16dfc332b96c91d3ced31bd0b.svg
paper_authors.fullname: Shangqing Tu
paper_authors.type: user
paper_authors.user: tsq2000
paper_authors._id: 648be739152fc5a80d2c7c8b
paper_authors.name: Shulin Cao
paper_authors._id: 648be739152fc5a80d2c7c8c
paper_authors.name: Daniel Zhang-Li
paper_authors._id: 648be739152fc5a80d2c7c8d
paper_authors.name: Xin Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:35:20.032Z
paper_authors._id: 648c64829c935db2b527a764
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
paper_authors.fullname: Xin Lv
paper_authors.type: user
paper_authors.user: davidlvxin
paper_authors._id: 648be739152fc5a80d2c7c8e
paper_authors.name: Hao Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:20:31.787Z
paper_authors._id: 625a5446f1063e7085d5178a
paper_authors.avatarUrl: /avatars/5e78186f13f74b14e01583e06ff6c4dc.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: Wesleythu
paper_authors._id: 648be739152fc5a80d2c7c8f
paper_authors.name: Zijun Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:25:21.350Z
paper_authors._id: 648c4b46e549be47af1aafcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c4b46e549be47af1aafcd/YgOHbmUM2EDM-lb7GdqXz.jpeg
paper_authors.fullname: Zijun
paper_authors.type: user
paper_authors.user: TranSirius
paper_authors._id: 648be739152fc5a80d2c7c90
paper_authors.name: Xiaohan Zhang
paper_authors._id: 648be739152fc5a80d2c7c91
paper_authors.name: Hanming Li
paper_authors._id: 648be739152fc5a80d2c7c92
paper_authors.name: Chunyang Li
paper_authors._id: 648be739152fc5a80d2c7c93
paper_authors.name: Zheyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:03:43.894Z
paper_authors._id: 648c4af819bb04c06467189c
paper_authors.avatarUrl: /avatars/8b9372a233d4c00b555625fa7b5203e2.svg
paper_authors.fullname: Zheyuan Zhang
paper_authors.type: user
paper_authors.user: Zheyuan22
paper_authors._id: 648be739152fc5a80d2c7c94
paper_authors.name: Yushi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-01T08:45:17.984Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 648be739152fc5a80d2c7c95
paper_authors.name: Yantao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T17:38:29.470Z
paper_authors._id: 62e25e2247678ea5ce1b1786
paper_authors.avatarUrl: /avatars/1bb32e7597a9b1c89c434cbf550b5382.svg
paper_authors.fullname: Yantao
paper_authors.type: user
paper_authors.user: RicardoL1u
paper_authors._id: 648be739152fc5a80d2c7c96
paper_authors.name: Amy Xin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T08:17:47.017Z
paper_authors._id: 660bf98c3336a7e128a0e918
paper_authors.avatarUrl: /avatars/3e3f2886bd4a730ec19b13aecc99279f.svg
paper_authors.fullname: Amy Xin
paper_authors.type: user
paper_authors.user: amyxx2001
paper_authors._id: 648be739152fc5a80d2c7c97
paper_authors.name: Nianyi Lin
paper_authors._id: 648be739152fc5a80d2c7c98
paper_authors.name: Kaifeng Yun
paper_authors._id: 648be739152fc5a80d2c7c99
paper_authors.name: Linlu Gong
paper_authors._id: 648be739152fc5a80d2c7c9a
paper_authors.name: Jianhui Chen
paper_authors._id: 648be739152fc5a80d2c7c9b
paper_authors.name: Zhili Wu
paper_authors._id: 648be739152fc5a80d2c7c9c
paper_authors.name: Yunjia Qi
paper_authors._id: 648be739152fc5a80d2c7c9d
paper_authors.name: Weikai Li
paper_authors._id: 648be739152fc5a80d2c7c9e
paper_authors.name: Yong Guan
paper_authors._id: 648be739152fc5a80d2c7c9f
paper_authors.name: Kaisheng Zeng
paper_authors._id: 648be739152fc5a80d2c7ca0
paper_authors.name: Ji Qi
paper_authors._id: 648be739152fc5a80d2c7ca1
paper_authors.name: Hailong Jin
paper_authors._id: 648be739152fc5a80d2c7ca2
paper_authors.name: Jinxin Liu
paper_authors._id: 648be739152fc5a80d2c7ca3
paper_authors.name: Yu Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T14:08:46.423Z
paper_authors._id: 5e7e595230dc073f817a2bb5
paper_authors.avatarUrl: /avatars/d5ff36e45555d9e169cf56c845736444.svg
paper_authors.fullname: Yu Gu
paper_authors.type: user
paper_authors.user: entslscheia
paper_authors._id: 648be739152fc5a80d2c7ca4
paper_authors.name: Yuan Yao
paper_authors._id: 648be739152fc5a80d2c7ca5
paper_authors.name: Ning Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-31T09:51:06.866Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 648be739152fc5a80d2c7ca6
paper_authors.name: Lei Hou
paper_authors._id: 648be739152fc5a80d2c7ca7
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T08:02:54.391Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 648be739152fc5a80d2c7ca8
paper_authors.name: Bin Xu
paper_authors._id: 648be739152fc5a80d2c7ca9
paper_authors.name: Jie Tang
paper_authors._id: 648be739152fc5a80d2c7caa
paper_authors.name: Juanzi Li
paper_title: KoLA: Carefully Benchmarking World Knowledge of Large Language Models
paper_summary: The unprecedented performance of large language models (LLMs) necessitates
  improvements in evaluations. Rather than merely exploring the breadth of LLM
  abilities, we believe meticulous and thoughtful designs are essential to
  thorough, unbiased, and applicable evaluations. Given the importance of world
  knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark
  (KoLA), in which we carefully design three crucial factors: (1) For ability
  modeling, we mimic human cognition to form a four-level taxonomy of
  knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair
  comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs,
  along with continuously collected emerging corpora, aiming to evaluate the
  capacity to handle unseen data and evolving knowledge. (3) For evaluation
  criteria, we adopt a contrastive system, including overall standard scores for
  better numerical comparability across tasks and models and a unique
  self-contrast metric for automatically evaluating knowledge hallucination. We
  evaluate 21 open-source and commercial LLMs and obtain some intriguing
  findings. The KoLA dataset and open-participation leaderboard are publicly
  released at https://kola.xlore.cn and will be continuously updated to provide
  references for developing LLMs and knowledge-related systems.
paper_ai_summary: A benchmark for evaluating large language models emphasizes knowledge-related abilities, uses a mix of datasets, and incorporates self-contrast metrics to detect knowledge hallucination.
paper_ai_keywords: large language models (LLMs) Knowledge-oriented LLM Assessment benchmark (KoLA) four-level taxonomy knowledge-related abilities Wikipedia emerging corpora contrastive system self-contrast metric knowledge hallucination