title: StructEval: Deepen and Broaden Large Language Model Assessment via
    Structured Evaluation
summary: Evaluation is the baton for the development of large language models. Current
  evaluations typically employ a single-item assessment paradigm for each atomic
  test objective, which struggles to discern whether a model genuinely possesses
  the required capabilities or merely memorizes/guesses the answers to specific
  questions. To this end, we propose a novel evaluation framework referred to as
  StructEval. Starting from an atomic test objective, StructEval deepens and
  broadens the evaluation by conducting a structured assessment across multiple
  cognitive levels and critical concepts, and therefore offers a comprehensive,
  robust and consistent evaluation for LLMs. Experiments on three widely-used
  benchmarks demonstrate that StructEval serves as a reliable tool for resisting
  the risk of data contamination and reducing the interference of potential
  biases, thereby providing more reliable and consistent conclusions regarding
  model capabilities. Our framework also sheds light on the design of future
  principled and trustworthy LLM evaluation protocols.
paper_authors._id: 66b30deb9f3dfa895ce0493e
paper_authors.name: Boxi Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:26:41.560Z
paper_authors._id: 5fc052241160c47d1d438556
paper_authors.avatarUrl: /avatars/f508707fc92f1b42f7897b12b727754c.svg
paper_authors.fullname: Boxi Cao
paper_authors.type: user
paper_authors.user: Bowieee
paper_authors._id: 66b30deb9f3dfa895ce0493f
paper_authors.name: Mengjie Ren
paper_authors._id: 66b30deb9f3dfa895ce04940
paper_authors.name: Hongyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:27:09.968Z
paper_authors._id: 645cee9f680734460f9c3a55
paper_authors.avatarUrl: /avatars/efda716b31c241107f89ffbfe7466d80.svg
paper_authors.fullname: linhongyu
paper_authors.type: user
paper_authors.user: linhongyu
paper_authors._id: 66b30deb9f3dfa895ce04941
paper_authors.name: Xianpei Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:27:15.685Z
paper_authors._id: 65e99a77e71555ed193609cf
paper_authors.avatarUrl: /avatars/38ceb127883944677665da967d17dd18.svg
paper_authors.fullname: Xianpei Han
paper_authors.type: user
paper_authors.user: xphan
paper_authors._id: 66b30deb9f3dfa895ce04942
paper_authors.name: Feng Zhang
paper_authors._id: 66b30deb9f3dfa895ce04943
paper_authors.name: Junfeng Zhan
paper_authors._id: 66b30deb9f3dfa895ce04944
paper_authors.name: Le Sun
paper_title: StructEval: Deepen and Broaden Large Language Model Assessment via
    Structured Evaluation
paper_summary: Evaluation is the baton for the development of large language models. Current
  evaluations typically employ a single-item assessment paradigm for each atomic
  test objective, which struggles to discern whether a model genuinely possesses
  the required capabilities or merely memorizes/guesses the answers to specific
  questions. To this end, we propose a novel evaluation framework referred to as
  StructEval. Starting from an atomic test objective, StructEval deepens and
  broadens the evaluation by conducting a structured assessment across multiple
  cognitive levels and critical concepts, and therefore offers a comprehensive,
  robust and consistent evaluation for LLMs. Experiments on three widely-used
  benchmarks demonstrate that StructEval serves as a reliable tool for resisting
  the risk of data contamination and reducing the interference of potential
  biases, thereby providing more reliable and consistent conclusions regarding
  model capabilities. Our framework also sheds light on the design of future
  principled and trustworthy LLM evaluation protocols.
paper_ai_keywords: StructEval atomic test objective cognitive levels critical concepts comprehensive robust consistent large language models (LLMs) data contamination biases principled trustworthy evaluation protocols