StructEval: Deepen and Broaden Large Language Model Assessment via
  Structured Evaluation
Evaluation is the baton for the development of large language models. Current
evaluations typically employ a single-item assessment paradigm for each atomic
test objective, which struggles to discern whether a model genuinely possesses
the required capabilities or merely memorizes/guesses the answers to specific
questions. To this end, we propose a novel evaluation framework referred to as
StructEval. Starting from an atomic test objective, StructEval deepens and
broadens the evaluation by conducting a structured assessment across multiple
cognitive levels and critical concepts, and therefore offers a comprehensive,
robust and consistent evaluation for LLMs. Experiments on three widely-used
benchmarks demonstrate that StructEval serves as a reliable tool for resisting
the risk of data contamination and reducing the interference of potential
biases, thereby providing more reliable and consistent conclusions regarding
model capabilities. Our framework also sheds light on the design of future
principled and trustworthy LLM evaluation protocols.
66b30deb9f3dfa895ce0493e
Boxi Cao
admin_assigned
2024-08-08T12:26:41.560Z
5fc052241160c47d1d438556
/avatars/f508707fc92f1b42f7897b12b727754c.svg
Boxi Cao
user
Bowieee
66b30deb9f3dfa895ce0493f
Mengjie Ren
66b30deb9f3dfa895ce04940
Hongyu Lin
admin_assigned
2024-08-08T12:27:09.968Z
645cee9f680734460f9c3a55
/avatars/efda716b31c241107f89ffbfe7466d80.svg
linhongyu
user
linhongyu
66b30deb9f3dfa895ce04941
Xianpei Han
admin_assigned
2024-08-08T12:27:15.685Z
65e99a77e71555ed193609cf
/avatars/38ceb127883944677665da967d17dd18.svg
Xianpei Han
user
xphan
66b30deb9f3dfa895ce04942
Feng Zhang
66b30deb9f3dfa895ce04943
Junfeng Zhan
66b30deb9f3dfa895ce04944
Le Sun
StructEval: Deepen and Broaden Large Language Model Assessment via
  Structured Evaluation
Evaluation is the baton for the development of large language models. Current
evaluations typically employ a single-item assessment paradigm for each atomic
test objective, which struggles to discern whether a model genuinely possesses
the required capabilities or merely memorizes/guesses the answers to specific
questions. To this end, we propose a novel evaluation framework referred to as
StructEval. Starting from an atomic test objective, StructEval deepens and
broadens the evaluation by conducting a structured assessment across multiple
cognitive levels and critical concepts, and therefore offers a comprehensive,
robust and consistent evaluation for LLMs. Experiments on three widely-used
benchmarks demonstrate that StructEval serves as a reliable tool for resisting
the risk of data contamination and reducing the interference of potential
biases, thereby providing more reliable and consistent conclusions regarding
model capabilities. Our framework also sheds light on the design of future
principled and trustworthy LLM evaluation protocols.
StructEval provides a comprehensive, robust, and consistent evaluation framework for large language models by assessing multiple cognitive levels and critical concepts, reducing data contamination and bias.
StructEval large language models evaluation framework cognitive levels critical concepts data contamination bias