Instruction-Following Evaluation for Large Language Models
One core capability of Large Language Models (LLMs) is to follow natural
language instructions. However, the evaluation of such abilities is not
standardized: Human evaluations are expensive, slow, and not objectively
reproducible, while LLM-based auto-evaluation is potentially biased or limited
by the ability of the evaluator LLM. To overcome these issues, we introduce
Instruction-Following Eval (IFEval) for large language models. IFEval is a
straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set
of "verifiable instructions" such as "write in more than 400 words" and
"mention the keyword of AI at least 3 times". We identified 25 types of those
verifiable instructions and constructed around 500 prompts, with each prompt
containing one or more verifiable instructions. We show evaluation results of
two widely available LLMs on the market. Our code and data can be found at
https://github.com/google-research/google-research/tree/master/instruction_following_eval
65542ef44045e53f45001cc3
Jeffrey Zhou
65542ef44045e53f45001cc4
Tianjian Lu
admin_assigned
2023-11-15T10:20:49.025Z
651c5df1a6c59c976a297d17
/avatars/32033e8bc2ecb470e58a60dc8afbe03f.svg
Tianjian LU
user
tlu7
65542ef44045e53f45001cc5
Swaroop Mishra
admin_assigned
2023-11-15T10:20:55.510Z
61841feb7c380c66d40cad54
/avatars/7a79362a24b844860781fb0625527eae.svg
Swaroop Mishra
user
swaroop7
65542ef44045e53f45001cc6
Siddhartha Brahma
65542ef44045e53f45001cc7
Sujoy Basu
65542ef44045e53f45001cc8
Yi Luan
65542ef44045e53f45001cc9
Denny Zhou
65542ef44045e53f45001cca
Le Hou
admin_assigned
2023-11-15T10:21:23.182Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
Instruction-Following Evaluation for Large Language Models
One core capability of Large Language Models (LLMs) is to follow natural
language instructions. However, the evaluation of such abilities is not
standardized: Human evaluations are expensive, slow, and not objectively
reproducible, while LLM-based auto-evaluation is potentially biased or limited
by the ability of the evaluator LLM. To overcome these issues, we introduce
Instruction-Following Eval (IFEval) for large language models. IFEval is a
straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set
of "verifiable instructions" such as "write in more than 400 words" and
"mention the keyword of AI at least 3 times". We identified 25 types of those
verifiable instructions and constructed around 500 prompts, with each prompt
containing one or more verifiable instructions. We show evaluation results of
two widely available LLMs on the market. Our code and data can be found at
https://github.com/google-research/google-research/tree/master/instruction_following_eval
Instruction-Following Eval (IFEval) provides a standardized benchmark for evaluating Large Language Models' ability to follow natural language instructions using verifiable criteria.
Instruction-Following Eval IFEval Large Language Models verifiable instructions