title: Instruction-Following Evaluation for Large Language Models
summary: One core capability of Large Language Models (LLMs) is to follow natural
  language instructions. However, the evaluation of such abilities is not
  standardized: Human evaluations are expensive, slow, and not objectively
  reproducible, while LLM-based auto-evaluation is potentially biased or limited
  by the ability of the evaluator LLM. To overcome these issues, we introduce
  Instruction-Following Eval (IFEval) for large language models. IFEval is a
  straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set
  of "verifiable instructions" such as "write in more than 400 words" and
  "mention the keyword of AI at least 3 times". We identified 25 types of those
  verifiable instructions and constructed around 500 prompts, with each prompt
  containing one or more verifiable instructions. We show evaluation results of
  two widely available LLMs on the market. Our code and data can be found at
  https://github.com/google-research/google-research/tree/master/instruction_following_eval
paper_authors._id: 65542ef44045e53f45001cc3
paper_authors.name: Jeffrey Zhou
paper_authors._id: 65542ef44045e53f45001cc4
paper_authors.name: Tianjian Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:20:49.025Z
paper_authors._id: 651c5df1a6c59c976a297d17
paper_authors.avatarUrl: /avatars/32033e8bc2ecb470e58a60dc8afbe03f.svg
paper_authors.fullname: Tianjian LU
paper_authors.type: user
paper_authors.user: tlu7
paper_authors._id: 65542ef44045e53f45001cc5
paper_authors.name: Swaroop Mishra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:20:55.510Z
paper_authors._id: 61841feb7c380c66d40cad54
paper_authors.avatarUrl: /avatars/7a79362a24b844860781fb0625527eae.svg
paper_authors.fullname: Swaroop Mishra
paper_authors.type: user
paper_authors.user: swaroop7
paper_authors._id: 65542ef44045e53f45001cc6
paper_authors.name: Siddhartha Brahma
paper_authors._id: 65542ef44045e53f45001cc7
paper_authors.name: Sujoy Basu
paper_authors._id: 65542ef44045e53f45001cc8
paper_authors.name: Yi Luan
paper_authors._id: 65542ef44045e53f45001cc9
paper_authors.name: Denny Zhou
paper_authors._id: 65542ef44045e53f45001cca
paper_authors.name: Le Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:21:23.182Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_title: Instruction-Following Evaluation for Large Language Models
paper_summary: One core capability of Large Language Models (LLMs) is to follow natural
  language instructions. However, the evaluation of such abilities is not
  standardized: Human evaluations are expensive, slow, and not objectively
  reproducible, while LLM-based auto-evaluation is potentially biased or limited
  by the ability of the evaluator LLM. To overcome these issues, we introduce
  Instruction-Following Eval (IFEval) for large language models. IFEval is a
  straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set
  of "verifiable instructions" such as "write in more than 400 words" and
  "mention the keyword of AI at least 3 times". We identified 25 types of those
  verifiable instructions and constructed around 500 prompts, with each prompt
  containing one or more verifiable instructions. We show evaluation results of
  two widely available LLMs on the market. Our code and data can be found at
  https://github.com/google-research/google-research/tree/master/instruction_following_eval
paper_ai_summary: Instruction-Following Eval (IFEval) provides a standardized benchmark for evaluating Large Language Models' ability to follow natural language instructions using verifiable criteria.
paper_ai_keywords: Instruction-Following Eval IFEval Large Language Models verifiable instructions