title: Which of These Best Describes Multiple Choice Evaluation with LLMs? A)
    Forced B) Flawed C) Fixable D) All of the Above
summary: Multiple choice question answering (MCQA) is popular for LLM evaluation due
  to its simplicity and human-like testing, but we argue for its reform. We first
  reveal flaws in MCQA's format, as it struggles to: 1) test
  generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge.
  We instead advocate for generative formats based on human testing-where LLMs
  construct and explain answers-better capturing user needs and knowledge while
  remaining easy to score. We then show even when MCQA is a useful format, its
  datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In
  each issue, we give fixes from education, like rubrics to guide MCQ writing;
  scoring methods to bridle guessing; and Item Response Theory to build harder
  MCQs. Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful
  explanations-showing how our prior solutions better measure or address these
  issues. While we do not need to desert MCQA, we encourage more efforts in
  refining the task based on educational testing, advancing evaluations.
paper_authors._id: 67b7e12b92b9b5b8184c6580
paper_authors.name: Nishant Balepur
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:59:02.019Z
paper_authors._id: 62a3f93fe2b7740fe2a94c86
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a3f93fe2b7740fe2a94c86/ZiaPqiVqXI2ANIyWQY_hT.png
paper_authors.fullname: Nishant Balepur
paper_authors.type: user
paper_authors.user: nbalepur
paper_authors._id: 67b7e12b92b9b5b8184c6581
paper_authors.name: Rachel Rudinger
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-26T01:15:28.081Z
paper_authors._id: 6833c05d30cd9df52a0eec16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/DH8C51Ryt-8DS4TIzj-vL.png
paper_authors.fullname: Rachel Rudinger
paper_authors.type: user
paper_authors.user: rudinger
paper_authors._id: 67b7e12b92b9b5b8184c6582
paper_authors.name: Jordan Lee Boyd-Graber
paper_title: Which of These Best Describes Multiple Choice Evaluation with LLMs? A)
    Forced B) Flawed C) Fixable D) All of the Above
paper_summary: Multiple choice question answering (MCQA) is popular for LLM evaluation due
  to its simplicity and human-like testing, but we argue for its reform. We first
  reveal flaws in MCQA's format, as it struggles to: 1) test
  generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge.
  We instead advocate for generative formats based on human testing-where LLMs
  construct and explain answers-better capturing user needs and knowledge while
  remaining easy to score. We then show even when MCQA is a useful format, its
  datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In
  each issue, we give fixes from education, like rubrics to guide MCQ writing;
  scoring methods to bridle guessing; and Item Response Theory to build harder
  MCQs. Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful
  explanations-showing how our prior solutions better measure or address these
  issues. While we do not need to desert MCQA, we encourage more efforts in
  refining the task based on educational testing, advancing evaluations.
paper_ai_summary: Proposing improvements to MCQA for LLM evaluation by addressing format flaws, dataset issues, and errors in testing, advocating for generative formats that better capture user needs.
paper_ai_keywords: MCQA LLM generation subjectivity human testing rubrics Item Response Theory robustness biases unfaithful explanations