Which of These Best Describes Multiple Choice Evaluation with LLMs? A)
  Forced B) Flawed C) Fixable D) All of the Above
Multiple choice question answering (MCQA) is popular for LLM evaluation due
to its simplicity and human-like testing, but we argue for its reform. We first
reveal flaws in MCQA's format, as it struggles to: 1) test
generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge.
We instead advocate for generative formats based on human testing-where LLMs
construct and explain answers-better capturing user needs and knowledge while
remaining easy to score. We then show even when MCQA is a useful format, its
datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In
each issue, we give fixes from education, like rubrics to guide MCQ writing;
scoring methods to bridle guessing; and Item Response Theory to build harder
MCQs. Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful
explanations-showing how our prior solutions better measure or address these
issues. While we do not need to desert MCQA, we encourage more efforts in
refining the task based on educational testing, advancing evaluations.
67b7e12b92b9b5b8184c6580
Nishant Balepur
claimed_verified
2025-02-21T09:59:02.019Z
62a3f93fe2b7740fe2a94c86
https://cdn-avatars.huggingface.co/v1/production/uploads/62a3f93fe2b7740fe2a94c86/ZiaPqiVqXI2ANIyWQY_hT.png
Nishant Balepur
user
nbalepur
67b7e12b92b9b5b8184c6581
Rachel Rudinger
extracted_pending
2025-05-26T01:15:28.081Z
6833c05d30cd9df52a0eec16
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/DH8C51Ryt-8DS4TIzj-vL.png
Rachel Rudinger
user
rudinger
67b7e12b92b9b5b8184c6582
Jordan Lee Boyd-Graber
Which of These Best Describes Multiple Choice Evaluation with LLMs? A)
  Forced B) Flawed C) Fixable D) All of the Above
Multiple choice question answering (MCQA) is popular for LLM evaluation due
to its simplicity and human-like testing, but we argue for its reform. We first
reveal flaws in MCQA's format, as it struggles to: 1) test
generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge.
We instead advocate for generative formats based on human testing-where LLMs
construct and explain answers-better capturing user needs and knowledge while
remaining easy to score. We then show even when MCQA is a useful format, its
datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In
each issue, we give fixes from education, like rubrics to guide MCQ writing;
scoring methods to bridle guessing; and Item Response Theory to build harder
MCQs. Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful
explanations-showing how our prior solutions better measure or address these
issues. While we do not need to desert MCQA, we encourage more efforts in
refining the task based on educational testing, advancing evaluations.
Proposing improvements to MCQA for LLM evaluation by addressing format flaws, dataset issues, and errors in testing, advocating for generative formats that better capture user needs.
MCQA LLM generation subjectivity human testing rubrics Item Response Theory robustness biases unfaithful explanations