title: SATA-BENCH: Select All That Apply Benchmark for Multiple Choice
    Questions
summary: Large language models (LLMs) are increasingly evaluated on single-answer
  multiple-choice tasks, yet many real-world problems require identifying all
  correct answers from a set of options. This capability remains underexplored.
  We introduce SATA-BENCH, the first dedicated benchmark for evaluating LLMs on
  Select All That Apply (SATA) questions across diverse domains, including
  reading comprehension, law, and biomedicine. Our evaluation of 27 open-source
  and proprietary models reveals a significant gap: even the strongest model
  achieves only 41.8% exact match, exposing LLMs' inability to reliably identify
  all correct answers. We find that this weakness stems from two core challenges:
  selection bias - models favor certain choices regardless of content, and count
  bias - models fail to predict the correct number of answers. To address these
  issues, we propose Choice Funnel, a decoding strategy that combines token
  debiasing with adaptive thresholding to guide models toward complete and
  accurate selections. Choice Funnel achieves up to 29% higher exact match than
  competitive baselines while reducing inference cost by over 64%. Our findings
  expose fundamental limitations in current LLMs and introduce a new framework
  for diagnosing and improving multi-answer reasoning. We release SATA-BENCH and
  Choice Funnel to promote LLM development for robust decision-making in
  realistic, multi-answer applications.
paper_authors._id: 683e61338ebad8b7519bc7f3
paper_authors.name: Weijie Xu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-03T02:43:01.430Z
paper_authors._id: 63e3f57754f51ea342ce26be
paper_authors.avatarUrl: /avatars/df9d52c376bed6868d341bb006bec212.svg
paper_authors.fullname: Weijie Xu
paper_authors.type: user
paper_authors.user: xwjzds
paper_authors._id: 683e61338ebad8b7519bc7f4
paper_authors.name: Shixian Cui
paper_authors._id: 683e61338ebad8b7519bc7f5
paper_authors.name: Xi Fang
paper_authors._id: 683e61338ebad8b7519bc7f6
paper_authors.name: Chi Xue
paper_authors._id: 683e61338ebad8b7519bc7f7
paper_authors.name: Stephanie Eckman
paper_authors._id: 683e61338ebad8b7519bc7f8
paper_authors.name: Chandan Reddy
paper_title: SATA-BENCH: Select All That Apply Benchmark for Multiple Choice
    Questions
paper_summary: Large language models (LLMs) are increasingly evaluated on single-answer
  multiple-choice tasks, yet many real-world problems require identifying all
  correct answers from a set of options. This capability remains underexplored.
  We introduce SATA-BENCH, the first dedicated benchmark for evaluating LLMs on
  Select All That Apply (SATA) questions across diverse domains, including
  reading comprehension, law, and biomedicine. Our evaluation of 27 open-source
  and proprietary models reveals a significant gap: even the strongest model
  achieves only 41.8% exact match, exposing LLMs' inability to reliably identify
  all correct answers. We find that this weakness stems from two core challenges:
  selection bias - models favor certain choices regardless of content, and count
  bias - models fail to predict the correct number of answers. To address these
  issues, we propose Choice Funnel, a decoding strategy that combines token
  debiasing with adaptive thresholding to guide models toward complete and
  accurate selections. Choice Funnel achieves up to 29% higher exact match than
  competitive baselines while reducing inference cost by over 64%. Our findings
  expose fundamental limitations in current LLMs and introduce a new framework
  for diagnosing and improving multi-answer reasoning. We release SATA-BENCH and
  Choice Funnel to promote LLM development for robust decision-making in
  realistic, multi-answer applications.
paper_ai_summary: SATA-BENCH evaluates LLMs on multi-answer questions, revealing selections biases and proposing Choice Funnel to improve accuracy and reduce costs in multi-answer reasoning tasks.
paper_ai_keywords: Select All That Apply (SATA) questions SATA-BENCH token debiasing adaptive thresholding Choice Funnel