title: In Case You Missed It: ARC 'Challenge' Is Not That Challenging
summary: ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily
  due to an evaluation setup that prevents direct comparison of answer choices
  rather than inherent complexity. Although some researchers have quietly shifted
  to a more appropriate scheme over the last year, the implications of this
  change have yet to be widely acknowledged. We highlight this overlooked shift,
  show how similar evaluation practices falsely imply reasoning deficits in other
  benchmarks, and demonstrate that fairer methods dramatically reduce performance
  gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing
  so, we reveal how evaluation shapes perceived difficulty and offer guidelines
  to ensure that multiple-choice evaluations accurately reflect actual model
  capabilities.
paper_authors._id: 676bd06524bd46fa1990dcec
paper_authors.name: Łukasz Borchmann
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-01T05:01:19.976Z
paper_authors._id: 600b381d3cc3b87db94bc0ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/600b381d3cc3b87db94bc0ce/I3xpr4gzcG1uXawXBpWpD.jpeg
paper_authors.fullname: Łukasz Borchmann
paper_authors.type: user
paper_authors.user: Borchmann
paper_title: In Case You Missed It: ARC 'Challenge' Is Not That Challenging
paper_summary: ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily
  due to an evaluation setup that prevents direct comparison of answer choices
  rather than inherent complexity. Although some researchers have quietly shifted
  to a more appropriate scheme over the last year, the implications of this
  change have yet to be widely acknowledged. We highlight this overlooked shift,
  show how similar evaluation practices falsely imply reasoning deficits in other
  benchmarks, and demonstrate that fairer methods dramatically reduce performance
  gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing
  so, we reveal how evaluation shapes perceived difficulty and offer guidelines
  to ensure that multiple-choice evaluations accurately reflect actual model
  capabilities.
paper_ai_summary: The evaluation setup in ARC Challenge affects perceived model difficulty, leading to false implications of reasoning deficits, whereas fairer methods reduce performance gaps and reveal superhuman capabilities in tasks like OpenBookQA.
paper_ai_keywords: evaluation setup ARC Challenge ARC Easy modern LLMs answer choices reasoning deficits SIQA OpenBookQA multiple-choice evaluations model capabilities