title: WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More
    Challenging
summary: We introduce WiCkeD, a simple method to increase the complexity of existing
  multiple-choice benchmarks by randomly replacing a choice with "None of the
  above", a method often used in educational tests. We show that WiCkeD can be
  automatically applied to any existing benchmark, making it more challenging. We
  apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight
  LLMs. The performance of the models drops 12.1 points on average with respect
  to the original versions of the datasets. When using chain-of-thought on 3 MMLU
  datasets, the performance drop for the WiCkeD variant is similar to the one
  observed when using the LLMs directly, showing that WiCkeD is also challenging
  for models with enhanced reasoning abilities. WiCkeD also uncovers that some
  models are more sensitive to the extra reasoning required, providing additional
  information with respect to the original benchmarks. We relase our code and
  data at https://github.com/ahmedselhady/wicked-benchmarks.
paper_authors._id: 67bf0ccbb2f5c23eb0a69a7d
paper_authors.name: Ahmed Elhady
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T15:37:14.872Z
paper_authors._id: 6586f687ce38d143c4092ed7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6586f687ce38d143c4092ed7/sNB7h0r0ZSP-4bbW_gDgF.png
paper_authors.fullname: Ahmed Mohamed Elhady
paper_authors.type: user
paper_authors.user: ahmedselhady
paper_authors._id: 67bf0ccbb2f5c23eb0a69a7e
paper_authors.name: Eneko Agirre
paper_authors._id: 67bf0ccbb2f5c23eb0a69a7f
paper_authors.name: Mikel Artetxe
paper_title: WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More
    Challenging
paper_summary: We introduce WiCkeD, a simple method to increase the complexity of existing
  multiple-choice benchmarks by randomly replacing a choice with "None of the
  above", a method often used in educational tests. We show that WiCkeD can be
  automatically applied to any existing benchmark, making it more challenging. We
  apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight
  LLMs. The performance of the models drops 12.1 points on average with respect
  to the original versions of the datasets. When using chain-of-thought on 3 MMLU
  datasets, the performance drop for the WiCkeD variant is similar to the one
  observed when using the LLMs directly, showing that WiCkeD is also challenging
  for models with enhanced reasoning abilities. WiCkeD also uncovers that some
  models are more sensitive to the extra reasoning required, providing additional
  information with respect to the original benchmarks. We relase our code and
  data at https://github.com/ahmedselhady/wicked-benchmarks.
paper_ai_summary: WiCkeD increases the complexity of multiple-choice benchmarks by adding a "None of the above" option, demonstrating that this addition challenges models' performance and reasoning abilities.
paper_ai_keywords: multiple-choice benchmarks chain-of-thought open-weight LLMs LLMs MMLU reasoning abilities