WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More
  Challenging
We introduce WiCkeD, a simple method to increase the complexity of existing
multiple-choice benchmarks by randomly replacing a choice with "None of the
above", a method often used in educational tests. We show that WiCkeD can be
automatically applied to any existing benchmark, making it more challenging. We
apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight
LLMs. The performance of the models drops 12.1 points on average with respect
to the original versions of the datasets. When using chain-of-thought on 3 MMLU
datasets, the performance drop for the WiCkeD variant is similar to the one
observed when using the LLMs directly, showing that WiCkeD is also challenging
for models with enhanced reasoning abilities. WiCkeD also uncovers that some
models are more sensitive to the extra reasoning required, providing additional
information with respect to the original benchmarks. We relase our code and
data at https://github.com/ahmedselhady/wicked-benchmarks.
67bf0ccbb2f5c23eb0a69a7d
Ahmed Elhady
claimed_verified
2025-02-26T15:37:14.872Z
6586f687ce38d143c4092ed7
https://cdn-avatars.huggingface.co/v1/production/uploads/6586f687ce38d143c4092ed7/sNB7h0r0ZSP-4bbW_gDgF.png
Ahmed Mohamed Elhady
user
ahmedselhady
67bf0ccbb2f5c23eb0a69a7e
Eneko Agirre
67bf0ccbb2f5c23eb0a69a7f
Mikel Artetxe
WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More
  Challenging
We introduce WiCkeD, a simple method to increase the complexity of existing
multiple-choice benchmarks by randomly replacing a choice with "None of the
above", a method often used in educational tests. We show that WiCkeD can be
automatically applied to any existing benchmark, making it more challenging. We
apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight
LLMs. The performance of the models drops 12.1 points on average with respect
to the original versions of the datasets. When using chain-of-thought on 3 MMLU
datasets, the performance drop for the WiCkeD variant is similar to the one
observed when using the LLMs directly, showing that WiCkeD is also challenging
for models with enhanced reasoning abilities. WiCkeD also uncovers that some
models are more sensitive to the extra reasoning required, providing additional
information with respect to the original benchmarks. We relase our code and
data at https://github.com/ahmedselhady/wicked-benchmarks.
WiCkeD increases the complexity of multiple-choice benchmarks by adding a "None of the above" option, demonstrating that this addition challenges models' performance and reasoning abilities.
multiple-choice benchmarks chain-of-thought open-weight LLMs LLMs MMLU reasoning abilities