MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs
<|reference_start|>MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs: Existing benchmarks for large language models (LLMs) increasingly struggle to differentiate between top-performing models, underscoring the need for more challenging evaluation frameworks. We introduce MMLU-Pro+, an enhanced benchmark building upon MMLU-Pro to assess shortcut learning and higher-order reasoning in LLMs. By incorporating questions with multiple correct answers across diverse domains, MMLU-Pro+ tests LLMs' ability to engage in complex reasoning and resist simplistic problem-solving strategies. Our results show that MMLU-Pro+ maintains MMLU-Pro's difficulty while providing a more rigorous test of model discrimination, particularly in multi-correct answer scenarios. We introduce novel metrics like shortcut selection ratio and correct pair identification ratio, offering deeper insights into model behavior and anchoring bias. Evaluations of six state-of-the-art LLMs reveal significant performance gaps, highlighting variations in reasoning abilities and bias susceptibility. We release the dataset and evaluation codes at \url{https://github.com/asgsaeid/mmlu-pro-plus}.<|reference_end|>
@article{taghanaki2024mmlu-pro+:,
    title={MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in
  LLMs},
    author={Saeid Asgari Taghanaki, Aliasgahr Khani, Amir Khasahmadi},
    journal={arXiv preprint arXiv:2409.02257},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.02257},
    primaryClass={cs.CL cs.LG}
}