Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
<|reference_start|>Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models: We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances dynamically, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks.<|reference_end|>
@article{kurtic2024mathador-lm:,
    title={Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large
  Language Models},
    author={Eldar Kurtic, Amir Moeini, Dan Alistarh},
    journal={arXiv preprint arXiv:2406.12572},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12572},
    primaryClass={cs.CL cs.AI cs.LG}
}