MR-BEN: A Comprehensive Meta-Reasoning Benchmark for Large Language Models
<|reference_start|>MR-BEN: A Comprehensive Meta-Reasoning Benchmark for Large Language Models: Large language models (LLMs) have shown increasing capability in problem-solving and decision-making, largely based on the step-by-step chain-of-thought reasoning processes. However, it has been increasingly challenging to evaluate the reasoning capability of LLMs. Concretely, existing outcome-based benchmarks begin to saturate and become less sufficient to monitor the progress. To this end, we present a process-based benchmark MR-BEN that demands a meta reasoning skill, where LMs are asked to locate and analyse potential errors in automatically generated reasoning steps. MR-BEN is a comprehensive benchmark comprising 5,975 questions collected from human experts, covering various subjects such as physics, chemistry, logic, coding, and more. Through our designed metrics for assessing meta-reasoning on this benchmark, we identify interesting limitations and weaknesses of current LLMs (open-source and closed-source models). For example, open-source models are seemingly comparable to GPT-4 on outcome-based benchmarks, but they lag far behind on our benchmark, revealing the underlying reasoning capability gap between them. Our dataset and codes are available on https://randolph-zeng.github.io/Mr-Ben.github.io/.<|reference_end|>
@article{zeng2024mr-ben:,
    title={MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in
  LLMs},
    author={Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen,
  Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen,
  Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang,
  Zhijiang Guo, Jiaya Jia},
    journal={arXiv preprint arXiv:2406.13975},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.13975},
    primaryClass={cs.CL cs.AI}
}