Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
<|reference_start|>Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction: The rapid advancement of Large Language Models (LLMs) in the realm of mathematical reasoning necessitates comprehensive evaluations to gauge progress and inspire future directions. Existing assessments predominantly focus on problem-solving from the examinee perspective, overlooking a dual perspective of examiner regarding error identification and correction. From the examiner perspective, we define four evaluation tasks for error identification and correction along with a new dataset with annotated error types and steps. We also design diverse prompts to thoroughly evaluate eleven representative LLMs. Our principal findings indicate that GPT-4 outperforms all models, while open-source model LLaMA-2-7B demonstrates comparable abilities to closed-source models GPT-3.5 and Gemini Pro. Notably, calculation error proves the most challenging error type. Moreover, prompting LLMs with the error types can improve the average correction accuracy by 47.9\%. These results reveal potential directions for developing the mathematical reasoning abilities of LLMs. Our code and dataset is available on https://github.com/LittleCirc1e/EIC.<|reference_end|>
@article{li2024evaluating,
    title={Evaluating Mathematical Reasoning of Large Language Models: A Focus on
  Error Identification and Correction},
    author={Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, Fuli Feng},
    journal={arXiv preprint arXiv:2406.00755},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.00755},
    primaryClass={cs.CL cs.AI cs.LG}
}