Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange
<|reference_start|>Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange: Large Language Models (LLMs) have demonstrated exceptional capabilities in various natural language tasks, often achieving performances that surpass those of humans. Despite these advancements, the domain of mathematics presents a distinctive challenge, primarily due to its specialized structure and the precision it demands. In this study, we adopted a two-step approach for investigating the proficiency of LLMs in answering mathematical questions. First, we employ the most effective LLMs, as identified by their performance on math question-answer benchmarks, to generate answers to 78 questions from the Math Stack Exchange (MSE). Second, a case analysis is conducted on the LLM that showed the highest performance, focusing on the quality and accuracy of its answers through manual evaluation. We found that GPT-4 performs best (nDCG of 0.48 and P@10 of 0.37) amongst existing LLMs fine-tuned for answering mathematics questions and outperforms the current best approach on ArqMATH3 Task1, considering P@10. Our Case analysis indicates that while the GPT-4 can generate relevant responses in certain instances, it does not consistently answer all questions accurately. This paper explores the current limitations of LLMs in navigating complex mathematical problem-solving. Through case analysis, we shed light on the gaps in LLM capabilities within mathematics, thereby setting the stage for future research and advancements in AI-driven mathematical reasoning. We make our code and findings publicly available for research: \url{https://github.com/gipplab/LLM-Investig-MathStackExchange}<|reference_end|>
@article{satpute2024can,
    title={Can LLMs Master Math? Investigating Large Language Models on Math Stack
  Exchange},
    author={Ankit Satpute and Noah Giessing and Andre Greiner-Petter and Moritz
  Schubotz and Olaf Teschke and Akiko Aizawa and Bela Gipp},
    journal={arXiv preprint arXiv:2404.00344},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.00344},
    primaryClass={cs.CL cs.AI cs.IR}
}