ChatGPT exhibits a strong capability for arithmetic reasoning by outperforming GPT-3.5 in
the majority of tasks [150]. However, its proficiency in mathematical reasoning still requires
improvement [5, 42, 250]. On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5,
which may be because ChatGPT is prone to uncertain responses, leading to poor performance
[5]. Through the poor performance of LLMs on task variants of counterfactual conditions, Wu
et al. [215] showed that the current LLMs have certain limitations in abstract reasoning ability. On
abstract reasoning, Gendron et al. [52] found that existing LLMs have very limited ability. In logical
reasoning, Liu et al. [115] indicated that ChatGPT and GPT-4 outperform traditional fine-tuning
methods on most benchmarks, demonstrating their superiority in logical reasoning. However,
both models face challenges when handling new and out-of-distribution data. ChatGPT does not
ASurveyonEvaluationofLargeLanguageModels