Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models
<|reference_start|>Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models: As Large Language Models (LLMs) continue to exhibit remarkable performance in natural language understanding tasks, there is a crucial need to measure their ability for human-like multi-step logical reasoning. Existing logical reasoning evaluation benchmarks often focus primarily on simplistic single-step or multi-step reasoning with a limited set of inference rules. Furthermore, the lack of datasets for evaluating non-monotonic reasoning represents a crucial gap since it aligns more closely with human-like reasoning. To address these limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths. Multi-LogiEval covers three logic types--propositional, first-order, and non-monotonic--consisting of more than 30 inference rules and more than 60 of their combinations with various depths. Leveraging this dataset, we conduct evaluations on a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral, employing a zero-shot chain-of-thought. Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5). We further conduct a thorough investigation of reasoning chains generated by LLMs which reveals several important findings. We believe that Multi-LogiEval facilitates future research for evaluating and enhancing the logical reasoning ability of LLMs. Data is available at https://github.com/Mihir3009/Multi-LogiEval.<|reference_end|>
@article{patel2024multi-logieval:,
    title={Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability
  of Large Language Models},
    author={Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi
  Nakamura, Neeraj Varshney, Chitta Baral},
    journal={arXiv preprint arXiv:2406.17169},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17169},
    primaryClass={cs.CL cs.AI}
}