Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models
<|reference_start|>Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models: Although Large Language Models (LLMs) have demonstrated strong instruction-following ability, they are further supposed to be controlled and guided by rules in real-world scenarios to be safe, accurate, and intelligent. This demands the possession of inferential rule-following capability of LLMs. However, few works have made a clear evaluation of the inferential rule-following capability of LLMs. Previous studies that try to evaluate the inferential rule-following capability of LLMs fail to distinguish the inferential rule-following scenarios from the instruction-following scenarios. Therefore, this paper first clarifies the concept of inferential rule-following and proposes a comprehensive benchmark, RuleBench, to evaluate a diversified range of inferential rule-following abilities. Our experimental results on a variety of LLMs show that they are still limited in following rules. Our analysis based on the evaluation results provides insights into the improvements for LLMs toward a better inferential rule-following intelligent agent. We further propose Inferential Rule-Following Tuning (IRFT), which outperforms IFT in helping LLMs solve RuleBench. The data and code can be found at: https://anonymous.4open.science/r/llm-rule-following-B3E3/<|reference_end|>
@article{sun2024beyond,
    title={Beyond Instruction Following: Evaluating Inferential Rule Following of
  Large Language Models},
    author={Wangtao Sun, Chenxiang Zhang, XueYou Zhang, Xuanqing Yu, Ziyang Huang,
  Pei Chen, Haotian Xu, Shizhu He, Jun Zhao, Kang Liu},
    journal={arXiv preprint arXiv:2407.08440},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.08440},
    primaryClass={cs.CL cs.AI}
}