title: RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World
    Scenarios
summary: This paper introduces RuleArena, a novel and challenging benchmark designed
  to evaluate the ability of large language models (LLMs) to follow complex,
  real-world rules in reasoning. Covering three practical domains -- airline
  baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
  proficiency in handling intricate natural language instructions that demand
  long-context understanding, logical reasoning, and accurate mathematical
  computation. Two key attributes distinguish RuleArena from traditional
  rule-based reasoning benchmarks: (1) it extends beyond standard first-order
  logic representations, and (2) it is grounded in authentic, practical
  scenarios, providing insights into the suitability and reliability of LLMs for
  real-world applications. Our findings reveal several notable limitations in
  LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
  becoming confused by similar but distinct regulations, (2) they cannot
  consistently perform accurate mathematical computations, even when they
  correctly identify the relevant rules, and (3) in general, they perform poorly
  in the benchmark. These results highlight significant challenges in advancing
  LLMs' rule-guided reasoning capabilities in real-life applications.
paper_authors._id: 675bca99935c1403e2c788a2
paper_authors.name: Ruiwen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:18:29.856Z
paper_authors._id: 66ad5084bcede3c1e3445395
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66ad5084bcede3c1e3445395/egBqNuArOy0rz7aGVB-T6.jpeg
paper_authors.fullname: Ruiwen Zhou
paper_authors.type: user
paper_authors.user: rwzhou
paper_authors._id: 675bca99935c1403e2c788a3
paper_authors.name: Wenyue Hua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:18:35.880Z
paper_authors._id: 639a25aba2b0b1c9d85a51e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639a25aba2b0b1c9d85a51e8/pphz-MK62hPNbBkMHAkeR.jpeg
paper_authors.fullname: Wenyue Hua
paper_authors.type: user
paper_authors.user: wenyueH
paper_authors._id: 675bca99935c1403e2c788a4
paper_authors.name: Liangming Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:18:41.535Z
paper_authors._id: 628cd837ef14f971b699e4d4
paper_authors.avatarUrl: /avatars/cf64c88e71c730fc0a54df59e17bffbb.svg
paper_authors.fullname: Liangming Pan
paper_authors.type: user
paper_authors.user: Liangming
paper_authors._id: 675bca99935c1403e2c788a5
paper_authors.name: Sitao Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:18:48.386Z
paper_authors._id: 66d74853f3af37296e52e921
paper_authors.avatarUrl: /avatars/0683450a86ef3aa96f0813beaa078fa2.svg
paper_authors.fullname: Sitao Cheng
paper_authors.type: user
paper_authors.user: Stauskas
paper_authors._id: 675bca99935c1403e2c788a6
paper_authors.name: Xiaobao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:18:54.348Z
paper_authors._id: 64cb02869e30a46f7b80b355
paper_authors.avatarUrl: /avatars/81ce4ba78826b54f0e1b53eeaff87ee6.svg
paper_authors.fullname: Xiaobao Wu
paper_authors.type: user
paper_authors.user: bobxwu
paper_authors._id: 675bca99935c1403e2c788a7
paper_authors.name: En Yu
paper_authors._id: 675bca99935c1403e2c788a8
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:19:13.368Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_title: RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World
    Scenarios
paper_summary: This paper introduces RuleArena, a novel and challenging benchmark designed
  to evaluate the ability of large language models (LLMs) to follow complex,
  real-world rules in reasoning. Covering three practical domains -- airline
  baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
  proficiency in handling intricate natural language instructions that demand
  long-context understanding, logical reasoning, and accurate mathematical
  computation. Two key attributes distinguish RuleArena from traditional
  rule-based reasoning benchmarks: (1) it extends beyond standard first-order
  logic representations, and (2) it is grounded in authentic, practical
  scenarios, providing insights into the suitability and reliability of LLMs for
  real-world applications. Our findings reveal several notable limitations in
  LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
  becoming confused by similar but distinct regulations, (2) they cannot
  consistently perform accurate mathematical computations, even when they
  correctly identify the relevant rules, and (3) in general, they perform poorly
  in the benchmark. These results highlight significant challenges in advancing
  LLMs' rule-guided reasoning capabilities in real-life applications.