title: Premise Order Matters in Reasoning with Large Language Models
summary: Large language models (LLMs) have accomplished remarkable reasoning
  performance in various domains. However, in the domain of reasoning tasks, we
  discover a frailty: LLMs are surprisingly brittle to the ordering of the
  premises, despite the fact that such ordering does not alter the underlying
  task. In particular, we observe that LLMs achieve the best performance when the
  premise order aligns with the context required in intermediate reasoning steps.
  For example, in deductive reasoning tasks, presenting the premises in the same
  order as the ground truth proof in the prompt (as opposed to random ordering)
  drastically increases the model's accuracy. We first examine the effect of
  premise ordering on deductive reasoning on a variety of LLMs, and our
  evaluation shows that permuting the premise order can cause a performance drop
  of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to
  examine the ordering effect for mathematical problem-solving, and we again
  observe a significant drop in accuracy, relative to the original GSM8K
  benchmark.
paper_authors._id: 65cd969944908296a2cee745
paper_authors.name: Xinyun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:58:01.075Z
paper_authors._id: 64d0268001931c60161e026a
paper_authors.avatarUrl: /avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
paper_authors.fullname: Xinyun Chen
paper_authors.type: user
paper_authors.user: xinyunchen
paper_authors._id: 65cd969944908296a2cee746
paper_authors.name: Ryan A. Chi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:34:49.348Z
paper_authors._id: 65d2cd2e322cd4765ec4c3bc
paper_authors.avatarUrl: /avatars/f0ef8c8dff151afd6b84388998488091.svg
paper_authors.fullname: Ryan A. Chi
paper_authors.type: user
paper_authors.user: ryanchi
paper_authors._id: 65cd969944908296a2cee747
paper_authors.name: Xuezhi Wang
paper_authors._id: 65cd969944908296a2cee748
paper_authors.name: Denny Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T18:26:55.987Z
paper_authors._id: 65d4d62d493aeaf0bb295f1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
paper_authors.fullname: Denny Zhou
paper_authors.type: user
paper_authors.user: dennyzhou
paper_title: Premise Order Matters in Reasoning with Large Language Models
paper_summary: Large language models (LLMs) have accomplished remarkable reasoning
  performance in various domains. However, in the domain of reasoning tasks, we
  discover a frailty: LLMs are surprisingly brittle to the ordering of the
  premises, despite the fact that such ordering does not alter the underlying
  task. In particular, we observe that LLMs achieve the best performance when the
  premise order aligns with the context required in intermediate reasoning steps.
  For example, in deductive reasoning tasks, presenting the premises in the same
  order as the ground truth proof in the prompt (as opposed to random ordering)
  drastically increases the model's accuracy. We first examine the effect of
  premise ordering on deductive reasoning on a variety of LLMs, and our
  evaluation shows that permuting the premise order can cause a performance drop
  of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to
  examine the ordering effect for mathematical problem-solving, and we again
  observe a significant drop in accuracy, relative to the original GSM8K
  benchmark.
paper_ai_keywords: large language models (LLMs) deductive reasoning tasks premise order ground truth proof benchmark R-GSM GSM8K