Premise Order Matters in Reasoning with Large Language Models
Large language models (LLMs) have accomplished remarkable reasoning
performance in various domains. However, in the domain of reasoning tasks, we
discover a frailty: LLMs are surprisingly brittle to the ordering of the
premises, despite the fact that such ordering does not alter the underlying
task. In particular, we observe that LLMs achieve the best performance when the
premise order aligns with the context required in intermediate reasoning steps.
For example, in deductive reasoning tasks, presenting the premises in the same
order as the ground truth proof in the prompt (as opposed to random ordering)
drastically increases the model's accuracy. We first examine the effect of
premise ordering on deductive reasoning on a variety of LLMs, and our
evaluation shows that permuting the premise order can cause a performance drop
of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to
examine the ordering effect for mathematical problem-solving, and we again
observe a significant drop in accuracy, relative to the original GSM8K
benchmark.
65cd969944908296a2cee745
Xinyun Chen
admin_assigned
2024-02-15T09:58:01.075Z
64d0268001931c60161e026a
/avatars/0ea48c47b0270b449fd6b97b495e64a6.svg
Xinyun Chen
user
xinyunchen
65cd969944908296a2cee746
Ryan A. Chi
claimed_verified
2024-02-19T09:34:49.348Z
65d2cd2e322cd4765ec4c3bc
/avatars/f0ef8c8dff151afd6b84388998488091.svg
Ryan A. Chi
user
ryanchi
65cd969944908296a2cee747
Xuezhi Wang
65cd969944908296a2cee748
Denny Zhou
claimed_verified
2024-02-20T18:26:55.987Z
65d4d62d493aeaf0bb295f1b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
Denny Zhou
user
dennyzhou
Premise Order Matters in Reasoning with Large Language Models
Large language models (LLMs) have accomplished remarkable reasoning
performance in various domains. However, in the domain of reasoning tasks, we
discover a frailty: LLMs are surprisingly brittle to the ordering of the
premises, despite the fact that such ordering does not alter the underlying
task. In particular, we observe that LLMs achieve the best performance when the
premise order aligns with the context required in intermediate reasoning steps.
For example, in deductive reasoning tasks, presenting the premises in the same
order as the ground truth proof in the prompt (as opposed to random ordering)
drastically increases the model's accuracy. We first examine the effect of
premise ordering on deductive reasoning on a variety of LLMs, and our
evaluation shows that permuting the premise order can cause a performance drop
of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to
examine the ordering effect for mathematical problem-solving, and we again
observe a significant drop in accuracy, relative to the original GSM8K
benchmark.
LLMs exhibit reduced performance on reasoning tasks when premises are reordered, affecting both deductive reasoning and mathematical problem-solving.
LLMs reasoning performance premise ordering deductive reasoning R-GSM GSM8K