title: The Impact of Reasoning Step Length on Large Language Models
summary: Chain of Thought (CoT) is significant in improving the reasoning abilities of
  large language models (LLMs). However, the correlation between the
  effectiveness of CoT and the length of reasoning steps in prompts remains
  largely unknown. To shed light on this, we have conducted several empirical
  experiments to explore the relations. Specifically, we design experiments that
  expand and compress the rationale reasoning steps within CoT demonstrations,
  while keeping all other factors constant. We have the following key findings.
  First, the results indicate that lengthening the reasoning steps in prompts,
  even without adding new information into the prompt, considerably enhances
  LLMs' reasoning abilities across multiple datasets. Alternatively, shortening
  the reasoning steps, even while preserving the key information, significantly
  diminishes the reasoning abilities of models. This finding highlights the
  importance of the number of steps in CoT prompts and provides practical
  guidance to make better use of LLMs' potential in complex problem-solving
  scenarios. Second, we also investigated the relationship between the
  performance of CoT and the rationales used in demonstrations. Surprisingly, the
  result shows that even incorrect rationales can yield favorable outcomes if
  they maintain the requisite length of inference. Third, we observed that the
  advantages of increasing reasoning steps are task-dependent: simpler tasks
  require fewer steps, whereas complex tasks gain significantly from longer
  inference sequences.
paper_authors._id: 659f5b037edb2a072edfaca6
paper_authors.name: Mingyu Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T10:54:45.675Z
paper_authors._id: 653b47abf44bc2eacef5c05f
paper_authors.avatarUrl: /avatars/aa4e3a9dd2e4c6c474790f169b70d52e.svg
paper_authors.fullname: Mingyu.Jin
paper_authors.type: user
paper_authors.user: Mingyu111
paper_authors._id: 659f5b037edb2a072edfaca7
paper_authors.name: Qinkai Yu
paper_authors._id: 659f5b037edb2a072edfaca8
paper_authors.name: Dong shu
paper_authors._id: 659f5b037edb2a072edfaca9
paper_authors.name: Haiyan Zhao
paper_authors._id: 659f5b037edb2a072edfacaa
paper_authors.name: Wenyue Hua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T10:55:32.448Z
paper_authors._id: 639a25aba2b0b1c9d85a51e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639a25aba2b0b1c9d85a51e8/pphz-MK62hPNbBkMHAkeR.jpeg
paper_authors.fullname: Wenyue Hua
paper_authors.type: user
paper_authors.user: wenyueH
paper_authors._id: 659f5b037edb2a072edfacab
paper_authors.name: Yanda Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T10:55:39.764Z
paper_authors._id: 64a7e092948958fd727b8204
paper_authors.avatarUrl: /avatars/1b52669fd3d01947a8035462e7063aea.svg
paper_authors.fullname: yanda meng
paper_authors.type: user
paper_authors.user: bigmaxmax
paper_authors._id: 659f5b037edb2a072edfacac
paper_authors.name: Yongfeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T10:55:46.463Z
paper_authors._id: 63b0deb7f2eb87a4d695055c
paper_authors.avatarUrl: /avatars/85a87fa82f7579019a0c6212f03b6480.svg
paper_authors.fullname: Yongfeng Zhang
paper_authors.type: user
paper_authors.user: evison
paper_authors._id: 659f5b037edb2a072edfacad
paper_authors.name: Mengnan Du
paper_title: The Impact of Reasoning Step Length on Large Language Models
paper_summary: Chain of Thought (CoT) is significant in improving the reasoning abilities of
  large language models (LLMs). However, the correlation between the
  effectiveness of CoT and the length of reasoning steps in prompts remains
  largely unknown. To shed light on this, we have conducted several empirical
  experiments to explore the relations. Specifically, we design experiments that
  expand and compress the rationale reasoning steps within CoT demonstrations,
  while keeping all other factors constant. We have the following key findings.
  First, the results indicate that lengthening the reasoning steps in prompts,
  even without adding new information into the prompt, considerably enhances
  LLMs' reasoning abilities across multiple datasets. Alternatively, shortening
  the reasoning steps, even while preserving the key information, significantly
  diminishes the reasoning abilities of models. This finding highlights the
  importance of the number of steps in CoT prompts and provides practical
  guidance to make better use of LLMs' potential in complex problem-solving
  scenarios. Second, we also investigated the relationship between the
  performance of CoT and the rationales used in demonstrations. Surprisingly, the
  result shows that even incorrect rationales can yield favorable outcomes if
  they maintain the requisite length of inference. Third, we observed that the
  advantages of increasing reasoning steps are task-dependent: simpler tasks
  require fewer steps, whereas complex tasks gain significantly from longer
  inference sequences.
paper_ai_keywords: Chain of Thought (CoT) large language models (LLMs) empirical experiments rationale reasoning steps reasoning abilities datasets performance inference complex problem-solving scenarios