title: Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on
    Tasks where Thinking Makes Humans Worse
summary: Chain-of-thought (CoT) prompting has become a widely used strategy for
  working with large language and multimodal models. While CoT has been shown to
  improve performance across many tasks, determining the settings in which it is
  effective remains an ongoing effort. In particular, it is still an open
  question in what settings CoT systematically reduces model performance. In this
  paper, we seek to identify the characteristics of tasks where CoT reduces
  performance by drawing inspiration from cognitive psychology, looking at cases
  where (i) verbal thinking or deliberation hurts performance in humans, and (ii)
  the constraints governing human performance generalize to language models.
  Three such cases are implicit statistical learning, visual recognition, and
  classifying with patterns containing exceptions. In extensive experiments
  across all three settings, we find that a diverse collection of
  state-of-the-art models exhibit significant drop-offs in performance (e.g., up
  to 36.3% absolute accuracy for OpenAI o1-preview compared to GPT-4o) when using
  inference-time reasoning compared to zero-shot counterparts. We also identify
  three tasks that satisfy condition (i) but not (ii), and find that while verbal
  thinking reduces human performance in these tasks, CoT retains or increases
  model performance. Overall, our results show that while there is not an exact
  parallel between the cognitive processes of models and those of humans,
  considering cases where thinking has negative consequences for human
  performance can help us identify settings where it negatively impacts models.
  By connecting the literature on human deliberation with evaluations of CoT, we
  offer a new tool that can be used in understanding the impact of prompt choices
  and inference-time reasoning.
paper_authors._id: 672221bd3c54079d2cdeb460
paper_authors.name: Ryan Liu
paper_authors._id: 672221bd3c54079d2cdeb461
paper_authors.name: Jiayi Geng
paper_authors._id: 672221bd3c54079d2cdeb462
paper_authors.name: Addison J. Wu
paper_authors._id: 672221bd3c54079d2cdeb463
paper_authors.name: Ilia Sucholutsky
paper_authors._id: 672221bd3c54079d2cdeb464
paper_authors.name: Tania Lombrozo
paper_authors._id: 672221bd3c54079d2cdeb465
paper_authors.name: Thomas L. Griffiths
paper_title: Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on
    Tasks where Thinking Makes Humans Worse
paper_summary: Chain-of-thought (CoT) prompting has become a widely used strategy for
  working with large language and multimodal models. While CoT has been shown to
  improve performance across many tasks, determining the settings in which it is
  effective remains an ongoing effort. In particular, it is still an open
  question in what settings CoT systematically reduces model performance. In this
  paper, we seek to identify the characteristics of tasks where CoT reduces
  performance by drawing inspiration from cognitive psychology, looking at cases
  where (i) verbal thinking or deliberation hurts performance in humans, and (ii)
  the constraints governing human performance generalize to language models.
  Three such cases are implicit statistical learning, visual recognition, and
  classifying with patterns containing exceptions. In extensive experiments
  across all three settings, we find that a diverse collection of
  state-of-the-art models exhibit significant drop-offs in performance (e.g., up
  to 36.3% absolute accuracy for OpenAI o1-preview compared to GPT-4o) when using
  inference-time reasoning compared to zero-shot counterparts. We also identify
  three tasks that satisfy condition (i) but not (ii), and find that while verbal
  thinking reduces human performance in these tasks, CoT retains or increases
  model performance. Overall, our results show that while there is not an exact
  parallel between the cognitive processes of models and those of humans,
  considering cases where thinking has negative consequences for human
  performance can help us identify settings where it negatively impacts models.
  By connecting the literature on human deliberation with evaluations of CoT, we
  offer a new tool that can be used in understanding the impact of prompt choices
  and inference-time reasoning.
paper_ai_summary: Chain-of-thought prompting reduces performance in tasks such as implicit statistical learning, visual recognition, and classifying with patterns containing exceptions, while it retains or increases performance in tasks where humans benefit from verbal thinking.
paper_ai_keywords: chain-of-thought prompting implicit statistical learning visual recognition patterns containing exceptions inference-time reasoning zero-shot counterparts