title: Language Models Don't Always Say What They Think: Unfaithful
    Explanations in Chain-of-Thought Prompting
summary: Large Language Models (LLMs) can achieve strong performance on many tasks by
  producing step-by-step reasoning before giving a final output, often referred
  to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT
  explanations as the LLM's process for solving a task. However, we find that CoT
  explanations can systematically misrepresent the true reason for a model's
  prediction. We demonstrate that CoT explanations can be heavily influenced by
  adding biasing features to model inputs -- e.g., by reordering the
  multiple-choice options in a few-shot prompt to make the answer always "(A)" --
  which models systematically fail to mention in their explanations. When we bias
  models toward incorrect answers, they frequently generate CoT explanations
  supporting those answers. This causes accuracy to drop by as much as 36% on a
  suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI
  and Claude 1.0 from Anthropic. On a social-bias task, model explanations
  justify giving answers in line with stereotypes without mentioning the
  influence of these social biases. Our findings indicate that CoT explanations
  can be plausible yet misleading, which risks increasing our trust in LLMs
  without guaranteeing their safety. CoT is promising for explainability, but our
  results highlight the need for targeted efforts to evaluate and improve
  explanation faithfulness.
paper_authors._id: 6459a6d35e44a749d5a58d6a
paper_authors.name: Miles Turpin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T15:56:31.825Z
paper_authors._id: 61708e0dc22c7566454415d9
paper_authors.avatarUrl: /avatars/6403a103280ed4258b3b4400d603294d.svg
paper_authors.fullname: Miles Turpin
paper_authors.type: user
paper_authors.user: miles
paper_authors._id: 6459a6d35e44a749d5a58d6b
paper_authors.name: Julian Michael
paper_authors._id: 6459a6d35e44a749d5a58d6c
paper_authors.name: Ethan Perez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T15:57:56.275Z
paper_authors._id: 62bb28dc4d3faf9c8938f742
paper_authors.avatarUrl: /avatars/cceeb6b8e679b7460519024cd5e301b5.svg
paper_authors.fullname: Ethan
paper_authors.type: user
paper_authors.user: ethanjperez
paper_authors._id: 6459a6d35e44a749d5a58d6d
paper_authors.name: Samuel R. Bowman
paper_title: Language Models Don't Always Say What They Think: Unfaithful
    Explanations in Chain-of-Thought Prompting
paper_summary: Large Language Models (LLMs) can achieve strong performance on many tasks by
  producing step-by-step reasoning before giving a final output, often referred
  to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT
  explanations as the LLM's process for solving a task. However, we find that CoT
  explanations can systematically misrepresent the true reason for a model's
  prediction. We demonstrate that CoT explanations can be heavily influenced by
  adding biasing features to model inputs -- e.g., by reordering the
  multiple-choice options in a few-shot prompt to make the answer always "(A)" --
  which models systematically fail to mention in their explanations. When we bias
  models toward incorrect answers, they frequently generate CoT explanations
  supporting those answers. This causes accuracy to drop by as much as 36% on a
  suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI
  and Claude 1.0 from Anthropic. On a social-bias task, model explanations
  justify giving answers in line with stereotypes without mentioning the
  influence of these social biases. Our findings indicate that CoT explanations
  can be plausible yet misleading, which risks increasing our trust in LLMs
  without guaranteeing their safety. CoT is promising for explainability, but our
  results highlight the need for targeted efforts to evaluate and improve
  explanation faithfulness.
paper_ai_keywords: Large Language Models (LLMs) chain-of-thought reasoning (CoT) step-by-step reasoning biasing features few-shot prompt multiple-choice options BIG-Bench Hard social-bias task stereotypes explanation faithfulness