title: System 2 Attention (is something you might need too)
summary: Soft attention in Transformer-based Large Language Models (LLMs) is
  susceptible to incorporating irrelevant information from the context into its
  latent representations, which adversely affects next token generations. To help
  rectify these issues, we introduce System 2 Attention (S2A), which leverages
  the ability of LLMs to reason in natural language and follow instructions in
  order to decide what to attend to. S2A regenerates the input context to only
  include the relevant portions, before attending to the regenerated context to
  elicit the final response. In experiments, S2A outperforms standard
  attention-based LLMs on three tasks containing opinion or irrelevant
  information, QA, math word problems and longform generation, where S2A
  increases factuality and objectivity, and decreases sycophancy.
paper_authors._id: 655c2e1bb27f103d7b8eb5c9
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:15:56.450Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 655c2e1bb27f103d7b8eb5ca
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:32.514Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_title: System 2 Attention (is something you might need too)
paper_summary: Soft attention in Transformer-based Large Language Models (LLMs) is
  susceptible to incorporating irrelevant information from the context into its
  latent representations, which adversely affects next token generations. To help
  rectify these issues, we introduce System 2 Attention (S2A), which leverages
  the ability of LLMs to reason in natural language and follow instructions in
  order to decide what to attend to. S2A regenerates the input context to only
  include the relevant portions, before attending to the regenerated context to
  elicit the final response. In experiments, S2A outperforms standard
  attention-based LLMs on three tasks containing opinion or irrelevant
  information, QA, math word problems and longform generation, where S2A
  increases factuality and objectivity, and decreases sycophancy.
paper_ai_summary: System 2 Attention in Transformer-based Large Language Models improves factual accuracy and reduces bias by refining input context.
paper_ai_keywords: soft attention Transformer-based Large Language Models (LLMs) System 2 Attention (S2A) natural language reasoning context regeneration factuality objectivity sycophancy QA math word problems longform generation