title: Quiet-STaR: Language Models Can Teach Themselves to Think Before
    Speaking
summary: When writing and talking, people sometimes pause to think. Although
  reasoning-focused works have often framed reasoning as a method of answering
  questions or completing agentic tasks, reasoning is implicit in almost all
  written text. For example, this applies to the steps not stated between the
  lines of a proof or to the theory of mind underlying a conversation. In the
  Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned
  by inferring rationales from few-shot examples in question-answering and
  learning from those that lead to a correct answer. This is a highly constrained
  setting -- ideally, a language model could instead learn to infer unstated
  rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR
  in which LMs learn to generate rationales at each token to explain future text,
  improving their predictions. We address key challenges, including 1) the
  computational cost of generating continuations, 2) the fact that the LM does
  not initially know how to generate or use internal thoughts, and 3) the need to
  predict beyond individual next tokens. To resolve these, we propose a tokenwise
  parallel sampling algorithm, using learnable tokens indicating a thought's
  start and end, and an extended teacher-forcing technique. Encouragingly,
  generated rationales disproportionately help model difficult-to-predict tokens
  and improve the LM's ability to directly answer difficult questions. In
  particular, after continued pretraining of an LM on a corpus of internet text
  with Quiet-STaR, we find zero-shot improvements on GSM8K
  (5.9%rightarrow10.9%) and CommonsenseQA (36.3%rightarrow47.2%) and
  observe a perplexity improvement of difficult tokens in natural text.
  Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR
  marks a step towards LMs that can learn to reason in a more general and
  scalable way.
paper_authors._id: 65f3b1f154ecda1ecb3dde3c
paper_authors.name: Eric Zelikman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:18:12.848Z
paper_authors._id: 626aa3efe9326611a509a2fc
paper_authors.avatarUrl: /avatars/4cea5cbd4ee918f1de7ccf191bfdf118.svg
paper_authors.fullname: Eric Zelikman
paper_authors.type: user
paper_authors.user: ezelikman
paper_authors._id: 65f3b1f154ecda1ecb3dde3d
paper_authors.name: Georges Harik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:18:22.832Z
paper_authors._id: 6200d379a3f03cce621e1eaa
paper_authors.avatarUrl: /avatars/d452b3bb0fe7f9299463ea551601eebf.svg
paper_authors.fullname: Georges Harik
paper_authors.type: user
paper_authors.user: gharik
paper_authors._id: 65f3b1f154ecda1ecb3dde3e
paper_authors.name: Yijia Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:24:55.599Z
paper_authors._id: 6171514808b4ddb7392f2eb0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6171514808b4ddb7392f2eb0/6TjLoLeKwos0JoGcTueeI.jpeg
paper_authors.fullname: Yijia Shao
paper_authors.type: user
paper_authors.user: EchoShao8899
paper_authors._id: 65f3b1f154ecda1ecb3dde3f
paper_authors.name: Varuna Jayasiri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:18:37.809Z
paper_authors._id: 62ceaf9654477562dcb4afda
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ceaf9654477562dcb4afda/cWLv6_qWAxY7_biXfjiGU.jpeg
paper_authors.fullname: Varuna Jayasiri
paper_authors.type: user
paper_authors.user: vpj
paper_authors._id: 65f3b1f154ecda1ecb3dde40
paper_authors.name: Nick Haber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:18:43.946Z
paper_authors._id: 657d3b7d69cfe9b4cd0bfa04
paper_authors.avatarUrl: /avatars/a97b0df7187d08ccca8a6a597cba7f67.svg
paper_authors.fullname: Nick Haber
paper_authors.type: user
paper_authors.user: nhaber
paper_authors._id: 65f3b1f154ecda1ecb3dde41
paper_authors.name: Noah D. Goodman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:49:42.038Z
paper_authors._id: 67321274c1f20c742bcf7a8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
paper_authors.fullname: Noah D. Goodman
paper_authors.type: user
paper_authors.user: ngoodman
paper_title: Quiet-STaR: Language Models Can Teach Themselves to Think Before
    Speaking
paper_summary: When writing and talking, people sometimes pause to think. Although
  reasoning-focused works have often framed reasoning as a method of answering
  questions or completing agentic tasks, reasoning is implicit in almost all
  written text. For example, this applies to the steps not stated between the
  lines of a proof or to the theory of mind underlying a conversation. In the
  Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned
  by inferring rationales from few-shot examples in question-answering and
  learning from those that lead to a correct answer. This is a highly constrained
  setting -- ideally, a language model could instead learn to infer unstated
  rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR
  in which LMs learn to generate rationales at each token to explain future text,
  improving their predictions. We address key challenges, including 1) the
  computational cost of generating continuations, 2) the fact that the LM does
  not initially know how to generate or use internal thoughts, and 3) the need to
  predict beyond individual next tokens. To resolve these, we propose a tokenwise
  parallel sampling algorithm, using learnable tokens indicating a thought's
  start and end, and an extended teacher-forcing technique. Encouragingly,
  generated rationales disproportionately help model difficult-to-predict tokens
  and improve the LM's ability to directly answer difficult questions. In
  particular, after continued pretraining of an LM on a corpus of internet text
  with Quiet-STaR, we find zero-shot improvements on GSM8K
  (5.9%rightarrow10.9%) and CommonsenseQA (36.3%rightarrow47.2%) and
  observe a perplexity improvement of difficult tokens in natural text.
  Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR
  marks a step towards LMs that can learn to reason in a more general and
  scalable way.
paper_ai_summary: Quiet-STaR is an extension of the Self-Taught Reasoner that teaches language models to infer and generate rationales for each token, improving their ability to predict difficult tokens and answer questions without task-specific fine-tuning.
paper_ai_keywords: Self-Taught Reasoner Quiet-STaR language models tokenwise parallel sampling internal thoughts extended teacher-forcing GSM8K CommonsenseQA perplexity reasoning