title: Training Large Language Models to Reason in a Continuous Latent Space
summary: Large language models (LLMs) are restricted to reason in the "language
  space", where they typically express the reasoning process with a
  chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue
  that language space may not always be optimal for reasoning. For example, most
  word tokens are primarily for textual coherence and not essential for
  reasoning, while some critical tokens require complex planning and pose huge
  challenges to LLMs. To explore the potential of LLM reasoning in an
  unrestricted latent space instead of using natural language, we introduce a new
  paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden
  state of the LLM as a representation of the reasoning state (termed "continuous
  thought"). Rather than decoding this into a word token, we feed it back to the
  LLM as the subsequent input embedding directly in the continuous space.
  Experiments show that Coconut can effectively augment the LLM on several
  reasoning tasks. This novel latent reasoning paradigm leads to emergent
  advanced reasoning patterns: the continuous thought can encode multiple
  alternative next reasoning steps, allowing the model to perform a breadth-first
  search (BFS) to solve the problem, rather than prematurely committing to a
  single deterministic path like CoT. Coconut outperforms CoT in certain logical
  reasoning tasks that require substantial backtracking during planning, with
  fewer thinking tokens during inference. These findings demonstrate the promise
  of latent reasoning and offer valuable insights for future research.
paper_authors._id: 6757e0f3411d2c9cb4906fef
paper_authors.name: Shibo Hao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:15:26.794Z
paper_authors._id: 660ee5df35d092e3fc2a3685
paper_authors.avatarUrl: /avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
paper_authors.fullname: Shibo Hao
paper_authors.type: user
paper_authors.user: Shibo-UCSD
paper_authors._id: 6757e0f3411d2c9cb4906ff0
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:15:21.216Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 6757e0f3411d2c9cb4906ff1
paper_authors.name: DiJia Su
paper_authors._id: 6757e0f3411d2c9cb4906ff2
paper_authors.name: Xian Li
paper_authors._id: 6757e0f3411d2c9cb4906ff3
paper_authors.name: Zhiting Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:15:00.114Z
paper_authors._id: 665bfa1b0d71762b8613282d
paper_authors.avatarUrl: /avatars/edbde7b1b47032339a1ecc59f8ea8f1a.svg
paper_authors.fullname: Zhiting Hu
paper_authors.type: user
paper_authors.user: zhitinghu
paper_authors._id: 6757e0f3411d2c9cb4906ff4
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:14:54.266Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 6757e0f3411d2c9cb4906ff5
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:14:48.475Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_title: Training Large Language Models to Reason in a Continuous Latent Space
paper_summary: Large language models (LLMs) are restricted to reason in the "language
  space", where they typically express the reasoning process with a
  chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue
  that language space may not always be optimal for reasoning. For example, most
  word tokens are primarily for textual coherence and not essential for
  reasoning, while some critical tokens require complex planning and pose huge
  challenges to LLMs. To explore the potential of LLM reasoning in an
  unrestricted latent space instead of using natural language, we introduce a new
  paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden
  state of the LLM as a representation of the reasoning state (termed "continuous
  thought"). Rather than decoding this into a word token, we feed it back to the
  LLM as the subsequent input embedding directly in the continuous space.
  Experiments show that Coconut can effectively augment the LLM on several
  reasoning tasks. This novel latent reasoning paradigm leads to emergent
  advanced reasoning patterns: the continuous thought can encode multiple
  alternative next reasoning steps, allowing the model to perform a breadth-first
  search (BFS) to solve the problem, rather than prematurely committing to a
  single deterministic path like CoT. Coconut outperforms CoT in certain logical
  reasoning tasks that require substantial backtracking during planning, with
  fewer thinking tokens during inference. These findings demonstrate the promise
  of latent reasoning and offer valuable insights for future research.
paper_ai_summary: A new reasoning paradigm, Coconut, utilizes continuous representations in latent space to enhance LLM performance on complex reasoning tasks, particularly those requiring backtracking.
paper_ai_keywords: LLMs language space chain-of-thought (CoT) latent space continuous thought continuous space breadth-first search (BFS)