Training Large Language Models to Reason in a Continuous Latent Space
Large language models (LLMs) are restricted to reason in the "language
space", where they typically express the reasoning process with a
chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue
that language space may not always be optimal for reasoning. For example, most
word tokens are primarily for textual coherence and not essential for
reasoning, while some critical tokens require complex planning and pose huge
challenges to LLMs. To explore the potential of LLM reasoning in an
unrestricted latent space instead of using natural language, we introduce a new
paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden
state of the LLM as a representation of the reasoning state (termed "continuous
thought"). Rather than decoding this into a word token, we feed it back to the
LLM as the subsequent input embedding directly in the continuous space.
Experiments show that Coconut can effectively augment the LLM on several
reasoning tasks. This novel latent reasoning paradigm leads to emergent
advanced reasoning patterns: the continuous thought can encode multiple
alternative next reasoning steps, allowing the model to perform a breadth-first
search (BFS) to solve the problem, rather than prematurely committing to a
single deterministic path like CoT. Coconut outperforms CoT in certain logical
reasoning tasks that require substantial backtracking during planning, with
fewer thinking tokens during inference. These findings demonstrate the promise
of latent reasoning and offer valuable insights for future research.
6757e0f3411d2c9cb4906fef
Shibo Hao
admin_assigned
2024-12-10T16:15:26.794Z
660ee5df35d092e3fc2a3685
/avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
Shibo Hao
user
Shibo-UCSD
6757e0f3411d2c9cb4906ff0
Sainbayar Sukhbaatar
admin_assigned
2024-12-10T16:15:21.216Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
6757e0f3411d2c9cb4906ff1
DiJia Su
6757e0f3411d2c9cb4906ff2
Xian Li
6757e0f3411d2c9cb4906ff3
Zhiting Hu
admin_assigned
2024-12-10T16:15:00.114Z
665bfa1b0d71762b8613282d
/avatars/edbde7b1b47032339a1ecc59f8ea8f1a.svg
Zhiting Hu
user
zhitinghu
6757e0f3411d2c9cb4906ff4
Jason Weston
admin_assigned
2024-12-10T16:14:54.266Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
6757e0f3411d2c9cb4906ff5
Yuandong Tian
admin_assigned
2024-12-10T16:14:48.475Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
Training Large Language Models to Reason in a Continuous Latent Space
Large language models (LLMs) are restricted to reason in the "language
space", where they typically express the reasoning process with a
chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue
that language space may not always be optimal for reasoning. For example, most
word tokens are primarily for textual coherence and not essential for
reasoning, while some critical tokens require complex planning and pose huge
challenges to LLMs. To explore the potential of LLM reasoning in an
unrestricted latent space instead of using natural language, we introduce a new
paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden
state of the LLM as a representation of the reasoning state (termed "continuous
thought"). Rather than decoding this into a word token, we feed it back to the
LLM as the subsequent input embedding directly in the continuous space.
Experiments show that Coconut can effectively augment the LLM on several
reasoning tasks. This novel latent reasoning paradigm leads to emergent
advanced reasoning patterns: the continuous thought can encode multiple
alternative next reasoning steps, allowing the model to perform a breadth-first
search (BFS) to solve the problem, rather than prematurely committing to a
single deterministic path like CoT. Coconut outperforms CoT in certain logical
reasoning tasks that require substantial backtracking during planning, with
fewer thinking tokens during inference. These findings demonstrate the promise
of latent reasoning and offer valuable insights for future research.
A new reasoning paradigm, Coconut, utilizes continuous representations in latent space to enhance LLM performance on complex reasoning tasks, particularly those requiring backtracking.
LLMs language space chain-of-thought (CoT) latent space continuous thought continuous space breadth-first search (BFS)