title: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth
    Approach
summary: We study a novel language model architecture that is capable of scaling
  test-time computation by implicitly reasoning in latent space. Our model works
  by iterating a recurrent block, thereby unrolling to arbitrary depth at
  test-time. This stands in contrast to mainstream reasoning models that scale up
  compute by producing more tokens. Unlike approaches based on chain-of-thought,
  our approach does not require any specialized training data, can work with
  small context windows, and can capture types of reasoning that are not easily
  represented in words. We scale a proof-of-concept model to 3.5 billion
  parameters and 800 billion tokens. We show that the resulting model can improve
  its performance on reasoning benchmarks, sometimes dramatically, up to a
  computation load equivalent to 50 billion parameters.
paper_authors._id: 67a97e27495b23306cd5ea56
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:10:13.470Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 67a97e27495b23306cd5ea57
paper_authors.name: Sean McLeish
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T13:11:57.835Z
paper_authors._id: 65255f1073a043e50d043641
paper_authors.avatarUrl: /avatars/257085f01c439d7c84787a4e6d085b3d.svg
paper_authors.fullname: Sean McLeish
paper_authors.type: user
paper_authors.user: smcleish
paper_authors._id: 67a97e27495b23306cd5ea58
paper_authors.name: Neel Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:10:34.976Z
paper_authors._id: 63e2b1ec282ee5f9624cfbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
paper_authors.fullname: Neel Jain
paper_authors.type: user
paper_authors.user: nsjain
paper_authors._id: 67a97e27495b23306cd5ea59
paper_authors.name: John Kirchenbauer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:10:40.754Z
paper_authors._id: 63d98af1897746d6496177df
paper_authors.avatarUrl: /avatars/c5d0031c796a3c11bcb0d01b959168dc.svg
paper_authors.fullname: John Kirchenbauer
paper_authors.type: user
paper_authors.user: jwkirchenbauer
paper_authors._id: 67a97e27495b23306cd5ea5a
paper_authors.name: Siddharth Singh
paper_authors._id: 67a97e27495b23306cd5ea5b
paper_authors.name: Brian R. Bartoldson
paper_authors._id: 67a97e27495b23306cd5ea5c
paper_authors.name: Bhavya Kailkhura
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:11:04.907Z
paper_authors._id: 65cb79db6427380bc21261e2
paper_authors.avatarUrl: /avatars/a003eb5d0955417329c1a4170ae65879.svg
paper_authors.fullname: Bhavya Kailkhura
paper_authors.type: user
paper_authors.user: bhavyakailkhura
paper_authors._id: 67a97e27495b23306cd5ea5d
paper_authors.name: Abhinav Bhatele
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:11:12.069Z
paper_authors._id: 6361d9ce6bd72c97d005b4db
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6361d9ce6bd72c97d005b4db/vjKaW2JFavVffoRxwgFwn.jpeg
paper_authors.fullname: Abhinav Bhatele
paper_authors.type: user
paper_authors.user: bhatele
paper_authors._id: 67a97e27495b23306cd5ea5e
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:11:20.220Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_title: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth
    Approach
paper_summary: We study a novel language model architecture that is capable of scaling
  test-time computation by implicitly reasoning in latent space. Our model works
  by iterating a recurrent block, thereby unrolling to arbitrary depth at
  test-time. This stands in contrast to mainstream reasoning models that scale up
  compute by producing more tokens. Unlike approaches based on chain-of-thought,
  our approach does not require any specialized training data, can work with
  small context windows, and can capture types of reasoning that are not easily
  represented in words. We scale a proof-of-concept model to 3.5 billion
  parameters and 800 billion tokens. We show that the resulting model can improve
  its performance on reasoning benchmarks, sometimes dramatically, up to a
  computation load equivalent to 50 billion parameters.
paper_ai_summary: A novel language model scales test-time computation by reasoning in latent space, outperforming token-based models on reasoning benchmarks.
paper_ai_keywords: recurrent block latent space chain-of-thought reasoning benchmarks