title: Reasoning Language Models: A Blueprint
summary: Reasoning language models (RLMs), also known as Large Reasoning Models
  (LRMs), such as OpenAI's o1 and o3, DeepSeek-V3, and Alibaba's QwQ, have
  redefined AI's problem-solving capabilities by extending large language models
  (LLMs) with advanced reasoning mechanisms. Yet, their high costs, proprietary
  nature, and complex architectures - uniquely combining Reinforcement Learning
  (RL), search heuristics, and LLMs - present accessibility and scalability
  challenges. To address these, we propose a comprehensive blueprint that
  organizes RLM components into a modular framework, based on a survey and
  analysis of all RLM works. This blueprint incorporates diverse reasoning
  structures (chains, trees, graphs, and nested forms), reasoning strategies
  (e.g., Monte Carlo Tree Search, Beam Search), RL concepts (policy, value models
  and others), and supervision schemes (Output-Based and Process-Based
  Supervision). We also provide detailed mathematical formulations and
  algorithmic specifications to simplify RLM implementation. By showing how
  schemes like LLaMA-Berry, QwQ, Journey Learning, and Graph of Thoughts fit as
  special cases, we demonstrate the blueprint's versatility and unifying
  potential. To illustrate its utility, we introduce x1, a modular implementation
  for rapid RLM prototyping and experimentation. Using x1 and a literature
  review, we provide key insights, such as multi-phase training for policy and
  value models, and the importance of familiar training distributions. Finally,
  we outline how RLMs can integrate with a broader LLM ecosystem, including tools
  and databases. Our work demystifies RLM construction, democratizes advanced
  reasoning capabilities, and fosters innovation, aiming to mitigate the gap
  between "rich AI" and "poor AI" by lowering barriers to RLM development and
  experimentation.
paper_authors._id: 6790772b8d7df822f1fb4405
paper_authors.name: Maciej Besta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:48:29.913Z
paper_authors._id: 6613cdc25736e7f44f94df65
paper_authors.avatarUrl: /avatars/f5b398b4da03d7833e20ddb3ce4211be.svg
paper_authors.fullname: Maciej Besta
paper_authors.type: user
paper_authors.user: bestam
paper_authors._id: 6790772b8d7df822f1fb4406
paper_authors.name: Julia Barth
paper_authors._id: 6790772b8d7df822f1fb4407
paper_authors.name: Eric Schreiber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:48:45.137Z
paper_authors._id: 6712941f745634a65d916056
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/T5RZTWwb4FLZESYB5tu6m.png
paper_authors.fullname: Eric Schreiber
paper_authors.type: user
paper_authors.user: eschreibe1
paper_authors._id: 6790772b8d7df822f1fb4408
paper_authors.name: Ales Kubicek
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:48:52.532Z
paper_authors._id: 64baf20d12d00c4589bb12f7
paper_authors.avatarUrl: /avatars/2dbc10d369788c0ea048e1be97f0c5e6.svg
paper_authors.fullname: Ales Kubicek
paper_authors.type: user
paper_authors.user: aleskubicek
paper_authors._id: 6790772b8d7df822f1fb4409
paper_authors.name: Afonso Catarino
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:49:00.804Z
paper_authors._id: 66fc32aa787008467cfe20cb
paper_authors.avatarUrl: /avatars/b1c77ce8c7deaacf546a264467078673.svg
paper_authors.fullname: Afonso Catarino
paper_authors.type: user
paper_authors.user: AfonsoC
paper_authors._id: 6790772b8d7df822f1fb440a
paper_authors.name: Robert Gerstenberger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:49:11.846Z
paper_authors._id: 65a91420c46ce42ef5da96af
paper_authors.avatarUrl: /avatars/a9fa3a7973d0030292b1e23172112a1e.svg
paper_authors.fullname: Robert Gerstenberger
paper_authors.type: user
paper_authors.user: rgersten
paper_authors._id: 6790772b8d7df822f1fb440b
paper_authors.name: Piotr Nyczyk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:49:18.866Z
paper_authors._id: 63f89b579f87cc3e645d96f9
paper_authors.avatarUrl: /avatars/116be80348aa8f8be95b6ea774ecb65d.svg
paper_authors.fullname: Piotr Nyczyk
paper_authors.type: user
paper_authors.user: pnyczyk
paper_authors._id: 6790772b8d7df822f1fb440c
paper_authors.name: Patrick Iff
paper_authors._id: 6790772b8d7df822f1fb440d
paper_authors.name: Yueling Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:49:35.973Z
paper_authors._id: 641c07c4bbdbe642a79914df
paper_authors.avatarUrl: /avatars/5c89ee5f5cdf93e1d5ead6e46ae6c774.svg
paper_authors.fullname: Yueling Li
paper_authors.type: user
paper_authors.user: liy140
paper_authors._id: 6790772b8d7df822f1fb440e
paper_authors.name: Sam Houliston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:49:43.387Z
paper_authors._id: 66af74ddd59c09785e02d1e0
paper_authors.avatarUrl: /avatars/df0fec7228c35052b88cb5905bea809e.svg
paper_authors.fullname: Sam Houliston
paper_authors.type: user
paper_authors.user: samhouliston
paper_authors._id: 6790772b8d7df822f1fb440f
paper_authors.name: Tomasz Sternal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:49:51.528Z
paper_authors._id: 6535b041b66f4bf689267d91
paper_authors.avatarUrl: /avatars/6ef4a183ff08ec3f3595f0866f3129ac.svg
paper_authors.fullname: Tomasz Sternal
paper_authors.type: user
paper_authors.user: tsternal
paper_authors._id: 6790772b8d7df822f1fb4410
paper_authors.name: Marcin Copik
paper_authors._id: 6790772b8d7df822f1fb4411
paper_authors.name: Grzegorz Kwaśniewski
paper_authors._id: 6790772b8d7df822f1fb4412
paper_authors.name: Jürgen Müller
paper_authors._id: 6790772b8d7df822f1fb4413
paper_authors.name: Łukasz Flis
paper_authors._id: 6790772b8d7df822f1fb4414
paper_authors.name: Hannes Eberhard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:50:47.492Z
paper_authors._id: 62f80cbc04de855c35e32fdb
paper_authors.avatarUrl: /avatars/ab2600b96fe8b1787ad1eddaa45ad9ae.svg
paper_authors.fullname: Hannes Eberhard
paper_authors.type: user
paper_authors.user: HannesE
paper_authors._id: 6790772b8d7df822f1fb4415
paper_authors.name: Hubert Niewiadomski
paper_authors._id: 6790772b8d7df822f1fb4416
paper_authors.name: Torsten Hoefler
paper_title: Reasoning Language Models: A Blueprint
paper_summary: Reasoning language models (RLMs), also known as Large Reasoning Models
  (LRMs), such as OpenAI's o1 and o3, DeepSeek-V3, and Alibaba's QwQ, have
  redefined AI's problem-solving capabilities by extending large language models
  (LLMs) with advanced reasoning mechanisms. Yet, their high costs, proprietary
  nature, and complex architectures - uniquely combining Reinforcement Learning
  (RL), search heuristics, and LLMs - present accessibility and scalability
  challenges. To address these, we propose a comprehensive blueprint that
  organizes RLM components into a modular framework, based on a survey and
  analysis of all RLM works. This blueprint incorporates diverse reasoning
  structures (chains, trees, graphs, and nested forms), reasoning strategies
  (e.g., Monte Carlo Tree Search, Beam Search), RL concepts (policy, value models
  and others), and supervision schemes (Output-Based and Process-Based
  Supervision). We also provide detailed mathematical formulations and
  algorithmic specifications to simplify RLM implementation. By showing how
  schemes like LLaMA-Berry, QwQ, Journey Learning, and Graph of Thoughts fit as
  special cases, we demonstrate the blueprint's versatility and unifying
  potential. To illustrate its utility, we introduce x1, a modular implementation
  for rapid RLM prototyping and experimentation. Using x1 and a literature
  review, we provide key insights, such as multi-phase training for policy and
  value models, and the importance of familiar training distributions. Finally,
  we outline how RLMs can integrate with a broader LLM ecosystem, including tools
  and databases. Our work demystifies RLM construction, democratizes advanced
  reasoning capabilities, and fosters innovation, aiming to mitigate the gap
  between "rich AI" and "poor AI" by lowering barriers to RLM development and
  experimentation.
paper_ai_keywords: Reasoning language models (RLMs) Large Reasoning Models (LRMs) OpenAI's o1 OpenAI's o3 DeepSeek-V3 Alibaba's QwQ Reinforcement Learning (RL) search heuristics large language models (LLMs) reasoning structures (chains, trees, graphs, nested forms) reasoning strategies (e.g., Monte Carlo Tree Search, Beam Search) policy value models Output-Based Supervision Process-Based Supervision LLaMA-Berry Journey Learning Graph of Thoughts multi-phase training familiar training distributions LLM ecosystem