Multiverse: Your Language Models Secretly Decide How to Parallelize and
  Merge Generation
Autoregressive Large Language Models (AR-LLMs) frequently exhibit implicit
parallelism in sequential generation. Inspired by this, we introduce
Multiverse, a new generative model that enables natively parallel generation.
Multiverse internalizes a MapReduce paradigm, generating automatically through
three stages: (i) a Map stage for adaptive task decomposition, (ii) a Process
stage for parallel subtask execution, and (iii) a Reduce stage for lossless
result synthesis. Next, we build a real-world Multiverse reasoning model with
co-design of data, algorithm, and system, enabling rapid and seamless transfer
from frontier AR-LLMs. Starting from sequential reasoning chains, we create
Multiverse 1K by converting them into structured training data using an
automated LLM-assisted pipeline, avoiding costly human annotations.
Algorithmically, we design Multiverse Attention to separate parallel reasoning
steps while keeping compatibility with causal attention for efficient training.
Systematically, we implement Multiverse Engine to enable parallel inference. It
features a dedicated scheduler that dynamically switches between sequential and
parallel generation, triggered directly by the model. After a 3-hour
fine-tuning with 1K examples, our Multiverse-32B stands as the only
open-sourced non-AR model achieving performance on par with leading AR-LLMs of
the same scale, evidenced by AIME24 & 25 scores of 54% and 46%, respectively.
Moreover, our budget control experiments show that Multiverse-32B exhibits
superior scaling, outperforming AR-LLMs by 1.87% on average using the same
context length. Such scaling further leads to practical efficiency gain,
achieving up to 2x speedup across varying batch sizes. We have open-sourced the
entire Multiverse ecosystem, including data, model weights, engine, supporting
tools, as well as complete data curation prompts and detailed training and
evaluation recipes.
684adf25dbd21a9cc27b0ec8
Xinyu Yang
684adf25dbd21a9cc27b0ec9
Yuwei An
684adf25dbd21a9cc27b0eca
Hongyi Liu
684adf25dbd21a9cc27b0ecb
Tianqi Chen
684adf25dbd21a9cc27b0ecc
Beidi Chen
Multiverse: Your Language Models Secretly Decide How to Parallelize and
  Merge Generation
Autoregressive Large Language Models (AR-LLMs) frequently exhibit implicit
parallelism in sequential generation. Inspired by this, we introduce
Multiverse, a new generative model that enables natively parallel generation.
Multiverse internalizes a MapReduce paradigm, generating automatically through
three stages: (i) a Map stage for adaptive task decomposition, (ii) a Process
stage for parallel subtask execution, and (iii) a Reduce stage for lossless
result synthesis. Next, we build a real-world Multiverse reasoning model with
co-design of data, algorithm, and system, enabling rapid and seamless transfer
from frontier AR-LLMs. Starting from sequential reasoning chains, we create
Multiverse 1K by converting them into structured training data using an
automated LLM-assisted pipeline, avoiding costly human annotations.
Algorithmically, we design Multiverse Attention to separate parallel reasoning
steps while keeping compatibility with causal attention for efficient training.
Systematically, we implement Multiverse Engine to enable parallel inference. It
features a dedicated scheduler that dynamically switches between sequential and
parallel generation, triggered directly by the model. After a 3-hour
fine-tuning with 1K examples, our Multiverse-32B stands as the only
open-sourced non-AR model achieving performance on par with leading AR-LLMs of
the same scale, evidenced by AIME24 & 25 scores of 54% and 46%, respectively.
Moreover, our budget control experiments show that Multiverse-32B exhibits
superior scaling, outperforming AR-LLMs by 1.87% on average using the same
context length. Such scaling further leads to practical efficiency gain,
achieving up to 2x speedup across varying batch sizes. We have open-sourced the
entire Multiverse ecosystem, including data, model weights, engine, supporting
tools, as well as complete data curation prompts and detailed training and
evaluation recipes.
Multiverse, a generative model enabling native parallel text generation, achieves performance comparable to autoregressive models through a three-stage MapReduce paradigm and efficient attention mechanisms.
Autoregressive Large Language Models MapReduce paradigm parallel generation adaptive task decomposition parallel subtask execution lossless result synthesis Multiverse Attention causal attention parallel inference Multiverse Engine AIME24 AIME25 superior scaling efficiency gain speedup batch sizes