Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive
  Token-Level Computation
Scaling language models unlocks impressive capabilities, but the accompanying
computational and memory demands make both training and deployment expensive.
Existing efficiency efforts typically target either parameter sharing or
adaptive computation, leaving open the question of how to attain both
simultaneously. We introduce Mixture-of-Recursions (MoR), a unified framework
that combines the two axes of efficiency inside a single Recursive Transformer.
MoR reuses a shared stack of layers across recursion steps to achieve parameter
efficiency, while lightweight routers enable adaptive token-level thinking by
dynamically assigning different recursion depths to individual tokens. This
allows MoR to focus quadratic attention computation only among tokens still
active at a given recursion depth, further improving memory access efficiency
by selectively caching only their key-value pairs. Beyond these core
mechanisms, we also propose a KV sharing variant that reuses KV pairs from the
first recursion, specifically designed to decrease prefill latency and memory
footprint. Across model scales ranging from 135M to 1.7B parameters, MoR forms
a new Pareto frontier: at equal training FLOPs and smaller model sizes, it
significantly lowers validation perplexity and improves few-shot accuracy,
while delivering higher throughput compared with vanilla and existing recursive
baselines. These gains demonstrate that MoR is an effective path towards
large-model quality without incurring large-model cost.
6875e531257d4f04353705d1
Sangmin Bae
6875e531257d4f04353705d2
Yujin Kim
claimed_verified
2025-07-15T19:09:27.342Z
65558caf38985840b368134f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XQSBrMezsvAcS78HzYiYs.png
Yujin Kim
user
kimyuji
6875e531257d4f04353705d3
Reza Bayat
6875e531257d4f04353705d4
Sungnyun Kim
claimed_verified
2025-07-24T14:11:20.972Z
63f0c2ac9cf89c9ed1bdd25c
/avatars/856b2cb482250fb83c6fe793e29dfd74.svg
Sungnyun Kim
user
sungnyun
6875e531257d4f04353705d5
Jiyoun Ha
6875e531257d4f04353705d6
Tal Schuster
6875e531257d4f04353705d7
Adam Fisch
6875e531257d4f04353705d8
Hrayr Harutyunyan
6875e531257d4f04353705d9
Ziwei Ji
6875e531257d4f04353705da
Aaron Courville
6875e531257d4f04353705db
Se-Young Yun
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive
  Token-Level Computation
Scaling language models unlocks impressive capabilities, but the accompanying
computational and memory demands make both training and deployment expensive.
Existing efficiency efforts typically target either parameter sharing or
adaptive computation, leaving open the question of how to attain both
simultaneously. We introduce Mixture-of-Recursions (MoR), a unified framework
that combines the two axes of efficiency inside a single Recursive Transformer.
MoR reuses a shared stack of layers across recursion steps to achieve parameter
efficiency, while lightweight routers enable adaptive token-level thinking by
dynamically assigning different recursion depths to individual tokens. This
allows MoR to focus quadratic attention computation only among tokens still
active at a given recursion depth, further improving memory access efficiency
by selectively caching only their key-value pairs. Beyond these core
mechanisms, we also propose a KV sharing variant that reuses KV pairs from the
first recursion, specifically designed to decrease prefill latency and memory
footprint. Across model scales ranging from 135M to 1.7B parameters, MoR forms
a new Pareto frontier: at equal training FLOPs and smaller model sizes, it
significantly lowers validation perplexity and improves few-shot accuracy,
while delivering higher throughput compared with vanilla and existing recursive
baselines. These gains demonstrate that MoR is an effective path towards
large-model quality without incurring large-model cost.
Mixture-of-Recursions (MoR) combines parameter sharing and adaptive computation in a Recursive Transformer to improve efficiency, reduce memory usage, and enhance performance across different model scales.
Mixture-of-Recursions MoR Recursive Transformer parameter efficiency adaptive computation lightweight routers quadratic attention computation key-value pairs KV sharing prefill latency memory footprint validation perplexity few-shot accuracy throughput