Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long
  Generation
Recent advances in language modeling have demonstrated the effectiveness of
State Space Models (SSMs) for efficient sequence modeling. While hybrid
architectures such as Samba and the decoder-decoder architecture, YOCO, have
shown promising performance gains over Transformers, prior works have not
investigated the efficiency potential of representation sharing between SSM
layers. In this paper, we introduce the Gated Memory Unit (GMU), a simple yet
effective mechanism for efficient memory sharing across layers. We apply it to
create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs in
the cross-decoder to share memory readout states from a Samba-based
self-decoder. SambaY significantly enhances decoding efficiency, preserves
linear pre-filling time complexity, and boosts long-context performance, all
while eliminating the need for explicit positional encoding. Through extensive
scaling experiments, we demonstrate that our model exhibits a significantly
lower irreducible loss compared to a strong YOCO baseline, indicating superior
performance scalability under large-scale compute regimes. Our largest model
enhanced with Differential Attention, Phi4-mini-Flash-Reasoning, achieves
significantly better performance than Phi4-mini-Reasoning on reasoning tasks
such as Math500, AIME24/25, and GPQA Diamond without any reinforcement
learning, while delivering up to 10x higher decoding throughput on 2K-length
prompts with 32K generation length under the vLLM inference framework. We
release our training codebase on open-source data at
https://github.com/microsoft/ArchScale.
686f629bd938c25d68441be3
Liliang Ren
claimed_verified
2025-07-31T08:16:14.227Z
63815eff4761ddfa00903762
https://cdn-avatars.huggingface.co/v1/production/uploads/63815eff4761ddfa00903762/YlkfX3sTZRQcoapctXJeJ.jpeg
Liliang Ren
user
renll
686f629bd938c25d68441be4
Congcong Chen
686f629bd938c25d68441be5
Haoran Xu
686f629bd938c25d68441be6
Young Jin Kim
claimed_verified
2025-07-11T08:03:31.160Z
63f5173bb51da4d61da6c038
/avatars/0ee530cf80476aa3985c4d591cd384a1.svg
Young Jin Kim
user
ykim362
686f629bd938c25d68441be7
Adam Atkinson
686f629bd938c25d68441be8
Zheng Zhan
686f629bd938c25d68441be9
Jiankai Sun
686f629bd938c25d68441bea
Baolin Peng
686f629bd938c25d68441beb
Liyuan Liu
686f629bd938c25d68441bec
Shuohang Wang
686f629bd938c25d68441bed
Hao Cheng
686f629bd938c25d68441bee
Jianfeng Gao
686f629bd938c25d68441bef
Weizhu Chen
686f629bd938c25d68441bf0
Yelong Shen
Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long
  Generation
Recent advances in language modeling have demonstrated the effectiveness of
State Space Models (SSMs) for efficient sequence modeling. While hybrid
architectures such as Samba and the decoder-decoder architecture, YOCO, have
shown promising performance gains over Transformers, prior works have not
investigated the efficiency potential of representation sharing between SSM
layers. In this paper, we introduce the Gated Memory Unit (GMU), a simple yet
effective mechanism for efficient memory sharing across layers. We apply it to
create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs in
the cross-decoder to share memory readout states from a Samba-based
self-decoder. SambaY significantly enhances decoding efficiency, preserves
linear pre-filling time complexity, and boosts long-context performance, all
while eliminating the need for explicit positional encoding. Through extensive
scaling experiments, we demonstrate that our model exhibits a significantly
lower irreducible loss compared to a strong YOCO baseline, indicating superior
performance scalability under large-scale compute regimes. Our largest model
enhanced with Differential Attention, Phi4-mini-Flash-Reasoning, achieves
significantly better performance than Phi4-mini-Reasoning on reasoning tasks
such as Math500, AIME24/25, and GPQA Diamond without any reinforcement
learning, while delivering up to 10x higher decoding throughput on 2K-length
prompts with 32K generation length under the vLLM inference framework. We
release our training codebase on open-source data at
https://github.com/microsoft/ArchScale.
Gated Memory Unit (GMU) enhances SambaY architecture, improving decoding efficiency and long-context performance while reducing irreducible loss and increasing throughput.
State Space Models (SSMs) Samba YOCO Gated Memory Unit (GMU) decoder-hybrid-decoder memory sharing linear pre-filling time complexity Differential Attention Phi4-mini-Flash-Reasoning Phi4-mini-Reasoning Math500 AIME24/25 GPQA Diamond vLLM inference framework