Esoteric Language Models
Diffusion-based language models offer a compelling alternative to
autoregressive (AR) models by enabling parallel and controllable generation.
Among this family of models, Masked Diffusion Models (MDMs) achieve the
strongest performance but still underperform AR models in perplexity and lack
key inference-time efficiency features--most notably, KV caching. In this work,
we introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms,
enabling smooth interpolation between their perplexities while overcoming their
respective limitations. Eso-LMs set a new state of the art on standard language
modeling benchmarks. Crucially, we are the **first to introduce KV caching for
MDMs** while preserving parallel generation, significantly improving inference
efficiency. Combined with an optimized sampling schedule, our method achieves
up to **65x** faster inference than standard MDMs and **4x** faster inference
than prior semi-autoregressive approaches. We provide the code and model
checkpoints on the project page:
[http://s-sahoo.github.io/Eso-LMs](http://s-sahoo.github.io/Eso-LMs)
683f0717c885f4668d274e08
Subham Sekhar Sahoo
683f0717c885f4668d274e09
Zhihan Yang
claimed_verified
2025-06-20T12:20:19.479Z
6723622b2f49e07c8a4bea78
https://cdn-avatars.huggingface.co/v1/production/uploads/6723622b2f49e07c8a4bea78/w5BdcmfUrnmiykzths639.png
Zhihan Yang
user
zhihanyang
683f0717c885f4668d274e0a
Yash Akhauri
683f0717c885f4668d274e0b
Johnna Liu
683f0717c885f4668d274e0c
Deepansha Singh
683f0717c885f4668d274e0d
Zhoujun Cheng
683f0717c885f4668d274e0e
Zhengzhong Liu
683f0717c885f4668d274e0f
Eric Xing
683f0717c885f4668d274e10
John Thickstun
683f0717c885f4668d274e11
Arash Vahdat
Esoteric Language Models
Diffusion-based language models offer a compelling alternative to
autoregressive (AR) models by enabling parallel and controllable generation.
Among this family of models, Masked Diffusion Models (MDMs) achieve the
strongest performance but still underperform AR models in perplexity and lack
key inference-time efficiency features--most notably, KV caching. In this work,
we introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms,
enabling smooth interpolation between their perplexities while overcoming their
respective limitations. Eso-LMs set a new state of the art on standard language
modeling benchmarks. Crucially, we are the **first to introduce KV caching for
MDMs** while preserving parallel generation, significantly improving inference
efficiency. Combined with an optimized sampling schedule, our method achieves
up to **65x** faster inference than standard MDMs and **4x** faster inference
than prior semi-autoregressive approaches. We provide the code and model
checkpoints on the project page:
[http://s-sahoo.github.io/Eso-LMs](http://s-sahoo.github.io/Eso-LMs)
Eso-LMs, a novel fusion of autoregressive and masked diffusion models, introduce KV caching to MDMs, achieving faster inference and superior performance on language modeling benchmarks.
diffusion-based language models autoregressive models Masked Diffusion Models (MDMs) Eso-LMs KV caching parallel generation inference-time efficiency sampling schedule