title: Esoteric Language Models
summary: Diffusion-based language models offer a compelling alternative to
  autoregressive (AR) models by enabling parallel and controllable generation.
  Among this family of models, Masked Diffusion Models (MDMs) achieve the
  strongest performance but still underperform AR models in perplexity and lack
  key inference-time efficiency features--most notably, KV caching. In this work,
  we introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms,
  enabling smooth interpolation between their perplexities while overcoming their
  respective limitations. Eso-LMs set a new state of the art on standard language
  modeling benchmarks. Crucially, we are the **first to introduce KV caching for
  MDMs** while preserving parallel generation, significantly improving inference
  efficiency. Combined with an optimized sampling schedule, our method achieves
  up to **65x** faster inference than standard MDMs and **4x** faster inference
  than prior semi-autoregressive approaches. We provide the code and model
  checkpoints on the project page:
  [http://s-sahoo.github.io/Eso-LMs](http://s-sahoo.github.io/Eso-LMs)
paper_authors._id: 683f0717c885f4668d274e08
paper_authors.name: Subham Sekhar Sahoo
paper_authors._id: 683f0717c885f4668d274e09
paper_authors.name: Zhihan Yang
paper_authors._id: 683f0717c885f4668d274e0a
paper_authors.name: Yash Akhauri
paper_authors._id: 683f0717c885f4668d274e0b
paper_authors.name: Johnna Liu
paper_authors._id: 683f0717c885f4668d274e0c
paper_authors.name: Deepansha Singh
paper_authors._id: 683f0717c885f4668d274e0d
paper_authors.name: Zhoujun Cheng
paper_authors._id: 683f0717c885f4668d274e0e
paper_authors.name: Zhengzhong Liu
paper_authors._id: 683f0717c885f4668d274e0f
paper_authors.name: Eric Xing
paper_authors._id: 683f0717c885f4668d274e10
paper_authors.name: John Thickstun
paper_authors._id: 683f0717c885f4668d274e11
paper_authors.name: Arash Vahdat
paper_title: Esoteric Language Models
paper_summary: Diffusion-based language models offer a compelling alternative to
  autoregressive (AR) models by enabling parallel and controllable generation.
  Among this family of models, Masked Diffusion Models (MDMs) achieve the
  strongest performance but still underperform AR models in perplexity and lack
  key inference-time efficiency features--most notably, KV caching. In this work,
  we introduce Eso-LMs, a new family of models that fuses AR and MDM paradigms,
  enabling smooth interpolation between their perplexities while overcoming their
  respective limitations. Eso-LMs set a new state of the art on standard language
  modeling benchmarks. Crucially, we are the **first to introduce KV caching for
  MDMs** while preserving parallel generation, significantly improving inference
  efficiency. Combined with an optimized sampling schedule, our method achieves
  up to **65x** faster inference than standard MDMs and **4x** faster inference
  than prior semi-autoregressive approaches. We provide the code and model
  checkpoints on the project page:
  [http://s-sahoo.github.io/Eso-LMs](http://s-sahoo.github.io/Eso-LMs)
paper_ai_summary: Eso-LMs, a novel fusion of autoregressive and masked diffusion models, introduce KV caching to MDMs, achieving faster inference and superior performance on language modeling benchmarks.
paper_ai_keywords: diffusion-based language models autoregressive models Masked Diffusion Models (MDMs) Eso-LMs KV caching parallel generation inference-time efficiency sampling schedule