title: Block Diffusion: Interpolating Between Autoregressive and Diffusion
    Language Models
summary: Diffusion language models offer unique benefits over autoregressive models
  due to their potential for parallelized generation and controllability, yet
  they lag in likelihood modeling and are limited to fixed-length generation. In
  this work, we introduce a class of block diffusion language models that
  interpolate between discrete denoising diffusion and autoregressive models.
  Block diffusion overcomes key limitations of both approaches by supporting
  flexible-length generation and improving inference efficiency with KV caching
  and parallel token sampling. We propose a recipe for building effective block
  diffusion models that includes an efficient training algorithm, estimators of
  gradient variance, and data-driven noise schedules to minimize the variance.
  Block diffusion sets a new state-of-the-art performance among diffusion models
  on language modeling benchmarks and enables generation of arbitrary-length
  sequences. We provide the code, along with the model weights and blog post on
  the project page: https://m-arriola.com/bd3lms/
paper_authors._id: 67d2511e7d0fc37e67269f85
paper_authors.name: Marianne Arriola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:16:23.456Z
paper_authors._id: 660d8a40e8379a6f240193de
paper_authors.avatarUrl: /avatars/64a99f756e7b5e15463c8265f3cc35c7.svg
paper_authors.fullname: Marianne Arriola
paper_authors.type: user
paper_authors.user: marriola
paper_authors._id: 67d2511e7d0fc37e67269f86
paper_authors.name: Aaron Gokaslan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T08:58:52.445Z
paper_authors._id: 620573d0522e40b4a18d8763
paper_authors.avatarUrl: /avatars/9353c064ef8ccac84d0397411d38fa90.svg
paper_authors.fullname: Aaron Gokaslan
paper_authors.type: user
paper_authors.user: Skylion007
paper_authors._id: 67d2511e7d0fc37e67269f87
paper_authors.name: Justin T Chiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:16:30.464Z
paper_authors._id: 629f763f540c0da640ae7e70
paper_authors.avatarUrl: /avatars/1e8005eca079748365e7243a5d100fe9.svg
paper_authors.fullname: Justin T Chiu
paper_authors.type: user
paper_authors.user: justintchiu
paper_authors._id: 67d2511e7d0fc37e67269f88
paper_authors.name: Zhihan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:16:37.189Z
paper_authors._id: 6723622b2f49e07c8a4bea78
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6723622b2f49e07c8a4bea78/w5BdcmfUrnmiykzths639.png
paper_authors.fullname: Zhihan Yang
paper_authors.type: user
paper_authors.user: zhihanyang
paper_authors._id: 67d2511e7d0fc37e67269f89
paper_authors.name: Zhixuan Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:16:43.642Z
paper_authors._id: 6600827b4d7ed9ef947597af
paper_authors.avatarUrl: /avatars/6d9c6b3ae91bfa8915c5e7bc516dcdb1.svg
paper_authors.fullname: Zhixuan Qi
paper_authors.type: user
paper_authors.user: Nauxihziq
paper_authors._id: 67d2511e7d0fc37e67269f8a
paper_authors.name: Jiaqi Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:16:51.044Z
paper_authors._id: 66ce1661d620749ad81dda84
paper_authors.avatarUrl: /avatars/e8a454c3d45b2e878388b1a6e466a5a1.svg
paper_authors.fullname: Jiaqi Han
paper_authors.type: user
paper_authors.user: jiaqihan99
paper_authors._id: 67d2511e7d0fc37e67269f8b
paper_authors.name: Subham Sekhar Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:16:56.803Z
paper_authors._id: 661839d73b412cdc851299c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661839d73b412cdc851299c1/xicwANPQPTFdWfblisL2-.png
paper_authors.fullname: Subham Sekhar Sahoo
paper_authors.type: user
paper_authors.user: s-sahoo
paper_authors._id: 67d2511e7d0fc37e67269f8c
paper_authors.name: Volodymyr Kuleshov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:17:03.227Z
paper_authors._id: 640ea1c43c82bd463ee80e19
paper_authors.avatarUrl: /avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
paper_authors.fullname: Volodymyr Kuleshov
paper_authors.type: user
paper_authors.user: kuleshov
paper_title: Block Diffusion: Interpolating Between Autoregressive and Diffusion
    Language Models
paper_summary: Diffusion language models offer unique benefits over autoregressive models
  due to their potential for parallelized generation and controllability, yet
  they lag in likelihood modeling and are limited to fixed-length generation. In
  this work, we introduce a class of block diffusion language models that
  interpolate between discrete denoising diffusion and autoregressive models.
  Block diffusion overcomes key limitations of both approaches by supporting
  flexible-length generation and improving inference efficiency with KV caching
  and parallel token sampling. We propose a recipe for building effective block
  diffusion models that includes an efficient training algorithm, estimators of
  gradient variance, and data-driven noise schedules to minimize the variance.
  Block diffusion sets a new state-of-the-art performance among diffusion models
  on language modeling benchmarks and enables generation of arbitrary-length
  sequences. We provide the code, along with the model weights and blog post on
  the project page: https://m-arriola.com/bd3lms/
paper_ai_summary: Block diffusion language models improve generation efficiency and sequence length flexibility compared to autoregressive and discrete denoising diffusion models.
paper_ai_keywords: diffusion language models autoregressive models denoising diffusion block diffusion parallelized generation controllability likelihood modeling fixed-length generation flexible-length generation inference efficiency KV caching parallel token sampling efficient training algorithm gradient variance estimators data-driven noise schedules language modeling benchmarks arbitrary-length sequences