title: Continuous Diffusion Model for Language Modeling
summary: Diffusion models have emerged as a promising alternative to autoregressive
  models in modeling discrete categorical data. Yet diffusion models that
  directly work on discrete data space do not fully exploit the power of
  iterative refinement, as the signals are lost during the transition between
  discrete states. Existing continuous diffusion models for discrete data have
  limited performance compared to discrete approaches, and the unclear link
  between them restricts the development of diffusion models for discrete data.
  In this work, we propose a continuous diffusion model for language modeling
  that incorporates the geometry of the underlying categorical distribution. We
  establish a connection between the discrete diffusion and continuous flow on
  the statistical manifold, and building on the analogy, we introduce a simple
  design for the diffusion process that generalizes previous discrete diffusion
  models. We further propose a simulation-free training framework based on radial
  symmetry and a simple technique to address the high dimensionality of the
  manifold. Comprehensive experiments on language modeling benchmarks and other
  modalities show that our method outperforms existing discrete diffusion models
  and approaches the performance of autoregressive models. Codes available at
  https://github.com/harryjo97/RDLM{https://github.com/harryjo97/RDLM}.
paper_authors._id: 67b40f93aba9e111862052ab
paper_authors.name: Jaehyeong Jo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:31:27.544Z
paper_authors._id: 65e5bd4568234ef5d6decadc
paper_authors.avatarUrl: /avatars/c41095a946c0176b949c0b3566136c05.svg
paper_authors.fullname: Jaehyeong Jo
paper_authors.type: user
paper_authors.user: harryjo97
paper_authors._id: 67b40f93aba9e111862052ac
paper_authors.name: Sung Ju Hwang
paper_title: Continuous Diffusion Model for Language Modeling
paper_summary: Diffusion models have emerged as a promising alternative to autoregressive
  models in modeling discrete categorical data. Yet diffusion models that
  directly work on discrete data space do not fully exploit the power of
  iterative refinement, as the signals are lost during the transition between
  discrete states. Existing continuous diffusion models for discrete data have
  limited performance compared to discrete approaches, and the unclear link
  between them restricts the development of diffusion models for discrete data.
  In this work, we propose a continuous diffusion model for language modeling
  that incorporates the geometry of the underlying categorical distribution. We
  establish a connection between the discrete diffusion and continuous flow on
  the statistical manifold, and building on the analogy, we introduce a simple
  design for the diffusion process that generalizes previous discrete diffusion
  models. We further propose a simulation-free training framework based on radial
  symmetry and a simple technique to address the high dimensionality of the
  manifold. Comprehensive experiments on language modeling benchmarks and other
  modalities show that our method outperforms existing discrete diffusion models
  and approaches the performance of autoregressive models. Codes available at
  https://github.com/harryjo97/RDLM{https://github.com/harryjo97/RDLM}.
paper_ai_keywords: diffusion models autoregressive models discrete categorical data iterative refinement continuous diffusion models expectation statistical manifold continuous flow logical diffusion models multimodal representation symmetry language modeling benchmarks