Continuous Diffusion Model for Language Modeling
Diffusion models have emerged as a promising alternative to autoregressive
models in modeling discrete categorical data. Yet diffusion models that
directly work on discrete data space do not fully exploit the power of
iterative refinement, as the signals are lost during the transition between
discrete states. Existing continuous diffusion models for discrete data have
limited performance compared to discrete approaches, and the unclear link
between them restricts the development of diffusion models for discrete data.
In this work, we propose a continuous diffusion model for language modeling
that incorporates the geometry of the underlying categorical distribution. We
establish a connection between the discrete diffusion and continuous flow on
the statistical manifold, and building on the analogy, we introduce a simple
design for the diffusion process that generalizes previous discrete diffusion
models. We further propose a simulation-free training framework based on radial
symmetry and a simple technique to address the high dimensionality of the
manifold. Comprehensive experiments on language modeling benchmarks and other
modalities show that our method outperforms existing discrete diffusion models
and approaches the performance of autoregressive models. Codes available at
https://github.com/harryjo97/RDLM{https://github.com/harryjo97/RDLM}.
67b40f93aba9e111862052ab
Jaehyeong Jo
claimed_verified
2025-02-18T09:31:27.544Z
65e5bd4568234ef5d6decadc
/avatars/c41095a946c0176b949c0b3566136c05.svg
Jaehyeong Jo
user
harryjo97
67b40f93aba9e111862052ac
Sung Ju Hwang
Continuous Diffusion Model for Language Modeling
Diffusion models have emerged as a promising alternative to autoregressive
models in modeling discrete categorical data. Yet diffusion models that
directly work on discrete data space do not fully exploit the power of
iterative refinement, as the signals are lost during the transition between
discrete states. Existing continuous diffusion models for discrete data have
limited performance compared to discrete approaches, and the unclear link
between them restricts the development of diffusion models for discrete data.
In this work, we propose a continuous diffusion model for language modeling
that incorporates the geometry of the underlying categorical distribution. We
establish a connection between the discrete diffusion and continuous flow on
the statistical manifold, and building on the analogy, we introduce a simple
design for the diffusion process that generalizes previous discrete diffusion
models. We further propose a simulation-free training framework based on radial
symmetry and a simple technique to address the high dimensionality of the
manifold. Comprehensive experiments on language modeling benchmarks and other
modalities show that our method outperforms existing discrete diffusion models
and approaches the performance of autoregressive models. Codes available at
https://github.com/harryjo97/RDLM{https://github.com/harryjo97/RDLM}.
A continuous diffusion model for language modeling that leverages the geometry of discrete distributions outperforms existing discrete models and matches autoregressive models in performance.
diffusion models autoregressive models discrete data space iterative refinement continuous diffusion models statistical manifold radial symmetry high dimensionality