Simplified and Generalized Masked Diffusion for Discrete Data
Masked (or absorbing) diffusion is actively explored as an alternative to
autoregressive models for generative modeling of discrete data. However,
existing work in this area has been hindered by unnecessarily complex model
formulations and unclear relationships between different perspectives, leading
to suboptimal parameterization, training objectives, and ad hoc adjustments to
counteract these issues. In this work, we aim to provide a simple and general
framework that unlocks the full potential of masked diffusion models. We show
that the continuous-time variational objective of masked diffusion models is a
simple weighted integral of cross-entropy losses. Our framework also enables
training generalized masked diffusion models with state-dependent masking
schedules. When evaluated by perplexity, our models trained on OpenWebText
surpass prior diffusion language models at GPT-2 scale and demonstrate superior
performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our
models vastly outperform previous discrete diffusion models on pixel-level
image modeling, achieving 2.78~(CIFAR-10) and 3.42 (ImageNet 64times64) bits
per dimension that are comparable or better than autoregressive models of
similar sizes.
666a38939a3e3ce05a3ea38f
Jiaxin Shi
claimed_verified
2024-06-13T09:31:21.737Z
666a1ef22c3271b9b5ac6850
/avatars/18ba64e9d7453e1a6078a9022adee860.svg
Jiaxin Shi
user
thjashin
666a38939a3e3ce05a3ea390
Kehang Han
666a38939a3e3ce05a3ea391
Zhe Wang
admin_assigned
2024-06-13T18:41:35.853Z
637bc237be63f60f4ffe2e67
https://cdn-avatars.huggingface.co/v1/production/uploads/637bc237be63f60f4ffe2e67/SdvNBGA0s0bl6MtlDVh_5.jpeg
Zhe Wang
user
zhewang
666a38939a3e3ce05a3ea392
Arnaud Doucet
admin_assigned
2024-06-13T18:41:21.347Z
663f461a6fab5cd63c2b1f78
/avatars/bbd35a6fb48438dc7b143cf6f58479c1.svg
ArnaudD
user
ArnaudDoucet
666a38939a3e3ce05a3ea393
Michalis K. Titsias
Simplified and Generalized Masked Diffusion for Discrete Data
Masked (or absorbing) diffusion is actively explored as an alternative to
autoregressive models for generative modeling of discrete data. However,
existing work in this area has been hindered by unnecessarily complex model
formulations and unclear relationships between different perspectives, leading
to suboptimal parameterization, training objectives, and ad hoc adjustments to
counteract these issues. In this work, we aim to provide a simple and general
framework that unlocks the full potential of masked diffusion models. We show
that the continuous-time variational objective of masked diffusion models is a
simple weighted integral of cross-entropy losses. Our framework also enables
training generalized masked diffusion models with state-dependent masking
schedules. When evaluated by perplexity, our models trained on OpenWebText
surpass prior diffusion language models at GPT-2 scale and demonstrate superior
performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our
models vastly outperform previous discrete diffusion models on pixel-level
image modeling, achieving 2.78~(CIFAR-10) and 3.42 (ImageNet 64times64) bits
per dimension that are comparable or better than autoregressive models of
similar sizes.
A simple and general framework for masked diffusion models achieves superior performance in language and image modeling tasks compared to prior diffusion models and autoregressive models.
masked diffusion autoregressive models continuous-time variational objective cross-entropy losses state-dependent masking schedules zero-shot language modeling discrete diffusion models pixel-level image modeling perplexity CIFAR-10 ImageNet 64x64 bits per dimension