title: Simplified and Generalized Masked Diffusion for Discrete Data
summary: Masked (or absorbing) diffusion is actively explored as an alternative to
  autoregressive models for generative modeling of discrete data. However,
  existing work in this area has been hindered by unnecessarily complex model
  formulations and unclear relationships between different perspectives, leading
  to suboptimal parameterization, training objectives, and ad hoc adjustments to
  counteract these issues. In this work, we aim to provide a simple and general
  framework that unlocks the full potential of masked diffusion models. We show
  that the continuous-time variational objective of masked diffusion models is a
  simple weighted integral of cross-entropy losses. Our framework also enables
  training generalized masked diffusion models with state-dependent masking
  schedules. When evaluated by perplexity, our models trained on OpenWebText
  surpass prior diffusion language models at GPT-2 scale and demonstrate superior
  performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our
  models vastly outperform previous discrete diffusion models on pixel-level
  image modeling, achieving 2.78~(CIFAR-10) and 3.42 (ImageNet 64times64) bits
  per dimension that are comparable or better than autoregressive models of
  similar sizes.
paper_authors._id: 666a38939a3e3ce05a3ea38f
paper_authors.name: Jiaxin Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:21.737Z
paper_authors._id: 666a1ef22c3271b9b5ac6850
paper_authors.avatarUrl: /avatars/18ba64e9d7453e1a6078a9022adee860.svg
paper_authors.fullname: Jiaxin Shi
paper_authors.type: user
paper_authors.user: thjashin
paper_authors._id: 666a38939a3e3ce05a3ea390
paper_authors.name: Kehang Han
paper_authors._id: 666a38939a3e3ce05a3ea391
paper_authors.name: Zhe Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T18:41:35.853Z
paper_authors._id: 637bc237be63f60f4ffe2e67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637bc237be63f60f4ffe2e67/xitcPHGDiUaA-NDakitGt.png
paper_authors.fullname: Zhe Wang
paper_authors.type: user
paper_authors.user: zhewang
paper_authors._id: 666a38939a3e3ce05a3ea392
paper_authors.name: Arnaud Doucet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T18:41:21.347Z
paper_authors._id: 663f461a6fab5cd63c2b1f78
paper_authors.avatarUrl: /avatars/bbd35a6fb48438dc7b143cf6f58479c1.svg
paper_authors.fullname: ArnaudD
paper_authors.type: user
paper_authors.user: ArnaudDoucet
paper_authors._id: 666a38939a3e3ce05a3ea393
paper_authors.name: Michalis K. Titsias
paper_title: Simplified and Generalized Masked Diffusion for Discrete Data
paper_summary: Masked (or absorbing) diffusion is actively explored as an alternative to
  autoregressive models for generative modeling of discrete data. However,
  existing work in this area has been hindered by unnecessarily complex model
  formulations and unclear relationships between different perspectives, leading
  to suboptimal parameterization, training objectives, and ad hoc adjustments to
  counteract these issues. In this work, we aim to provide a simple and general
  framework that unlocks the full potential of masked diffusion models. We show
  that the continuous-time variational objective of masked diffusion models is a
  simple weighted integral of cross-entropy losses. Our framework also enables
  training generalized masked diffusion models with state-dependent masking
  schedules. When evaluated by perplexity, our models trained on OpenWebText
  surpass prior diffusion language models at GPT-2 scale and demonstrate superior
  performance on 4 out of 5 zero-shot language modeling tasks. Furthermore, our
  models vastly outperform previous discrete diffusion models on pixel-level
  image modeling, achieving 2.78~(CIFAR-10) and 3.42 (ImageNet 64times64) bits
  per dimension that are comparable or better than autoregressive models of
  similar sizes.
paper_ai_keywords: masked diffusion autoregressive models generative modeling discrete data continuous-time variational objective cross-entropy losses state-dependent masking schedules perplexity OpenWebText zero-shot language modeling discrete diffusion models pixel-level image modeling CIFAR-10 ImageNet 64×64 bits per dimension transgressive models