title: Simple and Effective Masked Diffusion Language Models
summary: While diffusion models excel at generating high-quality images, prior work
  reports a significant performance gap between diffusion and autoregressive (AR)
  methods in language modeling. In this work, we show that simple masked discrete
  diffusion is more performant than previously thought. We apply an effective
  training recipe that improves the performance of masked diffusion models and
  derive a simplified, Rao-Blackwellized objective that results in additional
  improvements. Our objective has a simple form -- it is a mixture of classical
  masked language modeling losses -- and can be used to train encoder-only
  language models that admit efficient samplers, including ones that can generate
  arbitrary lengths of text semi-autoregressively like a traditional language
  model. On language modeling benchmarks, a range of masked diffusion models
  trained with modern engineering practices achieves a new state-of-the-art among
  diffusion models, and approaches AR perplexity. We release our code at:
  https://github.com/kuleshov-group/mdlm
paper_authors._id: 66690f02aa74814619caee13
paper_authors.name: Subham Sekhar Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:36:17.388Z
paper_authors._id: 661839d73b412cdc851299c1
paper_authors.avatarUrl: /avatars/74c48e1ace77fd8c2138040bd2a18756.svg
paper_authors.fullname: Subham Sekhar Sahoo
paper_authors.type: user
paper_authors.user: subbham
paper_authors._id: 66690f02aa74814619caee14
paper_authors.name: Marianne Arriola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:36:23.171Z
paper_authors._id: 660d8a40e8379a6f240193de
paper_authors.avatarUrl: /avatars/64a99f756e7b5e15463c8265f3cc35c7.svg
paper_authors.fullname: Marianne Arriola
paper_authors.type: user
paper_authors.user: marriola
paper_authors._id: 66690f02aa74814619caee15
paper_authors.name: Yair Schiff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:36:28.591Z
paper_authors._id: 6418e5e722270b3ccf15dfef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418e5e722270b3ccf15dfef/ZiammTpL9FVMuE7DN8n5B.jpeg
paper_authors.fullname: Yair Schiff
paper_authors.type: user
paper_authors.user: yairschiff
paper_authors._id: 66690f02aa74814619caee16
paper_authors.name: Aaron Gokaslan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:29:31.761Z
paper_authors._id: 620573d0522e40b4a18d8763
paper_authors.avatarUrl: /avatars/9353c064ef8ccac84d0397411d38fa90.svg
paper_authors.fullname: Aaron Gokaslan
paper_authors.type: user
paper_authors.user: Skylion007
paper_authors._id: 66690f02aa74814619caee17
paper_authors.name: Edgar Marroquin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:36:41.104Z
paper_authors._id: 6616e84eceda8584fd0c109f
paper_authors.avatarUrl: /avatars/5261c6223571955260f7720ff88a8d67.svg
paper_authors.fullname: Edgar Marroquin
paper_authors.type: user
paper_authors.user: emarro
paper_authors._id: 66690f02aa74814619caee18
paper_authors.name: Justin T Chiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:36:46.781Z
paper_authors._id: 629f763f540c0da640ae7e70
paper_authors.avatarUrl: /avatars/1e8005eca079748365e7243a5d100fe9.svg
paper_authors.fullname: Justin T Chiu
paper_authors.type: user
paper_authors.user: justintchiu
paper_authors._id: 66690f02aa74814619caee19
paper_authors.name: Alexander Rush
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:36:58.268Z
paper_authors._id: 6372605b785dc489da5a037a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668440123408-noauth.png
paper_authors.fullname: Alex Rush
paper_authors.type: user
paper_authors.user: OfficialAL33900
paper_authors._id: 66690f02aa74814619caee1a
paper_authors.name: Volodymyr Kuleshov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:37:04.642Z
paper_authors._id: 640ea1c43c82bd463ee80e19
paper_authors.avatarUrl: /avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
paper_authors.fullname: Volodymyr Kuleshov
paper_authors.type: user
paper_authors.user: kuleshov
paper_title: Simple and Effective Masked Diffusion Language Models
paper_summary: While diffusion models excel at generating high-quality images, prior work
  reports a significant performance gap between diffusion and autoregressive (AR)
  methods in language modeling. In this work, we show that simple masked discrete
  diffusion is more performant than previously thought. We apply an effective
  training recipe that improves the performance of masked diffusion models and
  derive a simplified, Rao-Blackwellized objective that results in additional
  improvements. Our objective has a simple form -- it is a mixture of classical
  masked language modeling losses -- and can be used to train encoder-only
  language models that admit efficient samplers, including ones that can generate
  arbitrary lengths of text semi-autoregressively like a traditional language
  model. On language modeling benchmarks, a range of masked diffusion models
  trained with modern engineering practices achieves a new state-of-the-art among
  diffusion models, and approaches AR perplexity. We release our code at:
  https://github.com/kuleshov-group/mdlm
paper_ai_keywords: diffusion models language modeling masked discrete diffusion training recipe Rao-Blackwellized objective masked language modeling losses encoder-only language models efficient samplers semi-autoregressive state-of-the-art AR perplexity