title: Large Language Diffusion Models
summary: Autoregressive models (ARMs) are widely regarded as the cornerstone of large
  language models (LLMs). We challenge this notion by introducing LLaDA, a
  diffusion model trained from scratch under the pre-training and supervised
  fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
  masking process and a reverse process, parameterized by a vanilla Transformer
  to predict masked tokens. By optimizing a likelihood bound, it provides a
  principled generative approach for probabilistic inference. Across extensive
  benchmarks, LLaDA demonstrates strong scalability, outperforming our
  self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
  LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
  instruction-following abilities in case studies such as multi-turn dialogue.
  Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
  poem completion task. Our findings establish diffusion models as a viable and
  promising alternative to ARMs, challenging the assumption that key LLM
  capabilities discussed above are inherently tied to ARMs.
paper_authors._id: 67b2c31125f77e5fc242f4f8
paper_authors.name: Shen Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:48:04.969Z
paper_authors._id: 640dd9a5fdeaae13908208a7
paper_authors.avatarUrl: /avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg
paper_authors.fullname: nieshen
paper_authors.type: user
paper_authors.user: nieshen
paper_authors._id: 67b2c31125f77e5fc242f4f9
paper_authors.name: Fengqi Zhu
paper_authors._id: 67b2c31125f77e5fc242f4fa
paper_authors.name: Zebin You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:47:21.498Z
paper_authors._id: 624f909eac5dd186b01ac3f5
paper_authors.avatarUrl: /avatars/0aafdb1cbb492fda52a0303031cc6c14.svg
paper_authors.fullname: Zebin You
paper_authors.type: user
paper_authors.user: yyyou
paper_authors._id: 67b2c31125f77e5fc242f4fb
paper_authors.name: Xiaolu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:47:27.956Z
paper_authors._id: 67513d6d3b8586521cda5d76
paper_authors.avatarUrl: /avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
paper_authors.fullname: Xiaolu  Zhang
paper_authors.type: user
paper_authors.user: xiaolu0714
paper_authors._id: 67b2c31125f77e5fc242f4fc
paper_authors.name: Jingyang Ou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:51:46.024Z
paper_authors._id: 656949b71d7c2ca7b7aae5f2
paper_authors.avatarUrl: /avatars/e7b23e260eb348cc26b849aaa601a503.svg
paper_authors.fullname: Jingyang Ou
paper_authors.type: user
paper_authors.user: JingyangOu
paper_authors._id: 67b2c31125f77e5fc242f4fd
paper_authors.name: Jun Hu
paper_authors._id: 67b2c31125f77e5fc242f4fe
paper_authors.name: Jun Zhou
paper_authors._id: 67b2c31125f77e5fc242f4ff
paper_authors.name: Yankai Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:51:39.766Z
paper_authors._id: 657a651e1433ea7d44de6397
paper_authors.avatarUrl: /avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
paper_authors.fullname: Yankai Lin
paper_authors.type: user
paper_authors.user: lyk423
paper_authors._id: 67b2c31125f77e5fc242f500
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:51:33.428Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_authors._id: 67b2c31125f77e5fc242f501
paper_authors.name: Chongxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:51:26.668Z
paper_authors._id: 64c07b488e2612254361153b
paper_authors.avatarUrl: /avatars/ade0f783cc4c2d3e73f402637f595471.svg
paper_authors.fullname: chongxuan li
paper_authors.type: user
paper_authors.user: zhenxuan00
paper_title: Large Language Diffusion Models
paper_summary: Autoregressive models (ARMs) are widely regarded as the cornerstone of large
  language models (LLMs). We challenge this notion by introducing LLaDA, a
  diffusion model trained from scratch under the pre-training and supervised
  fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
  masking process and a reverse process, parameterized by a vanilla Transformer
  to predict masked tokens. By optimizing a likelihood bound, it provides a
  principled generative approach for probabilistic inference. Across extensive
  benchmarks, LLaDA demonstrates strong scalability, outperforming our
  self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
  LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
  instruction-following abilities in case studies such as multi-turn dialogue.
  Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
  poem completion task. Our findings establish diffusion models as a viable and
  promising alternative to ARMs, challenging the assumption that key LLM
  capabilities discussed above are inherently tied to ARMs.
paper_ai_summary: LLaDA, a diffusion model trained from scratch, outperforms autoregressive models in benchmarks and demonstrates strong instruction-following capabilities, challenging the dominance of ARMs in LLMs.
paper_ai_keywords: autoregressive models LLaDA diffusion model pre-training supervised fine-tuning vanilla Transformer likelihood bound probabilistic inference in-context learning instruction-following reversal curse LLaMA3 GPT-4o reversal poem completion