title: Scaling Diffusion Language Models via Adaptation from Autoregressive
    Models
summary: Diffusion Language Models (DLMs) have emerged as a promising new paradigm for
  text generative modeling, potentially addressing limitations of autoregressive
  (AR) models. However, current DLMs have been studied at a smaller scale
  compared to their AR counterparts and lack fair comparison on language modeling
  benchmarks. Additionally, training diffusion models from scratch at scale
  remains challenging. Given the prevalence of open-source AR language models, we
  propose adapting these models to build text diffusion models. We demonstrate
  connections between AR and diffusion modeling objectives and introduce a simple
  continual pre-training approach for training diffusion models. Through
  systematic evaluation on language modeling, reasoning, and commonsense
  benchmarks, we show that we can convert AR models ranging from 127M to 7B
  parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA,
  using less than 200B tokens for training. Our experimental results reveal that
  these models outperform earlier DLMs and are competitive with their AR
  counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters)
  capable of generating fluent text, performing in-context learning, filling in
  the middle without prompt re-ordering, and following instructions
  https://github.com/HKUNLP/DiffuLLaMA.
paper_authors._id: 6719a80204e2e351e5660f16
paper_authors.name: Shansan Gong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:33.126Z
paper_authors._id: 628c83d186fc004b14e1ed48
paper_authors.avatarUrl: /avatars/05ff943a9b89b5f67c5bc254bf45b8f5.svg
paper_authors.fullname: Shansan Gong
paper_authors.type: user
paper_authors.user: Sansa
paper_authors._id: 6719a80204e2e351e5660f17
paper_authors.name: Shivam Agarwal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-24T01:50:59.734Z
paper_authors._id: 64b83a53cbb0af9bfb106384
paper_authors.avatarUrl: /avatars/8fd05154d6c2c17e16f2a5b4272846fc.svg
paper_authors.fullname: Shivam Agarwal
paper_authors.type: user
paper_authors.user: shivamag99
paper_authors._id: 6719a80204e2e351e5660f18
paper_authors.name: Yizhe Zhang
paper_authors._id: 6719a80204e2e351e5660f19
paper_authors.name: Jiacheng Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:22:05.255Z
paper_authors._id: 6364cc3912188d67e6526cfa
paper_authors.avatarUrl: /avatars/bb5c3200c6d64d1260b7e418cf1a76f9.svg
paper_authors.fullname: Carson Ye
paper_authors.type: user
paper_authors.user: jiacheng-ye
paper_authors._id: 6719a80204e2e351e5660f1a
paper_authors.name: Lin Zheng
paper_authors._id: 6719a80204e2e351e5660f1b
paper_authors.name: Mukai Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:31.268Z
paper_authors._id: 624561c1939c9acec2103534
paper_authors.avatarUrl: /avatars/3efdf797c53d153cea7415213fb5afc7.svg
paper_authors.fullname: Mukai Li
paper_authors.type: user
paper_authors.user: kiaia
paper_authors._id: 6719a80204e2e351e5660f1c
paper_authors.name: Chenxin An
paper_authors._id: 6719a80204e2e351e5660f1d
paper_authors.name: Peilin Zhao
paper_authors._id: 6719a80204e2e351e5660f1e
paper_authors.name: Wei Bi
paper_authors._id: 6719a80204e2e351e5660f1f
paper_authors.name: Jiawei Han
paper_authors._id: 6719a80204e2e351e5660f20
paper_authors.name: Hao Peng
paper_authors._id: 6719a80204e2e351e5660f21
paper_authors.name: Lingpeng Kong
paper_title: Scaling Diffusion Language Models via Adaptation from Autoregressive
    Models
paper_summary: Diffusion Language Models (DLMs) have emerged as a promising new paradigm for
  text generative modeling, potentially addressing limitations of autoregressive
  (AR) models. However, current DLMs have been studied at a smaller scale
  compared to their AR counterparts and lack fair comparison on language modeling
  benchmarks. Additionally, training diffusion models from scratch at scale
  remains challenging. Given the prevalence of open-source AR language models, we
  propose adapting these models to build text diffusion models. We demonstrate
  connections between AR and diffusion modeling objectives and introduce a simple
  continual pre-training approach for training diffusion models. Through
  systematic evaluation on language modeling, reasoning, and commonsense
  benchmarks, we show that we can convert AR models ranging from 127M to 7B
  parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA,
  using less than 200B tokens for training. Our experimental results reveal that
  these models outperform earlier DLMs and are competitive with their AR
  counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters)
  capable of generating fluent text, performing in-context learning, filling in
  the middle without prompt re-ordering, and following instructions
  https://github.com/HKUNLP/DiffuLLaMA.
paper_ai_keywords: Diffusion Language Models (DLMs) generative modeling autoregressive (AR) models diffusion models continual pre-training language modeling reasoning commonsense benchmarks pre-training DiffuGPT DiffuLLaMA in-context learning instruction-following