title: Transfer Learning for Text Diffusion Models
summary: In this report, we explore the potential for text diffusion to replace
  autoregressive (AR) decoding for the training and deployment of large language
  models (LLMs). We are particularly interested to see whether pretrained AR
  models can be transformed into text diffusion models through a lightweight
  adaptation procedure we call ``AR2Diff''. We begin by establishing a strong
  baseline setup for training text diffusion models. Comparing across multiple
  architectures and pretraining objectives, we find that training a decoder-only
  model with a prefix LM objective is best or near-best across several tasks.
  Building on this finding, we test various transfer learning setups for text
  diffusion models. On machine translation, we find that text diffusion
  underperforms the standard AR approach. However, on code synthesis and
  extractive QA, we find diffusion models trained from scratch outperform AR
  models in many cases. We also observe quality gains from AR2Diff -- adapting AR
  models to use diffusion decoding. These results are promising given that text
  diffusion is relatively underexplored and can be significantly faster than AR
  decoding for long text generation.
paper_authors._id: 65b9b87e98bd7a573bbdeb81
paper_authors.name: Kehang Han
paper_authors._id: 65b9b87e98bd7a573bbdeb82
paper_authors.name: Kathleen Kenealy
paper_authors._id: 65b9b87e98bd7a573bbdeb83
paper_authors.name: Aditya Barua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:00:08.577Z
paper_authors._id: 63cb7741567105596c3f2f05
paper_authors.avatarUrl: /avatars/1c315a936e2ae04fc5b09d786cc899b7.svg
paper_authors.fullname: Aditya
paper_authors.type: user
paper_authors.user: adityabarua
paper_authors._id: 65b9b87e98bd7a573bbdeb84
paper_authors.name: Noah Fiedel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:00:15.834Z
paper_authors._id: 62f82ebd870a3f98bbf9e62a
paper_authors.avatarUrl: /avatars/3b03299e128abf2c8ba82db704855de7.svg
paper_authors.fullname: Noah Fiedel
paper_authors.type: user
paper_authors.user: nfiedel
paper_authors._id: 65b9b87e98bd7a573bbdeb85
paper_authors.name: Noah Constant
paper_title: Transfer Learning for Text Diffusion Models
paper_summary: In this report, we explore the potential for text diffusion to replace
  autoregressive (AR) decoding for the training and deployment of large language
  models (LLMs). We are particularly interested to see whether pretrained AR
  models can be transformed into text diffusion models through a lightweight
  adaptation procedure we call ``AR2Diff''. We begin by establishing a strong
  baseline setup for training text diffusion models. Comparing across multiple
  architectures and pretraining objectives, we find that training a decoder-only
  model with a prefix LM objective is best or near-best across several tasks.
  Building on this finding, we test various transfer learning setups for text
  diffusion models. On machine translation, we find that text diffusion
  underperforms the standard AR approach. However, on code synthesis and
  extractive QA, we find diffusion models trained from scratch outperform AR
  models in many cases. We also observe quality gains from AR2Diff -- adapting AR
  models to use diffusion decoding. These results are promising given that text
  diffusion is relatively underexplored and can be significantly faster than AR
  decoding for long text generation.
paper_ai_keywords: text diffusion autoregressive (AR) decoding large language models (LLMs) ``AR2Diff'' decoder-only model prefix LM objective transfer learning machine translation code synthesis extractive QA diffusion models long text generation