Transfer Learning for Text Diffusion Models
In this report, we explore the potential for text diffusion to replace
autoregressive (AR) decoding for the training and deployment of large language
models (LLMs). We are particularly interested to see whether pretrained AR
models can be transformed into text diffusion models through a lightweight
adaptation procedure we call ``AR2Diff''. We begin by establishing a strong
baseline setup for training text diffusion models. Comparing across multiple
architectures and pretraining objectives, we find that training a decoder-only
model with a prefix LM objective is best or near-best across several tasks.
Building on this finding, we test various transfer learning setups for text
diffusion models. On machine translation, we find that text diffusion
underperforms the standard AR approach. However, on code synthesis and
extractive QA, we find diffusion models trained from scratch outperform AR
models in many cases. We also observe quality gains from AR2Diff -- adapting AR
models to use diffusion decoding. These results are promising given that text
diffusion is relatively underexplored and can be significantly faster than AR
decoding for long text generation.
65b9b87e98bd7a573bbdeb81
Kehang Han
65b9b87e98bd7a573bbdeb82
Kathleen Kenealy
65b9b87e98bd7a573bbdeb83
Aditya Barua
admin_assigned
2024-01-31T10:00:08.577Z
63cb7741567105596c3f2f05
/avatars/1c315a936e2ae04fc5b09d786cc899b7.svg
Aditya
user
adityabarua
65b9b87e98bd7a573bbdeb84
Noah Fiedel
admin_assigned
2024-01-31T10:00:15.834Z
62f82ebd870a3f98bbf9e62a
/avatars/3b03299e128abf2c8ba82db704855de7.svg
Noah Fiedel
user
nfiedel
65b9b87e98bd7a573bbdeb85
Noah Constant
Transfer Learning for Text Diffusion Models
In this report, we explore the potential for text diffusion to replace
autoregressive (AR) decoding for the training and deployment of large language
models (LLMs). We are particularly interested to see whether pretrained AR
models can be transformed into text diffusion models through a lightweight
adaptation procedure we call ``AR2Diff''. We begin by establishing a strong
baseline setup for training text diffusion models. Comparing across multiple
architectures and pretraining objectives, we find that training a decoder-only
model with a prefix LM objective is best or near-best across several tasks.
Building on this finding, we test various transfer learning setups for text
diffusion models. On machine translation, we find that text diffusion
underperforms the standard AR approach. However, on code synthesis and
extractive QA, we find diffusion models trained from scratch outperform AR
models in many cases. We also observe quality gains from AR2Diff -- adapting AR
models to use diffusion decoding. These results are promising given that text
diffusion is relatively underexplored and can be significantly faster than AR
decoding for long text generation.
Text diffusion models show promise in code synthesis and extractive QA, outperforming autoregressive models in some cases and offering faster long text generation through lightweight adaptation.
text diffusion autoregressive (AR) decoding large language models (LLMs) AR2Diff decoder-only model prefix LM objective machine translation code synthesis extractive QA