AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Diffusion models have gained significant attention in the realm of image
generation due to their exceptional performance. Their success has been
recently expanded to text generation via generating all tokens within a
sequence concurrently. However, natural language exhibits a far more pronounced
sequential dependency in comparison to images, and the majority of existing
language models are trained utilizing a left-to-right auto-regressive approach.
To account for the inherent sequential characteristic of natural language, we
introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
the generation of tokens on the right depends on the generated ones on the
left, a mechanism achieved through employing a dynamic number of denoising
steps that vary based on token position. This results in tokens on the left
undergoing fewer denoising steps than those on the right, thereby enabling them
to generate earlier and subsequently influence the generation of tokens on the
right. In a series of experiments on various text generation tasks including
text summarization, machine translation, and common sense generation,
AR-Diffusion clearly demonstrated the superiority over existing diffusion
language models and that it can be 100timessim600times faster when
achieving comparable results. Our code will be publicly released.
6464221131063f253c22e295
Tong Wu
6464221131063f253c22e296
Zhihao Fan
admin_assigned
2023-05-26T09:32:56.762Z
6049ae30f7b7068fd1ede43e
/avatars/920365cd5e7c45f358d7c97e77cd5b9a.svg
Zhihao Fan
user
Libert
6464221131063f253c22e297
Xiao Liu
claimed_verified
2023-05-17T13:56:04.999Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
Xiao Liu
user
lx865712528
6464221131063f253c22e298
Yeyun Gong
admin_assigned
2023-05-26T09:30:42.337Z
643f615aa16cd6d1f4c581de
/avatars/47753a3e82b44f81881600c52e1e8495.svg
Yeyun Gong
user
yegong
6464221131063f253c22e299
Yelong Shen
admin_assigned
2023-05-26T09:30:52.616Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
6464221131063f253c22e29a
Jian Jiao
6464221131063f253c22e29b
Hai-Tao Zheng
6464221131063f253c22e29c
Juntao Li
admin_assigned
2023-05-26T09:33:26.317Z
60d1d7f94672e8c13e7c2bce
/avatars/be3a3990492ef1c5f3ff33cc043ae166.svg
Juntao Li
user
ljtsuda
6464221131063f253c22e29d
Zhongyu Wei
6464221131063f253c22e29e
Jian Guo
6464221131063f253c22e29f
Nan Duan
6464221131063f253c22e2a0
Weizhu Chen
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Diffusion models have gained significant attention in the realm of image
generation due to their exceptional performance. Their success has been
recently expanded to text generation via generating all tokens within a
sequence concurrently. However, natural language exhibits a far more pronounced
sequential dependency in comparison to images, and the majority of existing
language models are trained utilizing a left-to-right auto-regressive approach.
To account for the inherent sequential characteristic of natural language, we
introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
the generation of tokens on the right depends on the generated ones on the
left, a mechanism achieved through employing a dynamic number of denoising
steps that vary based on token position. This results in tokens on the left
undergoing fewer denoising steps than those on the right, thereby enabling them
to generate earlier and subsequently influence the generation of tokens on the
right. In a series of experiments on various text generation tasks including
text summarization, machine translation, and common sense generation,
AR-Diffusion clearly demonstrated the superiority over existing diffusion
language models and that it can be 100timessim600times faster when
achieving comparable results. Our code will be publicly released.
Auto-Regressive Diffusion (AR-Diffusion) enhances text generation by dynamically varying denoising steps based on token position, improving efficiency and performance compared to existing diffusion models.
diffusion models auto-regressive denoising steps token position text summarization machine translation common sense generation