title: AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
summary: Diffusion models have gained significant attention in the realm of image
  generation due to their exceptional performance. Their success has been
  recently expanded to text generation via generating all tokens within a
  sequence concurrently. However, natural language exhibits a far more pronounced
  sequential dependency in comparison to images, and the majority of existing
  language models are trained utilizing a left-to-right auto-regressive approach.
  To account for the inherent sequential characteristic of natural language, we
  introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
  the generation of tokens on the right depends on the generated ones on the
  left, a mechanism achieved through employing a dynamic number of denoising
  steps that vary based on token position. This results in tokens on the left
  undergoing fewer denoising steps than those on the right, thereby enabling them
  to generate earlier and subsequently influence the generation of tokens on the
  right. In a series of experiments on various text generation tasks including
  text summarization, machine translation, and common sense generation,
  AR-Diffusion clearly demonstrated the superiority over existing diffusion
  language models and that it can be 100timessim600times faster when
  achieving comparable results. Our code will be publicly released.
paper_authors._id: 6464221131063f253c22e295
paper_authors.name: Tong Wu
paper_authors._id: 6464221131063f253c22e296
paper_authors.name: Zhihao Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:32:56.762Z
paper_authors._id: 6049ae30f7b7068fd1ede43e
paper_authors.avatarUrl: /avatars/920365cd5e7c45f358d7c97e77cd5b9a.svg
paper_authors.fullname: Zhihao Fan
paper_authors.type: user
paper_authors.user: Libert
paper_authors._id: 6464221131063f253c22e297
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T13:56:04.999Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 6464221131063f253c22e298
paper_authors.name: Yeyun Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:30:42.337Z
paper_authors._id: 643f615aa16cd6d1f4c581de
paper_authors.avatarUrl: /avatars/47753a3e82b44f81881600c52e1e8495.svg
paper_authors.fullname: Yeyun Gong
paper_authors.type: user
paper_authors.user: yegong
paper_authors._id: 6464221131063f253c22e299
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:30:52.616Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_authors._id: 6464221131063f253c22e29a
paper_authors.name: Jian Jiao
paper_authors._id: 6464221131063f253c22e29b
paper_authors.name: Hai-Tao Zheng
paper_authors._id: 6464221131063f253c22e29c
paper_authors.name: Juntao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:33:26.317Z
paper_authors._id: 60d1d7f94672e8c13e7c2bce
paper_authors.avatarUrl: /avatars/be3a3990492ef1c5f3ff33cc043ae166.svg
paper_authors.fullname: Juntao Li
paper_authors.type: user
paper_authors.user: ljtsuda
paper_authors._id: 6464221131063f253c22e29d
paper_authors.name: Zhongyu Wei
paper_authors._id: 6464221131063f253c22e29e
paper_authors.name: Jian Guo
paper_authors._id: 6464221131063f253c22e29f
paper_authors.name: Nan Duan
paper_authors._id: 6464221131063f253c22e2a0
paper_authors.name: Weizhu Chen
paper_title: AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
paper_summary: Diffusion models have gained significant attention in the realm of image
  generation due to their exceptional performance. Their success has been
  recently expanded to text generation via generating all tokens within a
  sequence concurrently. However, natural language exhibits a far more pronounced
  sequential dependency in comparison to images, and the majority of existing
  language models are trained utilizing a left-to-right auto-regressive approach.
  To account for the inherent sequential characteristic of natural language, we
  introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
  the generation of tokens on the right depends on the generated ones on the
  left, a mechanism achieved through employing a dynamic number of denoising
  steps that vary based on token position. This results in tokens on the left
  undergoing fewer denoising steps than those on the right, thereby enabling them
  to generate earlier and subsequently influence the generation of tokens on the
  right. In a series of experiments on various text generation tasks including
  text summarization, machine translation, and common sense generation,
  AR-Diffusion clearly demonstrated the superiority over existing diffusion
  language models and that it can be 100timessim600times faster when
  achieving comparable results. Our code will be publicly released.
paper_ai_summary: Auto-Regressive Diffusion (AR-Diffusion) enhances text generation by dynamically varying denoising steps based on token position, improving efficiency and performance compared to existing diffusion models.
paper_ai_keywords: diffusion models auto-regressive denoising steps token position text summarization machine translation common sense generation