title: UT5: Pretraining Non autoregressive T5 with unrolled denoising
summary: Recent advances in Transformer-based Large Language Models have made great
  strides in natural language generation. However, to decode K tokens, an
  autoregressive model needs K sequential forward passes, which may be a
  performance bottleneck for large language models. Many non-autoregressive (NAR)
  research are aiming to address this sequentiality bottleneck, albeit many have
  focused on a dedicated architecture in supervised benchmarks. In this work, we
  studied unsupervised pretraining for non auto-regressive T5 models via unrolled
  denoising and shown its SoTA results in downstream generation tasks such as
  SQuAD question generation and XSum.
paper_authors._id: 6555a8a5c9d29746224c22b5
paper_authors.name: Mahmoud G. Salem
paper_authors._id: 6555a8a5c9d29746224c22b6
paper_authors.name: Jiayu Ye
paper_authors._id: 6555a8a5c9d29746224c22b7
paper_authors.name: Chu-Cheng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-16T10:26:08.528Z
paper_authors._id: 6137b1b20317b0a5c10877c6
paper_authors.avatarUrl: /avatars/8eccd696bffab67e431aed200e96af14.svg
paper_authors.fullname: Chu-Cheng Lin
paper_authors.type: user
paper_authors.user: kitsing
paper_authors._id: 6555a8a5c9d29746224c22b8
paper_authors.name: Frederick Liu
paper_title: UT5: Pretraining Non autoregressive T5 with unrolled denoising
paper_summary: Recent advances in Transformer-based Large Language Models have made great
  strides in natural language generation. However, to decode K tokens, an
  autoregressive model needs K sequential forward passes, which may be a
  performance bottleneck for large language models. Many non-autoregressive (NAR)
  research are aiming to address this sequentiality bottleneck, albeit many have
  focused on a dedicated architecture in supervised benchmarks. In this work, we
  studied unsupervised pretraining for non auto-regressive T5 models via unrolled
  denoising and shown its SoTA results in downstream generation tasks such as
  SQuAD question generation and XSum.
paper_ai_summary: Unsupervised pretraining of non-autoregressive T5 models using unrolled denoising achieves state-of-the-art results in downstream generation tasks.
paper_ai_keywords: Transformer-based Large Language Models autoregressive models non-autoregressive (NAR) models T5 models unrolled denoising SoTA results SQuAD question generation XSum