title: TESS: Text-to-Text Self-Conditioned Simplex Diffusion
summary: Diffusion models have emerged as a powerful paradigm for generation,
  obtaining strong performance in various domains with continuous-valued inputs.
  Despite the promises of fully non-autoregressive text generation, applying
  diffusion models to natural language remains challenging due to its discrete
  nature. In this work, we propose Text-to-text Self-conditioned Simplex
  Diffusion (TESS), a text diffusion model that is fully non-autoregressive,
  employs a new form of self-conditioning, and applies the diffusion process on
  the logit simplex space rather than the typical learned embedding space.
  Through extensive experiments on natural language understanding and generation
  tasks including summarization, text simplification, paraphrase generation, and
  question generation, we demonstrate that TESS outperforms state-of-the-art
  non-autoregressive models and is competitive with pretrained autoregressive
  sequence-to-sequence models.
paper_authors._id: 6462d5e9d9b4902188119f66
paper_authors.name: Rabeeh Karimi Mahabadi
paper_authors._id: 6462d5e9d9b4902188119f67
paper_authors.name: Jaesung Tae
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:32:27.927Z
paper_authors._id: 5fca69d8246881afd5b00163
paper_authors.avatarUrl: /avatars/028d72c5815d3cb1da3d95828bd5d5ea.svg
paper_authors.fullname: Jake Tae
paper_authors.type: user
paper_authors.user: jaketae
paper_authors._id: 6462d5e9d9b4902188119f68
paper_authors.name: Hamish Ivison
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:58:10.175Z
paper_authors._id: 62608fc2ffe8827cb1d89f9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
paper_authors.fullname: Hamish Ivison
paper_authors.type: user
paper_authors.user: hamishivi
paper_authors._id: 6462d5e9d9b4902188119f69
paper_authors.name: James Henderson
paper_authors._id: 6462d5e9d9b4902188119f6a
paper_authors.name: Iz Beltagy
paper_authors._id: 6462d5e9d9b4902188119f6b
paper_authors.name: Matthew E. Peters
paper_authors._id: 6462d5e9d9b4902188119f6c
paper_authors.name: Arman Cohan
paper_title: TESS: Text-to-Text Self-Conditioned Simplex Diffusion
paper_summary: Diffusion models have emerged as a powerful paradigm for generation,
  obtaining strong performance in various domains with continuous-valued inputs.
  Despite the promises of fully non-autoregressive text generation, applying
  diffusion models to natural language remains challenging due to its discrete
  nature. In this work, we propose Text-to-text Self-conditioned Simplex
  Diffusion (TESS), a text diffusion model that is fully non-autoregressive,
  employs a new form of self-conditioning, and applies the diffusion process on
  the logit simplex space rather than the typical learned embedding space.
  Through extensive experiments on natural language understanding and generation
  tasks including summarization, text simplification, paraphrase generation, and
  question generation, we demonstrate that TESS outperforms state-of-the-art
  non-autoregressive models and is competitive with pretrained autoregressive
  sequence-to-sequence models.
paper_ai_summary: Text-to-text Self-conditioned Simplex Diffusion (TESS) achieves strong performance on natural language tasks by using a fully non-autoregressive approach that applies diffusion in logit simplex space.
paper_ai_keywords: diffusion models non-autoregressive text generation self-conditioning logit simplex space summarization text simplification paraphrase generation question generation state-of-the-art non-autoregressive models pretrained autoregressive sequence-to-sequence models