title: TESS 2: A Large-Scale Generalist Diffusion Language Model
summary: We introduce TESS 2, a general instruction-following diffusion language model
  that outperforms contemporary instruction-tuned diffusion models, as well as
  matches and sometimes exceeds strong autoregressive (AR) models. We train TESS
  2 by first adapting a strong AR model via continued pretraining with the usual
  cross-entropy as diffusion loss, and then performing further instruction
  tuning. We find that adaptation training as well as the choice of the base
  model is crucial for training good instruction-following diffusion models. We
  further propose reward guidance, a novel and modular inference-time guidance
  procedure to align model outputs without needing to train the underlying model.
  Finally, we show that TESS 2 further improves with increased inference-time
  compute, highlighting the utility of diffusion LMs in having fine-grained
  controllability over the amount of compute used at inference time. Code and
  models are available at https://github.com/hamishivi/tess-2.
paper_authors._id: 67b698422c8b2ef925e03f4f
paper_authors.name: Jaesung Tae
paper_authors._id: 67b698422c8b2ef925e03f50
paper_authors.name: Hamish Ivison
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-04T04:34:53.882Z
paper_authors._id: 62608fc2ffe8827cb1d89f9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
paper_authors.fullname: Hamish Ivison
paper_authors.type: user
paper_authors.user: hamishivi
paper_authors._id: 67b698422c8b2ef925e03f51
paper_authors.name: Sachin Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:12:28.834Z
paper_authors._id: 63f24d2d7ddf724fbcc0ea9c
paper_authors.avatarUrl: /avatars/3e24c1aa9c1b4066d2dd56aeb4b0f62e.svg
paper_authors.fullname: sachin kumar
paper_authors.type: user
paper_authors.user: sachinkumar
paper_authors._id: 67b698422c8b2ef925e03f52
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:12:35.587Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_title: TESS 2: A Large-Scale Generalist Diffusion Language Model
paper_summary: We introduce TESS 2, a general instruction-following diffusion language model
  that outperforms contemporary instruction-tuned diffusion models, as well as
  matches and sometimes exceeds strong autoregressive (AR) models. We train TESS
  2 by first adapting a strong AR model via continued pretraining with the usual
  cross-entropy as diffusion loss, and then performing further instruction
  tuning. We find that adaptation training as well as the choice of the base
  model is crucial for training good instruction-following diffusion models. We
  further propose reward guidance, a novel and modular inference-time guidance
  procedure to align model outputs without needing to train the underlying model.
  Finally, we show that TESS 2 further improves with increased inference-time
  compute, highlighting the utility of diffusion LMs in having fine-grained
  controllability over the amount of compute used at inference time. Code and
  models are available at https://github.com/hamishivi/tess-2.
paper_ai_keywords: diffusion language model instruction-following diffusion models step-contrained cross-entropy diffusion loss instruction tuning reward guidance inference-time guidance inference-time compute diffusion LMs