title: Transfusion: Predict the Next Token and Diffuse Images with One
    Multi-Modal Model
summary: We introduce Transfusion, a recipe for training a multi-modal model over
  discrete and continuous data. Transfusion combines the language modeling loss
  function (next token prediction) with diffusion to train a single transformer
  over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B
  parameters from scratch on a mixture of text and image data, establishing
  scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our
  experiments show that Transfusion scales significantly better than quantizing
  images and training a language model over discrete image tokens. By introducing
  modality-specific encoding and decoding layers, we can further improve the
  performance of Transfusion models, and even compress each image to just 16
  patches. We further demonstrate that scaling our Transfusion recipe to 7B
  parameters and 2T multi-modal tokens produces a model that can generate images
  and text on a par with similar scale diffusion models and language models,
  reaping the benefits of both worlds.
paper_authors._id: 66c556167690afd2f4c47e12
paper_authors.name: Chunting Zhou
paper_authors._id: 66c556167690afd2f4c47e13
paper_authors.name: Lili Yu
paper_authors._id: 66c556167690afd2f4c47e14
paper_authors.name: Arun Babu
paper_authors._id: 66c556167690afd2f4c47e15
paper_authors.name: Kushal Tirumala
paper_authors._id: 66c556167690afd2f4c47e16
paper_authors.name: Michihiro Yasunaga
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T16:39:16.320Z
paper_authors._id: 621e9388345a1d9ab65391c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e9388345a1d9ab65391c3/RxurNzyAWJOUdgeSHQi1R.jpeg
paper_authors.fullname: Michihiro Yasunaga
paper_authors.type: user
paper_authors.user: michiyasunaga
paper_authors._id: 66c556167690afd2f4c47e17
paper_authors.name: Leonid Shamis
paper_authors._id: 66c556167690afd2f4c47e18
paper_authors.name: Jacob Kahn
paper_authors._id: 66c556167690afd2f4c47e19
paper_authors.name: Xuezhe Ma
paper_authors._id: 66c556167690afd2f4c47e1a
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 66c556167690afd2f4c47e1b
paper_authors.name: Omer Levy
paper_title: Transfusion: Predict the Next Token and Diffuse Images with One
    Multi-Modal Model
paper_summary: We introduce Transfusion, a recipe for training a multi-modal model over
  discrete and continuous data. Transfusion combines the language modeling loss
  function (next token prediction) with diffusion to train a single transformer
  over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B
  parameters from scratch on a mixture of text and image data, establishing
  scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our
  experiments show that Transfusion scales significantly better than quantizing
  images and training a language model over discrete image tokens. By introducing
  modality-specific encoding and decoding layers, we can further improve the
  performance of Transfusion models, and even compress each image to just 16
  patches. We further demonstrate that scaling our Transfusion recipe to 7B
  parameters and 2T multi-modal tokens produces a model that can generate images
  and text on a par with similar scale diffusion models and language models,
  reaping the benefits of both worlds.
paper_ai_keywords: Transfusion language modeling loss function diffusion transformer mixed-modality sequences pretrain parameters scalar laws uni-modal benchmarks cross-modal benchmarks quantizing images discrete image tokens modality-specific encoding modality-specific decoding patches generate images generate text diffusion models