title: MonoFormer: One Transformer for Both Diffusion and Autoregression
summary: Most existing multimodality methods use separate backbones for
  autoregression-based discrete text generation and diffusion-based continuous
  visual generation, or the same backbone by discretizing the visual data to use
  autoregression for both text and visual generation. In this paper, we propose
  to study a simple idea: share one transformer for both autoregression and
  diffusion. The feasibility comes from two main aspects: (i) Transformer is
  successfully applied to diffusion for visual generation, and (ii) transformer
  training for autoregression and diffusion is very similar, and the difference
  merely lies in that diffusion uses bidirectional attention mask and
  autoregression uses causal attention mask. Experimental results show that our
  approach achieves comparable image generation performance to current
  state-of-the-art methods as well as maintains the text generation capability.
  The project is publicly available at https://monoformer.github.io/.
paper_authors._id: 66f36fd97a2d332b0d68dc5f
paper_authors.name: Chuyang Zhao
paper_authors._id: 66f36fd97a2d332b0d68dc60
paper_authors.name: Yuxing Song
paper_authors._id: 66f36fd97a2d332b0d68dc61
paper_authors.name: Wenhao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:23:31.477Z
paper_authors._id: 62b32a4429a410b7f6b06710
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b32a4429a410b7f6b06710/VzgvmnlYZWuifZTkIkCxy.jpeg
paper_authors.fullname: Wenhao Wang
paper_authors.type: user
paper_authors.user: WenhaoWang
paper_authors._id: 66f36fd97a2d332b0d68dc62
paper_authors.name: Haocheng Feng
paper_authors._id: 66f36fd97a2d332b0d68dc63
paper_authors.name: Errui Ding
paper_authors._id: 66f36fd97a2d332b0d68dc64
paper_authors.name: Yifan Sun
paper_authors._id: 66f36fd97a2d332b0d68dc65
paper_authors.name: Xinyan Xiao
paper_authors._id: 66f36fd97a2d332b0d68dc66
paper_authors.name: Jingdong Wang
paper_title: MonoFormer: One Transformer for Both Diffusion and Autoregression
paper_summary: Most existing multimodality methods use separate backbones for
  autoregression-based discrete text generation and diffusion-based continuous
  visual generation, or the same backbone by discretizing the visual data to use
  autoregression for both text and visual generation. In this paper, we propose
  to study a simple idea: share one transformer for both autoregression and
  diffusion. The feasibility comes from two main aspects: (i) Transformer is
  successfully applied to diffusion for visual generation, and (ii) transformer
  training for autoregression and diffusion is very similar, and the difference
  merely lies in that diffusion uses bidirectional attention mask and
  autoregression uses causal attention mask. Experimental results show that our
  approach achieves comparable image generation performance to current
  state-of-the-art methods as well as maintains the text generation capability.
  The project is publicly available at https://monoformer.github.io/.
paper_ai_keywords: autoregression diffusion transformer bidirectional attention mask causal attention mask image generation text generation