title: GenTron: Delving Deep into Diffusion Transformers for Image and Video
    Generation
summary: In this study, we explore Transformer-based diffusion models for image and
  video generation. Despite the dominance of Transformer architectures in various
  fields due to their flexibility and scalability, the visual generative domain
  primarily utilizes CNN-based U-Net architectures, particularly in
  diffusion-based models. We introduce GenTron, a family of Generative models
  employing Transformer-based diffusion, to address this gap. Our initial step
  was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
  process involving thorough empirical exploration of the conditioning mechanism.
  We then scale GenTron from approximately 900M to over 3B parameters, observing
  significant improvements in visual quality. Furthermore, we extend GenTron to
  text-to-video generation, incorporating novel motion-free guidance to enhance
  video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
  rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
  alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
  underscoring its strengths in compositional generation. We believe this work
  will provide meaningful insights and serve as a valuable reference for future
  research.
paper_authors._id: 657293329a5c2d6df9f62e52
paper_authors.name: Shoufa Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T08:00:13.363Z
paper_authors._id: 6412a33900634c4fe9873652
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
paper_authors.fullname: Shoufa Chen
paper_authors.type: user
paper_authors.user: ShoufaChen
paper_authors._id: 657293329a5c2d6df9f62e53
paper_authors.name: Mengmeng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T12:08:31.031Z
paper_authors._id: 6426983737d7e7882786af37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6426983737d7e7882786af37/5rG4gJIogEvyXuFahJ9yo.png
paper_authors.fullname: Frost XU
paper_authors.type: user
paper_authors.user: frostinassiky
paper_authors._id: 657293329a5c2d6df9f62e54
paper_authors.name: Jiawei Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:35:47.970Z
paper_authors._id: 64bf93139a69e8da48146cce
paper_authors.avatarUrl: /avatars/0be0719e77312b12b4b3520f98ed808b.svg
paper_authors.fullname: Jiawei Ren
paper_authors.type: user
paper_authors.user: Jiawei1992
paper_authors._id: 657293329a5c2d6df9f62e55
paper_authors.name: Yuren Cong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:35:54.417Z
paper_authors._id: 62f8040c47d782a6e286ea8e
paper_authors.avatarUrl: /avatars/f45224b851ae79fff9912f6e67159e52.svg
paper_authors.fullname: Yuren Cong
paper_authors.type: user
paper_authors.user: Yuren
paper_authors._id: 657293329a5c2d6df9f62e56
paper_authors.name: Sen He
paper_authors._id: 657293329a5c2d6df9f62e57
paper_authors.name: Yanping Xie
paper_authors._id: 657293329a5c2d6df9f62e58
paper_authors.name: Animesh Sinha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T08:00:09.848Z
paper_authors._id: 63772d28e069bdcdbb8e4a29
paper_authors.avatarUrl: /avatars/be71b00e009f2093a3ff79f07815e68e.svg
paper_authors.fullname: Animesh Sinha
paper_authors.type: user
paper_authors.user: animeshsinha
paper_authors._id: 657293329a5c2d6df9f62e59
paper_authors.name: Ping Luo
paper_authors._id: 657293329a5c2d6df9f62e5a
paper_authors.name: Tao Xiang
paper_authors._id: 657293329a5c2d6df9f62e5b
paper_authors.name: Juan-Manuel Perez-Rua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T11:21:21.728Z
paper_authors._id: 6572f4d86a1344d777fa7535
paper_authors.avatarUrl: /avatars/eb19a0d4e8d8019456a78a7d621d0955.svg
paper_authors.fullname: Juanma Perez
paper_authors.type: user
paper_authors.user: jperezrua
paper_title: GenTron: Delving Deep into Diffusion Transformers for Image and Video
    Generation
paper_summary: In this study, we explore Transformer-based diffusion models for image and
  video generation. Despite the dominance of Transformer architectures in various
  fields due to their flexibility and scalability, the visual generative domain
  primarily utilizes CNN-based U-Net architectures, particularly in
  diffusion-based models. We introduce GenTron, a family of Generative models
  employing Transformer-based diffusion, to address this gap. Our initial step
  was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
  process involving thorough empirical exploration of the conditioning mechanism.
  We then scale GenTron from approximately 900M to over 3B parameters, observing
  significant improvements in visual quality. Furthermore, we extend GenTron to
  text-to-video generation, incorporating novel motion-free guidance to enhance
  video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
  rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
  alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
  underscoring its strengths in compositional generation. We believe this work
  will provide meaningful insights and serve as a valuable reference for future
  research.
paper_ai_summary: GenTron, a Transformer-based diffusion model family, achieves superior visual quality and text alignment in image and video generation compared to existing methods.
paper_ai_keywords: Transformer-based diffusion models GenTron Diffusion Transformers text conditioning parameter scaling text-to-video generation motion-free guidance visual quality text alignment T2I-CompBench compositional generation