title: PixArt-α: Fast Training of Diffusion Transformer for
    Photorealistic Text-to-Image Synthesis
summary: The most advanced text-to-image (T2I) models require significant training
  costs (e.g., millions of GPU hours), seriously hindering the fundamental
  innovation for the AIGC community while increasing CO2 emissions. This paper
  introduces PIXART-alpha, a Transformer-based T2I diffusion model whose image
  generation quality is competitive with state-of-the-art image generators (e.g.,
  Imagen, SDXL, and even Midjourney), reaching near-commercial application
  standards. Additionally, it supports high-resolution image synthesis up to
  1024px resolution with low training cost, as shown in Figure 1 and 2. To
  achieve this goal, three core designs are proposed: (1) Training strategy
  decomposition: We devise three distinct training steps that separately optimize
  pixel dependency, text-image alignment, and image aesthetic quality; (2)
  Efficient T2I Transformer: We incorporate cross-attention modules into
  Diffusion Transformer (DiT) to inject text conditions and streamline the
  computation-intensive class-condition branch; (3) High-informative data: We
  emphasize the significance of concept density in text-image pairs and leverage
  a large Vision-Language model to auto-label dense pseudo-captions to assist
  text-image alignment learning. As a result, PIXART-alpha's training speed
  markedly surpasses existing large-scale T2I models, e.g., PIXART-alpha only
  takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU
  days), saving nearly \300,000 (26,000 vs. \320,000) and reducing 90% CO2
  emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training
  cost is merely 1%. Extensive experiments demonstrate that PIXART-\alpha
  excels in image quality, artistry, and semantic control. We hope
  PIXART-\alpha$ will provide new insights to the AIGC community and startups to
  accelerate building their own high-quality yet low-cost generative models from
  scratch.
paper_authors._id: 651b913d26b2dee1ed1d7f10
paper_authors.name: Junsong Chen
paper_authors._id: 651b913d26b2dee1ed1d7f11
paper_authors.name: Jincheng Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-04T07:37:42.255Z
paper_authors._id: 650c74e5d439bbbbadfcfbbe
paper_authors.avatarUrl: /avatars/1002835739dbb90214b5f2824a7c8c1f.svg
paper_authors.fullname: YU Jincheng
paper_authors.type: user
paper_authors.user: yujincheng08
paper_authors._id: 651b913d26b2dee1ed1d7f12
paper_authors.name: Chongjian Ge
paper_authors._id: 651b913d26b2dee1ed1d7f13
paper_authors.name: Lewei Yao
paper_authors._id: 651b913d26b2dee1ed1d7f14
paper_authors.name: Enze Xie1
paper_authors._id: 651b913d26b2dee1ed1d7f15
paper_authors.name: Yue Wu
paper_authors._id: 651b913d26b2dee1ed1d7f16
paper_authors.name: Zhongdao Wang
paper_authors._id: 651b913d26b2dee1ed1d7f17
paper_authors.name: James Kwok
paper_authors._id: 651b913d26b2dee1ed1d7f18
paper_authors.name: Ping Luo
paper_authors._id: 651b913d26b2dee1ed1d7f19
paper_authors.name: Huchuan Lu
paper_authors._id: 651b913d26b2dee1ed1d7f1a
paper_authors.name: Zhenguo Li
paper_title: PixArt-α: Fast Training of Diffusion Transformer for
    Photorealistic Text-to-Image Synthesis
paper_summary: The most advanced text-to-image (T2I) models require significant training
  costs (e.g., millions of GPU hours), seriously hindering the fundamental
  innovation for the AIGC community while increasing CO2 emissions. This paper
  introduces PIXART-alpha, a Transformer-based T2I diffusion model whose image
  generation quality is competitive with state-of-the-art image generators (e.g.,
  Imagen, SDXL, and even Midjourney), reaching near-commercial application
  standards. Additionally, it supports high-resolution image synthesis up to
  1024px resolution with low training cost, as shown in Figure 1 and 2. To
  achieve this goal, three core designs are proposed: (1) Training strategy
  decomposition: We devise three distinct training steps that separately optimize
  pixel dependency, text-image alignment, and image aesthetic quality; (2)
  Efficient T2I Transformer: We incorporate cross-attention modules into
  Diffusion Transformer (DiT) to inject text conditions and streamline the
  computation-intensive class-condition branch; (3) High-informative data: We
  emphasize the significance of concept density in text-image pairs and leverage
  a large Vision-Language model to auto-label dense pseudo-captions to assist
  text-image alignment learning. As a result, PIXART-alpha's training speed
  markedly surpasses existing large-scale T2I models, e.g., PIXART-alpha only
  takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU
  days), saving nearly \300,000 (26,000 vs. \320,000) and reducing 90% CO2
  emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training
  cost is merely 1%. Extensive experiments demonstrate that PIXART-\alpha
  excels in image quality, artistry, and semantic control. We hope
  PIXART-\alpha$ will provide new insights to the AIGC community and startups to
  accelerate building their own high-quality yet low-cost generative models from
  scratch.
paper_ai_summary: PIXART-$\alpha$ is a Transformer-based text-to-image diffusion model that achieves high-quality image synthesis with low training costs using advanced training strategies and efficient architecture.
paper_ai_keywords: Transformer-based T2I diffusion model pixel dependency text-image alignment image aesthetic quality cross-attention modules Diffusion Transformer (DiT) high-informative data concept density Vision-Language model pseudo-captions RAPHAEL