title: On the Scalability of Diffusion-based Text-to-Image Generation
summary: Scaling up model and data size has been quite successful for the evolution of
  LLMs. However, the scaling law for the diffusion based text-to-image (T2I)
  models is not fully explored. It is also unclear how to efficiently scale the
  model for better performance at reduced cost. The different training settings
  and expensive training cost make a fair model comparison extremely difficult.
  In this work, we empirically study the scaling properties of diffusion based
  T2I models by performing extensive and rigours ablations on scaling both
  denoising backbones and training set, including training scaled UNet and
  Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M
  images. For model scaling, we find the location and amount of cross attention
  distinguishes the performance of existing UNet designs. And increasing the
  transformer blocks is more parameter-efficient for improving text-image
  alignment than increasing channel numbers. We then identify an efficient UNet
  variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data
  scaling side, we show the quality and diversity of the training set matters
  more than simply dataset size. Increasing caption density and diversity
  improves text-image alignment performance and the learning efficiency. Finally,
  we provide scaling functions to predict the text-image alignment performance as
  functions of the scale of model size, compute and dataset size.
paper_authors._id: 660e17b6bc5f12490fd40f46
paper_authors.name: Hao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-05T10:05:47.329Z
paper_authors._id: 633bebbb20ceb9fcae5a5def
paper_authors.avatarUrl: /avatars/3b0c6587c5b9939e1da6a6dfdbe73ad7.svg
paper_authors.fullname: Hao
paper_authors.type: user
paper_authors.user: hlimax
paper_authors._id: 660e17b6bc5f12490fd40f47
paper_authors.name: Yang Zou
paper_authors._id: 660e17b6bc5f12490fd40f48
paper_authors.name: Ying Wang
paper_authors._id: 660e17b6bc5f12490fd40f49
paper_authors.name: Orchid Majumder
paper_authors._id: 660e17b6bc5f12490fd40f4a
paper_authors.name: Yusheng Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-04T07:55:54.856Z
paper_authors._id: 627b3008f5ef71071a3f7f51
paper_authors.avatarUrl: /avatars/0e17d4f424d67a5983f2915d52c26860.svg
paper_authors.fullname: Yusheng Xie
paper_authors.type: user
paper_authors.user: yushx
paper_authors._id: 660e17b6bc5f12490fd40f4b
paper_authors.name: R. Manmatha
paper_authors._id: 660e17b6bc5f12490fd40f4c
paper_authors.name: Ashwin Swaminathan
paper_authors._id: 660e17b6bc5f12490fd40f4d
paper_authors.name: Zhuowen Tu
paper_authors._id: 660e17b6bc5f12490fd40f4e
paper_authors.name: Stefano Ermon
paper_authors._id: 660e17b6bc5f12490fd40f4f
paper_authors.name: Stefano Soatto
paper_title: On the Scalability of Diffusion-based Text-to-Image Generation
paper_summary: Scaling up model and data size has been quite successful for the evolution of
  LLMs. However, the scaling law for the diffusion based text-to-image (T2I)
  models is not fully explored. It is also unclear how to efficiently scale the
  model for better performance at reduced cost. The different training settings
  and expensive training cost make a fair model comparison extremely difficult.
  In this work, we empirically study the scaling properties of diffusion based
  T2I models by performing extensive and rigours ablations on scaling both
  denoising backbones and training set, including training scaled UNet and
  Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M
  images. For model scaling, we find the location and amount of cross attention
  distinguishes the performance of existing UNet designs. And increasing the
  transformer blocks is more parameter-efficient for improving text-image
  alignment than increasing channel numbers. We then identify an efficient UNet
  variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data
  scaling side, we show the quality and diversity of the training set matters
  more than simply dataset size. Increasing caption density and diversity
  improves text-image alignment performance and the learning efficiency. Finally,
  we provide scaling functions to predict the text-image alignment performance as
  functions of the scale of model size, compute and dataset size.
paper_ai_keywords: diffusion based text-to-image (T2I) models scaling law denoising backbones training set scaling properties ablations UNet Transformer variants parameter-efficient text-image alignment cross attention transformer blocks channel numbers efficient UNet variant SDXL's UNet caption density diversity scaling functions