title: Scaling Diffusion Transformers Efficiently via μP
summary: Diffusion Transformers have emerged as the foundation for vision generative
  models, but their scalability is limited by the high cost of hyperparameter
  (HP) tuning at large scales. Recently, Maximal Update Parametrization (muP)
  was proposed for vanilla Transformers, which enables stable HP transfer from
  small to large language models, and dramatically reduces tuning costs. However,
  it remains unclear whether muP of vanilla Transformers extends to diffusion
  Transformers, which differ architecturally and objectively. In this work, we
  generalize standard muP to diffusion Transformers and validate its
  effectiveness through large-scale experiments. First, we rigorously prove that
  muP of mainstream diffusion Transformers, including DiT, U-ViT,
  PixArt-alpha, and MMDiT, aligns with that of the vanilla Transformer,
  enabling the direct application of existing muP methodologies. Leveraging
  this result, we systematically demonstrate that DiT-muP enjoys robust HP
  transferability. Notably, DiT-XL-2-muP with transferred learning rate
  achieves 2.9 times faster convergence than the original DiT-XL-2. Finally, we
  validate the effectiveness of muP on text-to-image generation by scaling
  PixArt-alpha from 0.04B to 0.61B and MMDiT from 0.18B to 18B. In both cases,
  models under muP outperform their respective baselines while requiring small
  tuning cost, only 5.5% of one training run for PixArt-alpha and 3% of
  consumption by human experts for MMDiT-18B. These results establish muP as a
  principled and efficient framework for scaling diffusion Transformers.
paper_authors._id: 682e907a24b2bb08885b94dc
paper_authors.name: Chenyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:10.939Z
paper_authors._id: 682e8e6d007cd8c2f2cd0afd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/az0B5USOPlLR3vq872JeN.png
paper_authors.fullname: Chenyu Zheng
paper_authors.type: user
paper_authors.user: ChenyuZheng
paper_authors._id: 682e907a24b2bb08885b94dd
paper_authors.name: Xinyu Zhang
paper_authors._id: 682e907a24b2bb08885b94de
paper_authors.name: Rongzhen Wang
paper_authors._id: 682e907a24b2bb08885b94df
paper_authors.name: Wei Huang
paper_authors._id: 682e907a24b2bb08885b94e0
paper_authors.name: Zhi Tian
paper_authors._id: 682e907a24b2bb08885b94e1
paper_authors.name: Weilin Huang
paper_authors._id: 682e907a24b2bb08885b94e2
paper_authors.name: Jun Zhu
paper_authors._id: 682e907a24b2bb08885b94e3
paper_authors.name: Chongxuan Li
paper_title: Scaling Diffusion Transformers Efficiently via μP
paper_summary: Diffusion Transformers have emerged as the foundation for vision generative
  models, but their scalability is limited by the high cost of hyperparameter
  (HP) tuning at large scales. Recently, Maximal Update Parametrization (muP)
  was proposed for vanilla Transformers, which enables stable HP transfer from
  small to large language models, and dramatically reduces tuning costs. However,
  it remains unclear whether muP of vanilla Transformers extends to diffusion
  Transformers, which differ architecturally and objectively. In this work, we
  generalize standard muP to diffusion Transformers and validate its
  effectiveness through large-scale experiments. First, we rigorously prove that
  muP of mainstream diffusion Transformers, including DiT, U-ViT,
  PixArt-alpha, and MMDiT, aligns with that of the vanilla Transformer,
  enabling the direct application of existing muP methodologies. Leveraging
  this result, we systematically demonstrate that DiT-muP enjoys robust HP
  transferability. Notably, DiT-XL-2-muP with transferred learning rate
  achieves 2.9 times faster convergence than the original DiT-XL-2. Finally, we
  validate the effectiveness of muP on text-to-image generation by scaling
  PixArt-alpha from 0.04B to 0.61B and MMDiT from 0.18B to 18B. In both cases,
  models under muP outperform their respective baselines while requiring small
  tuning cost, only 5.5% of one training run for PixArt-alpha and 3% of
  consumption by human experts for MMDiT-18B. These results establish muP as a
  principled and efficient framework for scaling diffusion Transformers.
paper_ai_summary: Maximal Update Parametrization (μP) is extended to diffusion Transformers, demonstrating efficient hyperparameter transferability and reduced tuning costs across various models and tasks.
paper_ai_keywords: Diffusion Transformers Maximal Update Parametrization μP hyperparameter tuning DiT U-ViT PixArt-α MMDiT text-to-image generation transfer learning convergence training run