title: Efficient Generative Model Training via Embedded Representation Warmup
summary: Diffusion models excel at generating high-dimensional data but fall short in
  training efficiency and representation quality compared to self-supervised
  methods. We identify a key bottleneck: the underutilization of high-quality,
  semantically rich representations during training notably slows down
  convergence. Our systematic analysis reveals a critical representation
  processing region -- primarily in the early layers -- where semantic and
  structural pattern learning takes place before generation can occur. To address
  this, we propose Embedded Representation Warmup (ERW), a plug-and-play
  framework where in the first stage we get the ERW module serves as a warmup
  that initializes the early layers of the diffusion model with high-quality,
  pretrained representations. This warmup minimizes the burden of learning
  representations from scratch, thereby accelerating convergence and boosting
  performance. Our theoretical analysis demonstrates that ERW's efficacy depends
  on its precise integration into specific neural network layers -- termed the
  representation processing region -- where the model primarily processes and
  transforms feature representations for later generation. We further establish
  that ERW not only accelerates training convergence but also enhances
  representation quality: empirically, our method achieves a 40times
  acceleration in training speed compared to REPA, the current state-of-the-art
  methods. Code is available at https://github.com/LINs-lab/ERW.
paper_authors._id: 67fe602166c0e8f3c2df22a9
paper_authors.name: Deyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T16:46:56.270Z
paper_authors._id: 649d59cec6b4fdd84ebe0d47
paper_authors.avatarUrl: /avatars/a070e15659c0686fdfc69e559f3d6493.svg
paper_authors.fullname: Deyuan Liu
paper_authors.type: user
paper_authors.user: SempraETY
paper_authors._id: 67fe602166c0e8f3c2df22aa
paper_authors.name: Peng Sun
paper_authors._id: 67fe602166c0e8f3c2df22ab
paper_authors.name: Xufeng Li
paper_authors._id: 67fe602166c0e8f3c2df22ac
paper_authors.name: Tao Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:57:56.241Z
paper_authors._id: 65296b8a6cdea40585bd81e2
paper_authors.avatarUrl: /avatars/824452e8b8fd056eaf7549c46393c47b.svg
paper_authors.fullname: Tao LIN
paper_authors.type: user
paper_authors.user: tlin-taolin
paper_title: Efficient Generative Model Training via Embedded Representation Warmup
paper_summary: Diffusion models excel at generating high-dimensional data but fall short in
  training efficiency and representation quality compared to self-supervised
  methods. We identify a key bottleneck: the underutilization of high-quality,
  semantically rich representations during training notably slows down
  convergence. Our systematic analysis reveals a critical representation
  processing region -- primarily in the early layers -- where semantic and
  structural pattern learning takes place before generation can occur. To address
  this, we propose Embedded Representation Warmup (ERW), a plug-and-play
  framework where in the first stage we get the ERW module serves as a warmup
  that initializes the early layers of the diffusion model with high-quality,
  pretrained representations. This warmup minimizes the burden of learning
  representations from scratch, thereby accelerating convergence and boosting
  performance. Our theoretical analysis demonstrates that ERW's efficacy depends
  on its precise integration into specific neural network layers -- termed the
  representation processing region -- where the model primarily processes and
  transforms feature representations for later generation. We further establish
  that ERW not only accelerates training convergence but also enhances
  representation quality: empirically, our method achieves a 40times
  acceleration in training speed compared to REPA, the current state-of-the-art
  methods. Code is available at https://github.com/LINs-lab/ERW.
paper_ai_summary: Embedded Representation Warmup (ERW) accelerates convergence and enhances performance of diffusion models by initializing early layers with high-quality pretrained representations.
paper_ai_keywords: diffusion models training efficiency self-supervised methods semantic representations representation processing region Embedded Representation Warmup (ERW) feature representations REPA