Efficient Generative Model Training via Embedded Representation Warmup
Diffusion models excel at generating high-dimensional data but fall short in
training efficiency and representation quality compared to self-supervised
methods. We identify a key bottleneck: the underutilization of high-quality,
semantically rich representations during training notably slows down
convergence. Our systematic analysis reveals a critical representation
processing region -- primarily in the early layers -- where semantic and
structural pattern learning takes place before generation can occur. To address
this, we propose Embedded Representation Warmup (ERW), a plug-and-play
framework where in the first stage we get the ERW module serves as a warmup
that initializes the early layers of the diffusion model with high-quality,
pretrained representations. This warmup minimizes the burden of learning
representations from scratch, thereby accelerating convergence and boosting
performance. Our theoretical analysis demonstrates that ERW's efficacy depends
on its precise integration into specific neural network layers -- termed the
representation processing region -- where the model primarily processes and
transforms feature representations for later generation. We further establish
that ERW not only accelerates training convergence but also enhances
representation quality: empirically, our method achieves a 40times
acceleration in training speed compared to REPA, the current state-of-the-art
methods. Code is available at https://github.com/LINs-lab/ERW.
67fe602166c0e8f3c2df22a9
Deyuan Liu
claimed_verified
2025-04-15T16:46:56.270Z
649d59cec6b4fdd84ebe0d47
/avatars/a070e15659c0686fdfc69e559f3d6493.svg
Deyuan Liu
user
SempraETY
67fe602166c0e8f3c2df22aa
Peng Sun
67fe602166c0e8f3c2df22ab
Xufeng Li
67fe602166c0e8f3c2df22ac
Tao Lin
claimed_verified
2025-05-13T08:57:56.241Z
65296b8a6cdea40585bd81e2
/avatars/824452e8b8fd056eaf7549c46393c47b.svg
Tao LIN
user
tlin-taolin
Efficient Generative Model Training via Embedded Representation Warmup
Diffusion models excel at generating high-dimensional data but fall short in
training efficiency and representation quality compared to self-supervised
methods. We identify a key bottleneck: the underutilization of high-quality,
semantically rich representations during training notably slows down
convergence. Our systematic analysis reveals a critical representation
processing region -- primarily in the early layers -- where semantic and
structural pattern learning takes place before generation can occur. To address
this, we propose Embedded Representation Warmup (ERW), a plug-and-play
framework where in the first stage we get the ERW module serves as a warmup
that initializes the early layers of the diffusion model with high-quality,
pretrained representations. This warmup minimizes the burden of learning
representations from scratch, thereby accelerating convergence and boosting
performance. Our theoretical analysis demonstrates that ERW's efficacy depends
on its precise integration into specific neural network layers -- termed the
representation processing region -- where the model primarily processes and
transforms feature representations for later generation. We further establish
that ERW not only accelerates training convergence but also enhances
representation quality: empirically, our method achieves a 40times
acceleration in training speed compared to REPA, the current state-of-the-art
methods. Code is available at https://github.com/LINs-lab/ERW.
Embedded Representation Warmup (ERW) accelerates convergence and enhances performance of diffusion models by initializing early layers with high-quality pretrained representations.
diffusion models training efficiency self-supervised methods semantic representations representation processing region Embedded Representation Warmup (ERW) feature representations REPA