title: REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion
    Transformers
summary: In this paper we tackle a fundamental question: "Can we train latent
  diffusion models together with the variational auto-encoder (VAE) tokenizer in
  an end-to-end manner?" Traditional deep-learning wisdom dictates that
  end-to-end training is often preferable when possible. However, for latent
  diffusion transformers, it is observed that end-to-end training both VAE and
  diffusion-model using standard diffusion-loss is ineffective, even causing a
  degradation in final performance. We show that while diffusion loss is
  ineffective, end-to-end training can be unlocked through the
  representation-alignment (REPA) loss -- allowing both VAE and diffusion model
  to be jointly tuned during the training process. Despite its simplicity, the
  proposed training recipe (REPA-E) shows remarkable performance; speeding up
  diffusion model training by over 17x and 45x over REPA and vanilla training
  recipes, respectively. Interestingly, we observe that end-to-end tuning with
  REPA-E also improves the VAE itself; leading to improved latent space structure
  and downstream generation performance. In terms of final performance, our
  approach sets a new state-of-the-art; achieving FID of 1.26 and 1.83 with and
  without classifier-free guidance on ImageNet 256 x 256. Code is available at
  https://end2end-diffusion.github.io.
paper_authors._id: 67fdd3e3913c97aa32f94e9b
paper_authors.name: Xingjian Leng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:36.449Z
paper_authors._id: 641480554b1701c01cdb36c4
paper_authors.avatarUrl: /avatars/f1f6b294e0236d76a68c099164c81f36.svg
paper_authors.fullname: Xingjian Leng
paper_authors.type: user
paper_authors.user: xingjianleng
paper_authors._id: 67fdd3e3913c97aa32f94e9c
paper_authors.name: Jaskirat Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T08:35:52.907Z
paper_authors._id: 60cc389a0844fb1605fef405
paper_authors.avatarUrl: /avatars/ec11f85735e0525439e8821cf6d12e53.svg
paper_authors.fullname: Jaskirat Singh
paper_authors.type: user
paper_authors.user: jsingh
paper_authors._id: 67fdd3e3913c97aa32f94e9d
paper_authors.name: Yunzhong Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:34:11.946Z
paper_authors._id: 6752870ec63bc5b670b1b27e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6752870ec63bc5b670b1b27e/3CdHxnyTKbGup-1V67nEV.jpeg
paper_authors.fullname: Yunzhong Hou
paper_authors.type: user
paper_authors.user: yunzhong-hou
paper_authors._id: 67fdd3e3913c97aa32f94e9e
paper_authors.name: Zhenchang Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:34:21.190Z
paper_authors._id: 63bf533f4a2beec65568f813
paper_authors.avatarUrl: /avatars/bba583653b5cada8dd4a3ff2281e9dec.svg
paper_authors.fullname: Xing
paper_authors.type: user
paper_authors.user: Zhenchang
paper_authors._id: 67fdd3e3913c97aa32f94e9f
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:34:39.700Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_authors._id: 67fdd3e3913c97aa32f94ea0
paper_authors.name: Liang Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-17T08:34:56.041Z
paper_authors._id: 666351ebd86c026caa135e5c
paper_authors.avatarUrl: /avatars/50a37f7e999f660c69f518b71577eb7d.svg
paper_authors.fullname: Liang Zheng
paper_authors.type: user
paper_authors.user: liangzheng06
paper_title: REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion
    Transformers
paper_summary: In this paper we tackle a fundamental question: "Can we train latent
  diffusion models together with the variational auto-encoder (VAE) tokenizer in
  an end-to-end manner?" Traditional deep-learning wisdom dictates that
  end-to-end training is often preferable when possible. However, for latent
  diffusion transformers, it is observed that end-to-end training both VAE and
  diffusion-model using standard diffusion-loss is ineffective, even causing a
  degradation in final performance. We show that while diffusion loss is
  ineffective, end-to-end training can be unlocked through the
  representation-alignment (REPA) loss -- allowing both VAE and diffusion model
  to be jointly tuned during the training process. Despite its simplicity, the
  proposed training recipe (REPA-E) shows remarkable performance; speeding up
  diffusion model training by over 17x and 45x over REPA and vanilla training
  recipes, respectively. Interestingly, we observe that end-to-end tuning with
  REPA-E also improves the VAE itself; leading to improved latent space structure
  and downstream generation performance. In terms of final performance, our
  approach sets a new state-of-the-art; achieving FID of 1.26 and 1.83 with and
  without classifier-free guidance on ImageNet 256 x 256. Code is available at
  https://end2end-diffusion.github.io.
paper_ai_summary: End-to-end training of latent diffusion models and VAE tokenizers using representation-alignment loss enhances diffusion model training speed and VAE performance.
paper_ai_keywords: latent diffusion models variational auto-encoder VAE diffusion-loss representation-alignment loss REPA loss REPA-E FID ImageNet classifier-free guidance