In practice, to reduce the computational requirements to
a reasonable level, we follow the paradigm of latent diffu-
sion models proposed by Rombach et al. [57], where the
images are ﬁrst compressed to a latent space z = E(x) ∈
Rh×w×c by training a perceptual auto-encoder, consist-
ing of an encoder E and a decoder D. Using perceptual
and adversarial losses similar to VQGAN [20], the autoen-
coder achieves an excellent quality of reconstructed sam-
ples ˜x = D(E(x)), while allowing to efﬁciently train the
diffusion model on the lower dimensional pixel-space of
the learned embeddings.
In our case, we train four sim-
ilar auto-encoders, one for each of T, Ad, As and N, all
of them reducing the input resolution to latent dimensions
of h = w = 64, c = 3. Therefore, our latent diffusion
model [57] is trained on the concatenation of the 4 embed-
dings:

z = [zT, zAd , zAs, zN] ∈ R64×64×12

(7)
Relightify-Relightable3DFacesfromaSingleImageviaDiffusionModels