k = {Ad, As, N}

(9a)
(9b)
(9c)
(9d)

(cid:19)

(9e)

(9f)

∂L
∂zTt

t−1

Given a sample zt at timestep t, we ﬁrst sample the next de-
noised sample zt−1 using the original reverse diffusion step
(Eq. 9a). We term this as zunknown
(borrowing the notation
from [47]) as it does not take into account the known parts
of the observed texture. To exploit the known texture, we
at timestep t−1 via a for-
sample a noisy version of it zknown
Tt−1
ward diffusion step (Eq. 9b). Then, we directly impose this
((cid:12) denotes the Hadamard
known noisy texture m (cid:12) zknown
product) as in the ﬁrst half of Eq. 9e. Finally, for the un-
known pixels, we add the manifold constraint introduced in
MCG [11]; i.e. we make a prediction of the clean sample ˆz0
(Eq. 9c) based on the previous timestep zt, compare this ((cid:96)2
loss) with the ground truth in the known regions (Eq. 9d),
and use the gradient of this loss to update the unknown pix-
els of zt−1 (Eq. 9e and 9f) so as to minimize this distance.
Relightify-Relightable3DFacesfromaSingleImageviaDiffusionModels