L = Ez0,t,ct,cf,(cid:15)∼N (0,1)

(10)
where L is the overall learning objective of the entire diffusion model. This learning objective can be
directly used in ﬁne tuning diffusion models.
During the training, we randomly replace 50% text prompts ct with empty strings. This facilitates
ControlNet’s capability to recognize semantic contents from input condition maps, e.g., Canny edge
maps or human scribbles, etc. This is mainly because when the prompt is not visible for the SD
model, the encoder tends to learn more semantics from input control maps as a replacement for the
prompt.

2

(cid:104)(cid:107)(cid:15) − (cid:15)θ(zt, t, ct, cf))(cid:107)2

(cid:105)

3.4

Improved Training
Adding Conditional Control to Text-to-Image Diffusion Models