3. Method

ControlNet is a neural network architecture that can en-
hance large pretrained text-to-image diffusion models with
spatially localized, task-specific image conditions. We first
introduce the basic structure of a ControlNet in Section 3.1
and then describe how we apply a ControlNet to the image
diffusion model Stable Diffusion [72] in Section 3.2. We
elaborate on our training in Section 3.3 and detail several
extra considerations during inference such as composing
multiple ControlNets in Section 3.4.

3.1. ControlNet

ControlNet injects additional conditions into the blocks of
a neural network (Figure 2). Herein, we use the term network
block to refer to a set of neural layers that are commonly
put together to form a single unit of a neural network, e.g.,
resnet block, conv-bn-relu block, multi-head attention block,
transformer block, etc. Suppose F(·; Θ) is such a trained
neural block, with parameters Θ, that transforms an input
feature map x, into another feature map y as
AddingConditionalControltoText-to-ImageDiffusionModels