title: FreeControl: Training-Free Spatial Control of Any Text-to-Image
    Diffusion Model with Any Condition
summary: Recent approaches such as ControlNet offer users fine-grained spatial control
  over text-to-image (T2I) diffusion models. However, auxiliary modules have to
  be trained for each type of spatial condition, model architecture, and
  checkpoint, putting them at odds with the diverse intents and preferences a
  human designer would like to convey to the AI models during the content
  creation process. In this work, we present FreeControl, a training-free
  approach for controllable T2I generation that supports multiple conditions,
  architectures, and checkpoints simultaneously. FreeControl designs structure
  guidance to facilitate the structure alignment with a guidance image, and
  appearance guidance to enable the appearance sharing between images generated
  using the same seed. Extensive qualitative and quantitative experiments
  demonstrate the superior performance of FreeControl across a variety of
  pre-trained T2I models. In particular, FreeControl facilitates convenient
  training-free control over many different architectures and checkpoints, allows
  the challenging input conditions on which most of the existing training-free
  methods fail, and achieves competitive synthesis quality with training-based
  approaches.
paper_authors._id: 65794194c17736d75f8391aa
paper_authors.name: Sicheng Mo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T08:32:29.083Z
paper_authors._id: 637c94d3f219c71f93eda9ad
paper_authors.avatarUrl: /avatars/6dae0c30755196ccc0a5a06b3981c47f.svg
paper_authors.fullname: Sicheng Mo
paper_authors.type: user
paper_authors.user: Sichengmo
paper_authors._id: 65794194c17736d75f8391ab
paper_authors.name: Fangzhou Mu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T08:58:39.371Z
paper_authors._id: 65484f95c6dadd513f51f495
paper_authors.avatarUrl: /avatars/cf7ea78cac16500df451556a071101e2.svg
paper_authors.fullname: Fangzhou Mu
paper_authors.type: user
paper_authors.user: fmu2
paper_authors._id: 65794194c17736d75f8391ac
paper_authors.name: Kuan Heng Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T08:32:34.357Z
paper_authors._id: 638830946e484fea6da1783f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638830946e484fea6da1783f/-0KSxFtp9d_TVUo9HUKrK.png
paper_authors.fullname: Kuan Heng (Jordan) Lin
paper_authors.type: user
paper_authors.user: jordanlin
paper_authors._id: 65794194c17736d75f8391ad
paper_authors.name: Yanli Liu
paper_authors._id: 65794194c17736d75f8391ae
paper_authors.name: Bochen Guan
paper_authors._id: 65794194c17736d75f8391af
paper_authors.name: Yin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-14T10:34:43.710Z
paper_authors._id: 6310311358d83e8f64dbc3f0
paper_authors.avatarUrl: /avatars/85c0bbfddb44871cbefaa8dd17e3a246.svg
paper_authors.fullname: Yin Li
paper_authors.type: user
paper_authors.user: happyharrycn
paper_authors._id: 65794194c17736d75f8391b0
paper_authors.name: Bolei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T08:59:28.229Z
paper_authors._id: 64aedf11c693a19547999a59
paper_authors.avatarUrl: /avatars/83a54b7c217168c91cea15dd000f4af2.svg
paper_authors.fullname: Bolei Zhou
paper_authors.type: user
paper_authors.user: zhoubolei
paper_title: FreeControl: Training-Free Spatial Control of Any Text-to-Image
    Diffusion Model with Any Condition
paper_summary: Recent approaches such as ControlNet offer users fine-grained spatial control
  over text-to-image (T2I) diffusion models. However, auxiliary modules have to
  be trained for each type of spatial condition, model architecture, and
  checkpoint, putting them at odds with the diverse intents and preferences a
  human designer would like to convey to the AI models during the content
  creation process. In this work, we present FreeControl, a training-free
  approach for controllable T2I generation that supports multiple conditions,
  architectures, and checkpoints simultaneously. FreeControl designs structure
  guidance to facilitate the structure alignment with a guidance image, and
  appearance guidance to enable the appearance sharing between images generated
  using the same seed. Extensive qualitative and quantitative experiments
  demonstrate the superior performance of FreeControl across a variety of
  pre-trained T2I models. In particular, FreeControl facilitates convenient
  training-free control over many different architectures and checkpoints, allows
  the challenging input conditions on which most of the existing training-free
  methods fail, and achieves competitive synthesis quality with training-based
  approaches.
paper_ai_summary: FreeControl provides a training-free method to control text-to-image diffusion models with multiple conditions, architectures, and checkpoints, demonstrating competitive performance to training-based approaches.
paper_ai_keywords: ControlNet text-to-image (T2I) diffusion models free control structure guidance appearance guidance