title: UniControl: A Unified Diffusion Model for Controllable Visual Generation
    In the Wild
summary: Achieving machine autonomy and human control often represent divergent
  objectives in the design of interactive AI systems. Visual generative
  foundation models such as Stable Diffusion show promise in navigating these
  goals, especially when prompted with arbitrary languages. However, they often
  fall short in generating images with spatial, structural, or geometric
  controls. The integration of such controls, which can accommodate various
  visual conditions in a single unified model, remains an unaddressed challenge.
  In response, we introduce UniControl, a new generative foundation model that
  consolidates a wide array of controllable condition-to-image (C2I) tasks within
  a singular framework, while still allowing for arbitrary language prompts.
  UniControl enables pixel-level-precise image generation, where visual
  conditions primarily influence the generated structures and language prompts
  guide the style and context. To equip UniControl with the capacity to handle
  diverse visual conditions, we augment pretrained text-to-image diffusion models
  and introduce a task-aware HyperNet to modulate the diffusion models, enabling
  the adaptation to different C2I tasks simultaneously. Trained on nine unique
  C2I tasks, UniControl demonstrates impressive zero-shot generation abilities
  with unseen visual conditions. Experimental results show that UniControl often
  surpasses the performance of single-task-controlled methods of comparable model
  sizes. This control versatility positions UniControl as a significant
  advancement in the realm of controllable visual generation.
paper_authors._id: 6466de79cdea48159aa5263f
paper_authors.name: Can Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:33.320Z
paper_authors._id: 62ccfc52bcaa438a5dbe34de
paper_authors.avatarUrl: /avatars/b7534388b0da3b3c6d805c515c76d672.svg
paper_authors.fullname: Can Qin
paper_authors.type: user
paper_authors.user: Robert001
paper_authors._id: 6466de79cdea48159aa52640
paper_authors.name: Shu Zhang
paper_authors._id: 6466de79cdea48159aa52641
paper_authors.name: Ning Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:36.728Z
paper_authors._id: 6362bcbe8f43a912fc722969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
paper_authors.fullname: Ning Yu
paper_authors.type: user
paper_authors.user: ningyu1991
paper_authors._id: 6466de79cdea48159aa52642
paper_authors.name: Yihao Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:03:57.933Z
paper_authors._id: 6354d31106d707b3324a3a5a
paper_authors.avatarUrl: /avatars/a662d176dfc4b809cd75aa726f24b74c.svg
paper_authors.fullname: Yihao Feng
paper_authors.type: user
paper_authors.user: yihaocs
paper_authors._id: 6466de79cdea48159aa52643
paper_authors.name: Xinyi Yang
paper_authors._id: 6466de79cdea48159aa52644
paper_authors.name: Yingbo Zhou
paper_authors._id: 6466de79cdea48159aa52645
paper_authors.name: Huan Wang
paper_authors._id: 6466de79cdea48159aa52646
paper_authors.name: Juan Carlos Niebles
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-01T08:45:19.907Z
paper_authors._id: 65bb26b5d61b51a50805aba2
paper_authors.avatarUrl: /avatars/966c8469f55776948a7cf42d98accff3.svg
paper_authors.fullname: Juan Carlos Niebles
paper_authors.type: user
paper_authors.user: niebles
paper_authors._id: 6466de79cdea48159aa52647
paper_authors.name: Caiming Xiong
paper_authors._id: 6466de79cdea48159aa52648
paper_authors.name: Silvio Savarese
paper_authors._id: 6466de79cdea48159aa52649
paper_authors.name: Stefano Ermon
paper_authors._id: 6466de79cdea48159aa5264a
paper_authors.name: Yun Fu
paper_authors._id: 6466de79cdea48159aa5264b
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:03:54.237Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_title: UniControl: A Unified Diffusion Model for Controllable Visual Generation
    In the Wild
paper_summary: Achieving machine autonomy and human control often represent divergent
  objectives in the design of interactive AI systems. Visual generative
  foundation models such as Stable Diffusion show promise in navigating these
  goals, especially when prompted with arbitrary languages. However, they often
  fall short in generating images with spatial, structural, or geometric
  controls. The integration of such controls, which can accommodate various
  visual conditions in a single unified model, remains an unaddressed challenge.
  In response, we introduce UniControl, a new generative foundation model that
  consolidates a wide array of controllable condition-to-image (C2I) tasks within
  a singular framework, while still allowing for arbitrary language prompts.
  UniControl enables pixel-level-precise image generation, where visual
  conditions primarily influence the generated structures and language prompts
  guide the style and context. To equip UniControl with the capacity to handle
  diverse visual conditions, we augment pretrained text-to-image diffusion models
  and introduce a task-aware HyperNet to modulate the diffusion models, enabling
  the adaptation to different C2I tasks simultaneously. Trained on nine unique
  C2I tasks, UniControl demonstrates impressive zero-shot generation abilities
  with unseen visual conditions. Experimental results show that UniControl often
  surpasses the performance of single-task-controlled methods of comparable model
  sizes. This control versatility positions UniControl as a significant
  advancement in the realm of controllable visual generation.
paper_ai_keywords: Stable Diffusion generative foundation model condition-to-image (C2I) tasks pixel-level-precise image generation pretrained text-to-image diffusion models task-aware HyperNet diffusion models zero-shot generation abilities