title: OminiControl: Minimal and Universal Control for Diffusion Transformer
summary: In this paper, we introduce OminiControl, a highly versatile and
  parameter-efficient framework that integrates image conditions into pre-trained
  Diffusion Transformer (DiT) models. At its core, OminiControl leverages a
  parameter reuse mechanism, enabling the DiT to encode image conditions using
  itself as a powerful backbone and process them with its flexible multi-modal
  attention processors. Unlike existing methods, which rely heavily on additional
  encoder modules with complex architectures, OminiControl (1) effectively and
  efficiently incorporates injected image conditions with only ~0.1% additional
  parameters, and (2) addresses a wide range of image conditioning tasks in a
  unified manner, including subject-driven generation and spatially-aligned
  conditions such as edges, depth, and more. Remarkably, these capabilities are
  achieved by training on images generated by the DiT itself, which is
  particularly beneficial for subject-driven generation. Extensive evaluations
  demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted
  models in both subject-driven and spatially-aligned conditional generation.
  Additionally, we release our training dataset, Subjects200K, a diverse
  collection of over 200,000 identity-consistent images, along with an efficient
  data synthesis pipeline to advance research in subject-consistent generation.
paper_authors._id: 6743e90bed93f631f50d711b
paper_authors.name: Zhenxiong Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:16:53.102Z
paper_authors._id: 664ef197d3f8759dddfbc765
paper_authors.avatarUrl: /avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
paper_authors.fullname: Tan Zhen Xiong
paper_authors.type: user
paper_authors.user: ZhenXiong
paper_authors._id: 6743e90bed93f631f50d711c
paper_authors.name: Songhua Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:16:58.817Z
paper_authors._id: 642ad435a096201096ecc580
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642ad435a096201096ecc580/uRcnkMrXbX5aTNiGcpC9d.png
paper_authors.fullname: Songhua Liu
paper_authors.type: user
paper_authors.user: flyingman
paper_authors._id: 6743e90bed93f631f50d711d
paper_authors.name: Xingyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:17:03.973Z
paper_authors._id: 634cfebc350bcee9bed20a4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
paper_authors.fullname: Xingyi Yang
paper_authors.type: user
paper_authors.user: adamdad
paper_authors._id: 6743e90bed93f631f50d711e
paper_authors.name: Qiaochu Xue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-29T10:08:20.580Z
paper_authors._id: 640ebdfefdeaae139086f4d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640ebdfefdeaae139086f4d8/2N94gbHubplYD8njmUTPf.jpeg
paper_authors.fullname: Yuanshi
paper_authors.type: user
paper_authors.user: Yuanshi
paper_authors._id: 6743e90bed93f631f50d711f
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:17:40.006Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: OminiControl: Minimal and Universal Control for Diffusion Transformer
paper_summary: In this paper, we introduce OminiControl, a highly versatile and
  parameter-efficient framework that integrates image conditions into pre-trained
  Diffusion Transformer (DiT) models. At its core, OminiControl leverages a
  parameter reuse mechanism, enabling the DiT to encode image conditions using
  itself as a powerful backbone and process them with its flexible multi-modal
  attention processors. Unlike existing methods, which rely heavily on additional
  encoder modules with complex architectures, OminiControl (1) effectively and
  efficiently incorporates injected image conditions with only ~0.1% additional
  parameters, and (2) addresses a wide range of image conditioning tasks in a
  unified manner, including subject-driven generation and spatially-aligned
  conditions such as edges, depth, and more. Remarkably, these capabilities are
  achieved by training on images generated by the DiT itself, which is
  particularly beneficial for subject-driven generation. Extensive evaluations
  demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted
  models in both subject-driven and spatially-aligned conditional generation.
  Additionally, we release our training dataset, Subjects200K, a diverse
  collection of over 200,000 identity-consistent images, along with an efficient
  data synthesis pipeline to advance research in subject-consistent generation.
paper_ai_summary: OminiControl, a parameter-efficient framework, integrates image conditions into Diffusion Transformer models using a parameter reuse mechanism, excelling in subject-driven and spatially-aligned conditional generation.
paper_ai_keywords: Diffusion Transformer DiT parameter reuse mechanism multi-modal attention processors Subjects200K data synthesis pipeline