OminiControl: Minimal and Universal Control for Diffusion Transformer
In this paper, we introduce OminiControl, a highly versatile and
parameter-efficient framework that integrates image conditions into pre-trained
Diffusion Transformer (DiT) models. At its core, OminiControl leverages a
parameter reuse mechanism, enabling the DiT to encode image conditions using
itself as a powerful backbone and process them with its flexible multi-modal
attention processors. Unlike existing methods, which rely heavily on additional
encoder modules with complex architectures, OminiControl (1) effectively and
efficiently incorporates injected image conditions with only ~0.1% additional
parameters, and (2) addresses a wide range of image conditioning tasks in a
unified manner, including subject-driven generation and spatially-aligned
conditions such as edges, depth, and more. Remarkably, these capabilities are
achieved by training on images generated by the DiT itself, which is
particularly beneficial for subject-driven generation. Extensive evaluations
demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted
models in both subject-driven and spatially-aligned conditional generation.
Additionally, we release our training dataset, Subjects200K, a diverse
collection of over 200,000 identity-consistent images, along with an efficient
data synthesis pipeline to advance research in subject-consistent generation.
6743e90bed93f631f50d711b
Zhenxiong Tan
admin_assigned
2024-11-25T21:16:53.102Z
664ef197d3f8759dddfbc765
/avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
Tan Zhen Xiong
user
ZhenXiong
6743e90bed93f631f50d711c
Songhua Liu
admin_assigned
2024-11-25T21:16:58.817Z
642ad435a096201096ecc580
https://cdn-avatars.huggingface.co/v1/production/uploads/642ad435a096201096ecc580/uRcnkMrXbX5aTNiGcpC9d.png
Songhua Liu
user
flyingman
6743e90bed93f631f50d711d
Xingyi Yang
admin_assigned
2024-11-25T21:17:03.973Z
634cfebc350bcee9bed20a4d
https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
Xingyi Yang
user
adamdad
6743e90bed93f631f50d711e
Qiaochu Xue
claimed_verified
2024-11-29T10:08:20.580Z
640ebdfefdeaae139086f4d8
https://cdn-avatars.huggingface.co/v1/production/uploads/640ebdfefdeaae139086f4d8/2N94gbHubplYD8njmUTPf.jpeg
Yuanshi
user
Yuanshi
6743e90bed93f631f50d711f
Xinchao Wang
admin_assigned
2024-11-25T21:17:40.006Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
OminiControl: Minimal and Universal Control for Diffusion Transformer
In this paper, we introduce OminiControl, a highly versatile and
parameter-efficient framework that integrates image conditions into pre-trained
Diffusion Transformer (DiT) models. At its core, OminiControl leverages a
parameter reuse mechanism, enabling the DiT to encode image conditions using
itself as a powerful backbone and process them with its flexible multi-modal
attention processors. Unlike existing methods, which rely heavily on additional
encoder modules with complex architectures, OminiControl (1) effectively and
efficiently incorporates injected image conditions with only ~0.1% additional
parameters, and (2) addresses a wide range of image conditioning tasks in a
unified manner, including subject-driven generation and spatially-aligned
conditions such as edges, depth, and more. Remarkably, these capabilities are
achieved by training on images generated by the DiT itself, which is
particularly beneficial for subject-driven generation. Extensive evaluations
demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted
models in both subject-driven and spatially-aligned conditional generation.
Additionally, we release our training dataset, Subjects200K, a diverse
collection of over 200,000 identity-consistent images, along with an efficient
data synthesis pipeline to advance research in subject-consistent generation.
OminiControl, a parameter-efficient framework, integrates image conditions into Diffusion Transformer models using a parameter reuse mechanism, excelling in subject-driven and spatially-aligned conditional generation.
Diffusion Transformer DiT parameter reuse mechanism multi-modal attention processors Subjects200K data synthesis pipeline