MagicTailor: Component-Controllable Personalization in Text-to-Image
  Diffusion Models
Recent advancements in text-to-image (T2I) diffusion models have enabled the
creation of high-quality images from text prompts, but they still struggle to
generate images with precise control over specific visual concepts. Existing
approaches can replicate a given concept by learning from reference images, yet
they lack the flexibility for fine-grained customization of the individual
component within the concept. In this paper, we introduce
component-controllable personalization, a novel task that pushes the boundaries
of T2I models by allowing users to reconfigure specific components when
personalizing visual concepts. This task is particularly challenging due to two
primary obstacles: semantic pollution, where unwanted visual elements corrupt
the personalized concept, and semantic imbalance, which causes disproportionate
learning of the concept and component. To overcome these challenges, we design
MagicTailor, an innovative framework that leverages Dynamic Masked Degradation
(DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream
Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual
semantics. Extensive comparisons, ablations, and analyses demonstrate that
MagicTailor not only excels in this challenging task but also holds significant
promise for practical applications, paving the way for more nuanced and
creative image generation.
671361859186942050bfeee7
Donghao Zhou
claimed_verified
2024-10-21T07:47:53.857Z
67136093d2e50f1e8c9fad52
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0q49MyGuav8lJ9CIeyLhu.png
Donghao Zhou
user
donghao-zhou
671361859186942050bfeee8
Jiancheng Huang
admin_assigned
2024-10-21T11:47:13.861Z
6395e7c311074713ad1fc29e
/avatars/6b1e61aaaed89af05daaf15f90747976.svg
jiancheng huang
user
jiancheng
671361859186942050bfeee9
Jinbin Bai
claimed_verified
2024-10-21T07:47:51.844Z
63fccdac93b993a4ebd7789a
https://cdn-avatars.huggingface.co/v1/production/uploads/63fccdac93b993a4ebd7789a/KRx8vpdoDjsZBRw0j8Vg8.jpeg
Jinbin Bai
user
BryanW
671361859186942050bfeeea
Jiaze Wang
admin_assigned
2024-10-21T11:52:48.671Z
65f04e01a97811922fbafc86
/avatars/8bd37a4ce857d1ed5f3d39891ce1b89d.svg
Jiaze Wang
user
jzwangcuhk
671361859186942050bfeeeb
Hao Chen
671361859186942050bfeeec
Guangyong Chen
admin_assigned
2024-10-21T11:47:38.866Z
6396829c4dffc883b735e22e
/avatars/c1785f212f50bfde1d202d245a6a5ca3.svg
Guangyong Chen
user
gychen
671361859186942050bfeeed
Xiaowei Hu
671361859186942050bfeeee
Pheng-Ann Heng
MagicTailor: Component-Controllable Personalization in Text-to-Image
  Diffusion Models
Recent advancements in text-to-image (T2I) diffusion models have enabled the
creation of high-quality images from text prompts, but they still struggle to
generate images with precise control over specific visual concepts. Existing
approaches can replicate a given concept by learning from reference images, yet
they lack the flexibility for fine-grained customization of the individual
component within the concept. In this paper, we introduce
component-controllable personalization, a novel task that pushes the boundaries
of T2I models by allowing users to reconfigure specific components when
personalizing visual concepts. This task is particularly challenging due to two
primary obstacles: semantic pollution, where unwanted visual elements corrupt
the personalized concept, and semantic imbalance, which causes disproportionate
learning of the concept and component. To overcome these challenges, we design
MagicTailor, an innovative framework that leverages Dynamic Masked Degradation
(DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream
Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual
semantics. Extensive comparisons, ablations, and analyses demonstrate that
MagicTailor not only excels in this challenging task but also holds significant
promise for practical applications, paving the way for more nuanced and
creative image generation.
MagicTailor addresses challenges in T2I models by dynamically targeting and balancing specific visual components through DM-Deg and DS-Bal, enhancing precise control over image generation.
text-to-image (T2I) diffusion models component-controllable personalization semantic pollution semantic imbalance Dynamic Masked Degradation (DM-Deg) Dual-Stream Balancing (DS-Bal)