title: MagicTailor: Component-Controllable Personalization in Text-to-Image
    Diffusion Models
summary: Recent advancements in text-to-image (T2I) diffusion models have enabled the
  creation of high-quality images from text prompts, but they still struggle to
  generate images with precise control over specific visual concepts. Existing
  approaches can replicate a given concept by learning from reference images, yet
  they lack the flexibility for fine-grained customization of the individual
  component within the concept. In this paper, we introduce
  component-controllable personalization, a novel task that pushes the boundaries
  of T2I models by allowing users to reconfigure specific components when
  personalizing visual concepts. This task is particularly challenging due to two
  primary obstacles: semantic pollution, where unwanted visual elements corrupt
  the personalized concept, and semantic imbalance, which causes disproportionate
  learning of the concept and component. To overcome these challenges, we design
  MagicTailor, an innovative framework that leverages Dynamic Masked Degradation
  (DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream
  Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual
  semantics. Extensive comparisons, ablations, and analyses demonstrate that
  MagicTailor not only excels in this challenging task but also holds significant
  promise for practical applications, paving the way for more nuanced and
  creative image generation.
paper_authors._id: 671361859186942050bfeee7
paper_authors.name: Donghao Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:47:53.857Z
paper_authors._id: 67136093d2e50f1e8c9fad52
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0q49MyGuav8lJ9CIeyLhu.png
paper_authors.fullname: Donghao Zhou
paper_authors.type: user
paper_authors.user: donghao-zhou
paper_authors._id: 671361859186942050bfeee8
paper_authors.name: Jiancheng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T11:47:13.861Z
paper_authors._id: 6395e7c311074713ad1fc29e
paper_authors.avatarUrl: /avatars/6b1e61aaaed89af05daaf15f90747976.svg
paper_authors.fullname: jiancheng huang
paper_authors.type: user
paper_authors.user: jiancheng
paper_authors._id: 671361859186942050bfeee9
paper_authors.name: Jinbin Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:47:51.844Z
paper_authors._id: 63fccdac93b993a4ebd7789a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63fccdac93b993a4ebd7789a/KRx8vpdoDjsZBRw0j8Vg8.jpeg
paper_authors.fullname: Jinbin Bai
paper_authors.type: user
paper_authors.user: BryanW
paper_authors._id: 671361859186942050bfeeea
paper_authors.name: Jiaze Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T11:52:48.671Z
paper_authors._id: 65f04e01a97811922fbafc86
paper_authors.avatarUrl: /avatars/8bd37a4ce857d1ed5f3d39891ce1b89d.svg
paper_authors.fullname: Jiaze Wang
paper_authors.type: user
paper_authors.user: jzwangcuhk
paper_authors._id: 671361859186942050bfeeeb
paper_authors.name: Hao Chen
paper_authors._id: 671361859186942050bfeeec
paper_authors.name: Guangyong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T11:47:38.866Z
paper_authors._id: 6396829c4dffc883b735e22e
paper_authors.avatarUrl: /avatars/c1785f212f50bfde1d202d245a6a5ca3.svg
paper_authors.fullname: Guangyong Chen
paper_authors.type: user
paper_authors.user: gychen
paper_authors._id: 671361859186942050bfeeed
paper_authors.name: Xiaowei Hu
paper_authors._id: 671361859186942050bfeeee
paper_authors.name: Pheng-Ann Heng
paper_title: MagicTailor: Component-Controllable Personalization in Text-to-Image
    Diffusion Models
paper_summary: Recent advancements in text-to-image (T2I) diffusion models have enabled the
  creation of high-quality images from text prompts, but they still struggle to
  generate images with precise control over specific visual concepts. Existing
  approaches can replicate a given concept by learning from reference images, yet
  they lack the flexibility for fine-grained customization of the individual
  component within the concept. In this paper, we introduce
  component-controllable personalization, a novel task that pushes the boundaries
  of T2I models by allowing users to reconfigure specific components when
  personalizing visual concepts. This task is particularly challenging due to two
  primary obstacles: semantic pollution, where unwanted visual elements corrupt
  the personalized concept, and semantic imbalance, which causes disproportionate
  learning of the concept and component. To overcome these challenges, we design
  MagicTailor, an innovative framework that leverages Dynamic Masked Degradation
  (DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream
  Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual
  semantics. Extensive comparisons, ablations, and analyses demonstrate that
  MagicTailor not only excels in this challenging task but also holds significant
  promise for practical applications, paving the way for more nuanced and
  creative image generation.
paper_ai_keywords: text-to-image (T2I) diffusion models component-controllable personalization Dynamic Masked Degradation (DM-Deg) Dual-Stream Balancing (RADS-Bal) semantic pollution semantic imbalance personalized concept undesired visual semantics desired visual semantics balanced learning paradigm extensive comparisons ablations analyses