title: One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and
    Erasing Applications
summary: The prevalent use of commercial and open-source diffusion models (DMs) for
  text-to-image generation prompts risk mitigation to prevent undesired
  behaviors. Existing concept erasing methods in academia are all based on full
  parameter or specification-based fine-tuning, from which we observe the
  following issues: 1) Generation alternation towards erosion: Parameter drift
  during target elimination causes alternations and potential deformations across
  all generations, even eroding other concepts at varying degrees, which is more
  evident with multi-concept erased; 2) Transfer inability & deployment
  inefficiency: Previous model-specific erasure impedes the flexible combination
  of concepts and the training-free transfer towards other models, resulting in
  linear cost growth as the deployment scenarios increase. To achieve
  non-invasive, precise, customizable, and transferable elimination, we ground
  our erasing framework on one-dimensional adapters to erase multiple concepts
  from most DMs at once across versatile erasing applications. The
  concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to
  learn targeted erasing, and meantime the alteration and erosion phenomenon is
  effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once
  obtained, SPMs can be flexibly combined and plug-and-play for other DMs without
  specific re-tuning, enabling timely and efficient adaptation to diverse
  scenarios. During generation, our Facilitated Transport mechanism dynamically
  regulates the permeability of each SPM to respond to different input prompts,
  further minimizing the impact on other concepts. Quantitative and qualitative
  results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated
  the superior erasing of SPM. Our code and pre-tuned SPMs will be available on
  the project page https://lyumengyao.github.io/projects/spm.
paper_authors._id: 658bb39370d10c8982e4f7fe
paper_authors.name: Mengyao Lyu
paper_authors._id: 658bb39370d10c8982e4f7ff
paper_authors.name: Yuhong Yang
paper_authors._id: 658bb39370d10c8982e4f800
paper_authors.name: Haiwen Hong
paper_authors._id: 658bb39370d10c8982e4f801
paper_authors.name: Hui Chen
paper_authors._id: 658bb39370d10c8982e4f802
paper_authors.name: Xuan Jin
paper_authors._id: 658bb39370d10c8982e4f803
paper_authors.name: Yuan He
paper_authors._id: 658bb39370d10c8982e4f804
paper_authors.name: Hui Xue
paper_authors._id: 658bb39370d10c8982e4f805
paper_authors.name: Jungong Han
paper_authors._id: 658bb39370d10c8982e4f806
paper_authors.name: Guiguang Ding
paper_title: One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and
    Erasing Applications
paper_summary: The prevalent use of commercial and open-source diffusion models (DMs) for
  text-to-image generation prompts risk mitigation to prevent undesired
  behaviors. Existing concept erasing methods in academia are all based on full
  parameter or specification-based fine-tuning, from which we observe the
  following issues: 1) Generation alternation towards erosion: Parameter drift
  during target elimination causes alternations and potential deformations across
  all generations, even eroding other concepts at varying degrees, which is more
  evident with multi-concept erased; 2) Transfer inability & deployment
  inefficiency: Previous model-specific erasure impedes the flexible combination
  of concepts and the training-free transfer towards other models, resulting in
  linear cost growth as the deployment scenarios increase. To achieve
  non-invasive, precise, customizable, and transferable elimination, we ground
  our erasing framework on one-dimensional adapters to erase multiple concepts
  from most DMs at once across versatile erasing applications. The
  concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to
  learn targeted erasing, and meantime the alteration and erosion phenomenon is
  effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once
  obtained, SPMs can be flexibly combined and plug-and-play for other DMs without
  specific re-tuning, enabling timely and efficient adaptation to diverse
  scenarios. During generation, our Facilitated Transport mechanism dynamically
  regulates the permeability of each SPM to respond to different input prompts,
  further minimizing the impact on other concepts. Quantitative and qualitative
  results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated
  the superior erasing of SPM. Our code and pre-tuned SPMs will be available on
  the project page https://lyumengyao.github.io/projects/spm.
paper_ai_keywords: diffusion models (DMs) text-to-image generation concept erasing full parameter fine-tuning specification-based fine-tuning parameter drift concept erosion one-dimensional adapters concept-SemiPermeable structure Membrane (SPM) targeted erasing Latent Anchoring fine-tuning strategy Facilitated Transport mechanism pre-tuned SPMs