Pruning for Robust Concept Erasing in Diffusion Models
<|reference_start|>Pruning for Robust Concept Erasing in Diffusion Models: Despite the impressive capabilities of generating images, text-to-image diffusion models are susceptible to producing undesirable outputs such as NSFW content and copyrighted artworks. To address this issue, recent studies have focused on fine-tuning model parameters to erase problematic concepts. However, existing methods exhibit a major flaw in robustness, as fine-tuned models often reproduce the undesirable outputs when faced with cleverly crafted prompts. This reveals a fundamental limitation in the current approaches and may raise risks for the deployment of diffusion models in the open world. To address this gap, we locate the concept-correlated neurons and find that these neurons show high sensitivity to adversarial prompts, thus could be deactivated when erasing and reactivated again under attacks. To improve the robustness, we introduce a new pruning-based strategy for concept erasing. Our method selectively prunes critical parameters associated with the concepts targeted for removal, thereby reducing the sensitivity of concept-related neurons. Our method can be easily integrated with existing concept-erasing techniques, offering a robust improvement against adversarial inputs. Experimental results show a significant enhancement in our model's ability to resist adversarial inputs, achieving nearly a 40% improvement in erasing the NSFW content and a 30% improvement in erasing artwork style.<|reference_end|>
@article{yang2024pruning,
    title={Pruning for Robust Concept Erasing in Diffusion Models},
    author={Tianyun Yang, Juan Cao, Chang Xu},
    journal={arXiv preprint arXiv:2405.16534},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16534},
    primaryClass={cs.CV}
}