EmoAttack: Emotion-to-Image Diffusion Models for Emotional Backdoor Generation
<|reference_start|>EmoAttack: Emotion-to-Image Diffusion Models for Emotional Backdoor Generation: Text-to-image diffusion models can create realistic images based on input texts. Users can describe an object to convey their opinions visually. In this work, we unveil a previously unrecognized and latent risk of using diffusion models to generate images; we utilize emotion in the input texts to introduce negative contents, potentially eliciting unfavorable emotions in users. Emotions play a crucial role in expressing personal opinions in our daily interactions, and the inclusion of maliciously negative content can lead users astray, exacerbating negative emotions. Specifically, we identify the emotion-aware backdoor attack (EmoAttack) that can incorporate malicious negative content triggered by emotional texts during image generation. We formulate such an attack as a diffusion personalization problem to avoid extensive model retraining and propose the EmoBooth. Unlike existing personalization methods, our approach fine-tunes a pre-trained diffusion model by establishing a mapping between a cluster of emotional words and a given reference image containing malicious negative content. To validate the effectiveness of our method, we built a dataset and conducted extensive analysis and discussion about its effectiveness. Given consumers' widespread use of diffusion models, uncovering this threat is critical for society.<|reference_end|>
@article{wei2024emoattack:,
    title={EmoAttack: Emotion-to-Image Diffusion Models for Emotional Backdoor
  Generation},
    author={Tianyu Wei,Shanmin Pang,Qi Guo,Yizhuo Ma,Qing Guo},
    journal={arXiv preprint arXiv:2406.15863},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.15863},
    primaryClass={cs.CV}
}