Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation
<|reference_start|>Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation: Existing generative adversarial network (GAN) based conditional image generative models typically produce fixed output for the same conditional input, which is unreasonable for highly subjective tasks, such as large-mask image inpainting or style transfer. On the other hand, GAN-based diverse image generative methods require retraining/fine-tuning the network or designing complex noise injection functions, which is computationally expensive, task-specific, or struggle to generate high-quality results. Given that many deterministic conditional image generative models have been able to produce high-quality yet fixed results, we raise an intriguing question: is it possible for pre-trained deterministic conditional image generative models to generate diverse results without changing network structures or parameters? To answer this question, we re-examine the conditional image generation tasks from the perspective of adversarial attack and propose a simple and efficient plug-in projected gradient descent (PGD) like method for diverse and controllable image generation. The key idea is attacking the pre-trained deterministic generative models by adding a micro perturbation to the input condition. In this way, diverse results can be generated without any adjustment of network structures or fine-tuning of the pre-trained models. In addition, we can also control the diverse results to be generated by specifying the attack direction according to a reference text or image. Our work opens the door to applying adversarial attack to low-level vision tasks, and experiments on various conditional image generation tasks demonstrate the effectiveness and superiority of the proposed method.<|reference_end|>
@article{chu2024attack,
    title={Attack Deterministic Conditional Image Generative Models for Diverse and
  Controllable Generation},
    author={Tianyi Chu, Wei Xing, Jiafu Chen, Zhizhong Wang, Jiakai Sun, Lei Zhao,
  Haibo Chen, Huaizhong Lin},
    journal={arXiv preprint arXiv:2403.08294},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.08294},
    primaryClass={cs.CV}
}