  Despite the success of diffusion-based customization methods on visual
content creation, increasing concerns have been raised about such techniques
from both privacy and political perspectives. To tackle this issue, several
anti-customization methods have been proposed in very recent months,
predominantly grounded in adversarial attacks. Unfortunately, most of these
methods adopt straightforward designs, such as end-to-end optimization with a
focus on adversarially maximizing the original training loss, thereby
neglecting nuanced internal properties intrinsic to the diffusion model, and
even leading to ineffective optimization in some diffusion time steps. In this
paper, we strive to bridge this gap by undertaking a comprehensive exploration
of these inherent properties, to boost the performance of current
anti-customization approaches. Two aspects of properties are investigated: 1)
We examine the relationship between time step selection and the model's
perception in the frequency domain of images and find that lower time steps can
give much more contributions to adversarial noises. This inspires us to propose
an adaptive greedy search for optimal time steps that seamlessly integrates
with existing anti-customization methods. 2) We scrutinize the roles of
features at different layers during denoising and devise a sophisticated
feature-based optimization framework for anti-customization. Experiments on
facial benchmarks demonstrate that our approach significantly increases
identity disruption, thereby enhancing user privacy and security.

Computer Vision and Pattern Recognition