By default IP-Adapters are inserted to all layers of the model. Use the [`~loaders.IPAdapterMixin.set_ip_adapter_scale`] method with a dictionary to assign scales to IP-Adapter at different layers.  
```py
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch