```python
pipe.load_lora_weights("CiroN2022/toy-face", weight_name="toy_face_sdxl.safetensors", adapter_name="toy")
```  
Make sure to include the token `toy_face` in the prompt and then you can perform inference:  
```python
prompt = "toy_face of a hacker with a hoodie"