You can use PEFT to easily fuse/unfuse multiple adapters directly into the model weights (both UNet and text encoder) using the [`~loaders.lora_base.LoraBaseMixin.fuse_lora`] method, which can lead to a speed-up in inference and lower VRAM usage.  
For example, if you have a base model and adapters loaded and set as active with the following adapter weights:  
```py
from diffusers import DiffusionPipeline
import torch