How to train a LoRA based on SD3?
How to train a LoRA based on SD3?
@lisifu It's best to wait for sd-scripts to add support or wait until a different tool releases a training script
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sd3.md
I use paperspace gradient and was able to get that working with a couple of small adjustments to the code they recommend in that readme. My script is below, if it's helpful to anyone (I acknowledge that there may be more proper/efficient ways to script this; I'm not a pro, just muddling through).
A few important notes about the script:
- My gradient notebook was instantiated using the "cyberes/gradient-base-py3.10" image ... I have no idea how any of the following will behave on a vanilla paperspace notebook
- You need to open Terminal and run hugging-cli login first, you also need gated access to the SD3 model here (duh)
- You have to manually delete a file called /notebooks/storage/diffusers/examples/dreambooth/dog/.huggingface ... you may have to run the script once before this file is even formed, I'm not sure. However it happens, once it exists, it must be nuked for the script to run.
- I couldn't get wandb to work, so I added a !wandb disabled line before accelerate gets run. I assume you can just remove the --report_to line from the training script at the end, if you prefer.
!pip install --upgrade diffusers[torch]
!pip install --upgrade diffusers[flax]
!pip install peft bitsandbytes
!git clone https://github.com/huggingface/diffusers
%cd diffusers
!pip install -e .
%cd examples/dreambooth
!pip install -r requirements_sd3.txt
!accelerate config default
#from accelerate.utils import write_basic_config
#write_basic_config()
from huggingface_hub import snapshot_download
local_dir = "./dog-ex"
snapshot_download(
"diffusers/dog-example",
local_dir=local_dir, repo_type="dataset",
ignore_patterns=".gitattributes",
)
MODEL_NAME="stabilityai/stable-diffusion-3-medium-diffusers"
INSTANCE_DIR="dog"
OUTPUT_DIR="trained-sd3-lora"
!wandb disabled
!accelerate launch train_dreambooth_lora_sd3.py
--pretrained_model_name_or_path=$MODEL_NAME
--instance_data_dir=$INSTANCE_DIR
--output_dir=/raid/.cache/$OUTPUT_DIR
--mixed_precision="fp16"
--instance_prompt="a photo of sks dog"
--resolution=1024
--train_batch_size=1
--gradient_accumulation_steps=4
--learning_rate=1e-5
--report_to="wandb"
--lr_scheduler="constant"
--lr_warmup_steps=0
--max_train_steps=500
--weighting_scheme="logit_normal"
--validation_prompt="A photo of sks dog in a bucket"
--validation_epochs=25
--seed="0"
--push_to_hub
Seems that using diffusers library for lora training wont't work on 8GB vram. tr5 text encoder doesn't fit into vram. :-(
Maybe one could skip tr5.. let's see
Is it possible to train using tags in accompanying .txt files? It doesn't seem like instance_prompt supports filewords.
Is it possible to train using tags in accompanying .txt files? It doesn't seem like instance_prompt supports filewords.
No. That's part of https://github.com/kohya-ss/sd-scripts
Is it possible to train using tags in accompanying .txt files? It doesn't seem like instance_prompt supports filewords.
Unfortunately not. The SD3 training with Dreambooth I've managed to do so far is both slow and terrible. I eagerly await Kohya's SD3 release. Running their test example there with the dog produced hilariously stupid results and running my own concept for 8000 steps was no better.
Is it possible to train using tags in accompanying .txt files? It doesn't seem like instance_prompt supports filewords.
Unfortunately not. The SD3 training with Dreambooth I've managed to do so far is both slow and terrible. I eagerly await Kohya's SD3 release. Running their test example there with the dog produced hilariously stupid results and running my own concept for 8000 steps was no better.
That's usually the case when you don't have enough vram.