LoRA text2image fine-tuning - vwu142/pokemon-lora

These are LoRA adaption weights for stabilityai/stable-diffusion-2-1. The weights were fine-tuned on the vwu142/Pokemon-Card-Plus-Pokemon-Actual-Image-And-Captions-13000 dataset. You can find some example images in the following.

Intended uses & limitations

How to use

# Importing LoRA Weights
from huggingface_hub import model_info

# LoRA weights ~3 MB
model_path = "vwu142/pokemon-lora"

# Getting Base Model
info = model_info(model_path)
model_base = info.cardData["base_model"]
print(model_base)

# Importing the Diffusion model with the weights added
import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

pipe = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.unet.load_attn_procs(model_path)
pipe.to("cuda")

Training details

The weights were trained on the Free GPU provided in Google Collab.

The data it was trained on comes from this dataset: https://huggingface.co/datasets/vwu142/Pokemon-Card-Plus-Pokemon-Actual-Image-And-Captions-13000

It has images of pokemon cards and pokemon with various descriptions of the image.

This was the parameters and the script used to train the weights

!accelerate launch --mixed_precision="fp16"  diffusers/examples/text_to_image/train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --mixed_precision="fp16" \
  --dataset_name=$DATASET_NAME --caption_column="caption"\
  --dataloader_num_workers=8 \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=1500 \
  --learning_rate=1e-04 \
  --max_grad_norm=1 \
  --lr_scheduler="cosine" --lr_warmup_steps=0 \
  --output_dir=${OUTPUT_DIR} \
  --push_to_hub \
  --hub_model_id=${HUB_MODEL_ID} \
  --report_to=wandb \
  --checkpointing_steps=500 \
  --validation_prompt="Ludicolo" \
  --seed=1337