|
|
--- |
|
|
license: creativeml-openrail-m |
|
|
tags: |
|
|
- text-to-image |
|
|
- stable-diffusion |
|
|
- lora |
|
|
- diffusers |
|
|
base_model: stabilityai/stable-diffusion-xl-base-1.0 |
|
|
pivotal_tuning: true |
|
|
textual_embeddings: embeddings.pti |
|
|
instance_prompt: <s0><s1> |
|
|
inference: false |
|
|
--- |
|
|
# sdxl-labyrinth LoRA by [fofr](https://replicate.com/fofr) |
|
|
### An SDXL fine-tune based on the Labyrinth movie |
|
|
|
|
|
 |
|
|
> |
|
|
|
|
|
## Inference with Replicate API |
|
|
Grab your replicate token [here](https://replicate.com/account) |
|
|
```bash |
|
|
pip install replicate |
|
|
export REPLICATE_API_TOKEN=r8_************************************* |
|
|
``` |
|
|
|
|
|
```py |
|
|
import replicate |
|
|
|
|
|
output = replicate.run( |
|
|
"sdxl-labyrinth@sha256:a95510c9f58c973633edd21fd871312505612991d1f7253db44519914bf680a2", |
|
|
input={"prompt": "A photo of a monster in the style of TOK"} |
|
|
) |
|
|
print(output) |
|
|
``` |
|
|
You may also do inference via the API with Node.js or curl, and locally with COG and Docker, [check out the Replicate API page for this model](https://replicate.com/fofr/sdxl-labyrinth/api) |
|
|
|
|
|
## Inference with 🧨 diffusers |
|
|
Replicate SDXL LoRAs are trained with Pivotal Tuning, which combines training a concept via Dreambooth LoRA with training a new token with Textual Inversion. |
|
|
As `diffusers` doesn't yet support textual inversion for SDXL, we will use cog-sdxl `TokenEmbeddingsHandler` class. |
|
|
|
|
|
The trigger tokens for your prompt will be `<s0><s1>` |
|
|
|
|
|
```shell |
|
|
pip install diffusers transformers accelerate safetensors huggingface_hub |
|
|
git clone https://github.com/replicate/cog-sdxl cog_sdxl |
|
|
``` |
|
|
|
|
|
```py |
|
|
import torch |
|
|
from huggingface_hub import hf_hub_download |
|
|
from diffusers import DiffusionPipeline |
|
|
from cog_sdxl.dataset_and_utils import TokenEmbeddingsHandler |
|
|
from diffusers.models import AutoencoderKL |
|
|
|
|
|
pipe = DiffusionPipeline.from_pretrained( |
|
|
"stabilityai/stable-diffusion-xl-base-1.0", |
|
|
torch_dtype=torch.float16, |
|
|
variant="fp16", |
|
|
).to("cuda") |
|
|
|
|
|
pipe.load_lora_weights("fofr/sdxl-labyrinth", weight_name="lora.safetensors") |
|
|
|
|
|
text_encoders = [pipe.text_encoder, pipe.text_encoder_2] |
|
|
tokenizers = [pipe.tokenizer, pipe.tokenizer_2] |
|
|
|
|
|
embedding_path = hf_hub_download(repo_id="fofr/sdxl-labyrinth", filename="embeddings.pti", repo_type="model") |
|
|
embhandler = TokenEmbeddingsHandler(text_encoders, tokenizers) |
|
|
embhandler.load_embeddings(embedding_path) |
|
|
prompt="A photo of a monster in the style of <s0><s1>" |
|
|
images = pipe( |
|
|
prompt, |
|
|
cross_attention_kwargs={"scale": 0.8}, |
|
|
).images |
|
|
#your output image |
|
|
images[0] |
|
|
``` |
|
|
|
