wan2.1 1.3b Lora - ValueError: Invalid LoRA checkpoint.

#13
by rahul7star - opened

all 1.3b model failing , can you suggest you model works with Wan-AI/Wan2.1-T2V-1.3B-Diffusers or which one to use

for exp I tried this
LORA_REPO = "Kijai/WanVideo_comfy"

LORA_FILE = "Wan2_1-T2V-1_3B_bf16.safetensors"

MODEL_ID = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"

That's a full model not a Lora, and it's targeted at Comfy, what are you trying to do?

I am trying to run via gradio

import torch
from diffusers import AutoencoderKLWan, WanPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
from diffusers.utils import export_to_video
import gradio as gr
import spaces

LORA_REPO = "Kijai/WanVideo_comfy"
LORA_FILE = "Wan2_1-T2V-1_3B_bf16.safetensors"
MODEL_ID = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"

LORA_SCALE = 1.0
FLOW_SHIFT = 7
EMBEDDED_GUIDANCE_SCALE = 6.0

=== Load model and VAE ===

vae = AutoencoderKLWan.from_pretrained(MODEL_ID, subfolder="vae", torch_dtype=torch.float32)

pipe = WanPipeline.from_pretrained(
MODEL_ID,
vae=vae,
torch_dtype=torch.bfloat16
)

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=FLOW_SHIFT)
pipe.to("cuda")

=== Load LoRA weights ====

pipe.load_lora_weights(
LORA_REPO,
weight_name=LORA_FILE,
lora_scale=LORA_SCALE,
lora_alpha=0.7 # matches your adapter config
)

pipe.enable_model_cpu_offload()

=== Default Negative Prompt ===

default_negative_prompt = (
"Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, "

)

=== Video generation ===

@spaces.GPU(duration=100)
def generate(prompt, negative_prompt, guidance_scale, height, width, num_frames):
output = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
height=height,
width=width,
num_frames=num_frames,
guidance_scale=guidance_scale + EMBEDDED_GUIDANCE_SCALE # embedding boost
).frames[0]

output_path = "output.mp4"
export_to_video(output, output_path, fps=16)
return output_path

=== Gradio UI ===

gr.Interface(
fn=generate,
inputs=[
gr.Textbox(label="Prompt", value="Low speed, documentary style picture, a lively white puppy is running quickly on the green grass..."),
gr.Textbox(label="Negative Prompt", value=default_negative_prompt, lines=4),
gr.Slider(label="Guidance Scale", minimum=1.0, maximum=10.0, value=5.0),
gr.Slider(label="Height", minimum=256, maximum=1024, step=64, value=480),
gr.Slider(label="Width", minimum=256, maximum=1024, step=64, value=832),
gr.Slider(label="Number of Frames", minimum=16, maximum=128, step=1, value=81)
],
outputs=gr.Video(label="Generated Video"),
title="Wan 2.1 T2V - Rahul7star LoRA v1",
description="Text-to-video generation using Wan 2.1 and custom LoRA"
).launch()

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment