README.md

---
license: apache-2.0
language:
- en
tags:
- cogvideox
- video-generation
- video-to-video
- diffusers
pipeline_tag: video-to-video
---
# 🎥 CogvideoX-5b LoRa: Camera Movement Control

### 🚀 Try it here: [Interactive Demo](https://nim.video/create/2855fa68-21b1-4114-b366-53e5e4705ebf?workflow=image2video)

---

## Description
This LoRa (Low-Rank Adapter) model extends CogVideoX to control camera movement in 6 directions: `left`, `right`, `up`, `down`, `zoom_in`, and `zoom_out`. The LoRa can generate smooth camera motions for enhanced video creation.

<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/63fde49f6315a264aba6a7ed/sFer38aCSYiWtqQIrVJm9.mp4"></video>

<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/63fde49f6315a264aba6a7ed/pwyqKU0xLHZ6TuUda7hez.mp4"></video>

---


## Usage
### Usage
The LoRa was trained to control camera movement in 6 directions: `left`, `right`, `up`, `down`, `zoom_in`, `zoom_out`.
#### Prompt Format
Start prompt with text like this:
```python
'Сamera moves to the {}...',
'Сamera is moving to the {}...',
'{} camera movement...',
'{} camera turn...',
```

### Inference examples

#### ComfyUI example

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63fde49f6315a264aba6a7ed/xoCwk68lKPks0rrctHhvI.png)
<a href="https://huggingface.co/NimVideo/cogvideox1.5-5b-prompt-camera-motion/blob/main/cogvideox_1_5_5b_I2V_prompt_motion_lora_example.json">JSON File Example</a>
#### Minimal code example
```python
import torch
from diffusers import CogVideoXImageToVideoPipeline
from diffusers.utils import export_to_video, load_image


pipe = CogVideoXImageToVideoPipeline.from_pretrained(
    "THUDM/CogVideoX1.5-5B-I2V", torch_dtype=torch.bfloat16
)

pipe.load_lora_weights("NimVideo/cogvideox1.5-5b-prompt-camera-motion", adapter_name="cogvideox-lora")
pipe.set_adapters(["cogvideox-lora"], [1.0])

pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

height = 768 
width = 1360
image = load_image("resources/car.jpg").resize((width, height))
prompt = "Camera is moving to the left. A red sports car driving on a winding road."

video_generate = pipe(
    image=image,
    prompt=prompt,
    height=height, 
    width=width, 
    num_inference_steps=50,  
    num_frames=81,  
    guidance_scale=6.0,
    generator=torch.Generator().manual_seed(42), 
).frames[0]

export_to_video(video_generate, output_path, fps=8)
```

#### Inference with cli and jupyter-notebook examlple you can find <a href="https://github.com/Nim-Video/cogvideox1.5-5b-prompt-camera-motion">on our Github</a>


## Acknowledgements
Original code and models [CogVideoX](https://github.com/THUDM/CogVideo/tree/main).  

## Contacts
<p>Issues should be raised directly in the repository.</p>