base_model:
- THUDM/CogVideoX-5b
language:
- en
library_name: diffusers
license: other
pipeline_tag: text-to-video
tags:
- video
- video-generation
- cogvideox
- alibaba

[🔥ACM MM'25]Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation
Zhenghao Zhang*, Junchao Liao*, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, Weizhi Wang
* equal contribution
Please visit our Github repo for more details.
💡 Abstract
Recent advances in diffusion transformer models for motion-guided video generation, such as Tora, have shown significant progress. In this paper, we present Tora2, an enhanced version of Tora, which introduces several design improvements to expand its capabilities in both appearance and motion customization. Specifically, we introduce a decoupled personalization extractor that generates comprehensive personalization embeddings for multiple open-set entities, better preserving fine-grained visual details compared to previous methods. Building on this, we design a gated self-attention mechanism to integrate trajectory, textual description, and visual information for each entity. This innovation significantly reduces misalignment in multimodal conditioning during training. Moreover, we introduce a contrastive loss that jointly optimizes trajectory dynamics and entity consistency through explicit mapping between motion and personalization embeddings. Tora2 is, to our best knowledge, the first method to achieve simultaneous multi-entity customization of appearance and motion for video generation. Experimental results demonstrate that Tora2 achieves competitive performance with state-of-the-art customization methods while providing advanced motion control capabilities, which marks a critical advancement in multi-condition video generation. Project page: this https URL .
📣 Updates
2025/07/08
🔥🔥 Our latest work, Tora2, has been accepted by ACM MM25. Tora2 builds on Tora with design improvements, enabling enhanced appearance and motion customization for multiple entities.2025/05/24
We open-sourced a LoRA-finetuned model of Wan. It turns things in the image into fluffy toys. Check this out: https://github.com/alibaba/wan-toy-transform2025/01/06
🔥🔥We released Tora Image-to-Video, including inference code and model weights.2024/12/13
SageAttention2 and model compilation are supported in diffusers version. Tested on the A10, these approaches speed up every inference step by approximately 52%, except for the first step.2024/12/09
🔥🔥Diffusers version of Tora and the corresponding model weights are released. Inference VRAM requirements are reduced to around 5 GiB. Please refer to this for details.2024/11/25
🔥Text-to-Video training code released.2024/10/31
Model weights uploaded to HuggingFace. We also provided an English demo on ModelScope.2024/10/23
🔥🔥Our ModelScope Demo is launched. Welcome to try it out! We also upload the model weights to ModelScope.2024/10/21
Thanks to @kijai for supporting Tora in ComfyUI! Link2024/10/15
🔥🔥We released our inference code and model weights. Please note that this is a CogVideoX version of Tora, built on the CogVideoX-5B model. This version of Tora is meant for academic research purposes only. Due to our commercial plans, we will not be open-sourcing the complete version of Tora at this time.2024/08/27
We released our v2 paper including appendix.2024/07/31
We submitted our paper on arXiv and released our project page.
📑 Table of Contents
- 🎞️ Showcases
- ✅ TODO List
- 🧨 Diffusers verision
- 🐍 Installation
- 📦 Model Weights
- 🔄 Inference
- 🖥️ Gradio Demo
- 🧠 Training
- 🎯 Troubleshooting
- 🤝 Acknowledgements
- 📄 Our previous work
- 📚 Citation
🎞️ Showcases
https://github.com/user-attachments/assets/949d5e99-18c9-49d6-b669-9003ccd44bf1
https://github.com/user-attachments/assets/7e7dbe87-a8ba-4710-afd0-9ef528ec329b
https://github.com/user-attachments/assets/4026c23d-229d-45d7-b5be-6f3eb9e4fd50
All videos are available in this Link
✅ TODO List
- Release our inference code and model weights
- Provide a ModelScope Demo
- Release our training code
- Release diffusers version and optimize the GPU memory usage
- Release complete version of Tora
📦 Model Weights
Folder Structure
Tora
└── sat
└── ckpts
├── t5-v1_1-xxl
│ ├── model-00001-of-00002.safetensors
│ └── ...
├── vae
│ └── 3d-vae.pt
├── tora
│ ├── i2v
│ │ └── mp_rank_00_model_states.pt
│ └── t2v
│ └── mp_rank_00_model_states.pt
└── CogVideoX-5b-sat # for training stage 1
└── mp_rank_00_model_states.pt
Download Links
Note: Downloading the tora
weights requires following the CogVideoX License. You can choose one of the following options: HuggingFace, ModelScope, or native links.
After downloading the model weights, you can put them in the Tora/sat/ckpts
folder.
HuggingFace
# This can be faster
pip install "huggingface_hub[hf_transfer]"
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download Alibaba-Research-Intelligence-Computing/Tora --local-dir ckpts
or
# use git
git lfs install
git clone https://huggingface.co/Alibaba-Research-Intelligence-Computing/Tora
ModelScope
- SDK
from modelscope import snapshot_download
model_dir = snapshot_download('xiaoche/Tora')
- Git
git clone https://www.modelscope.cn/xiaoche/Tora.git
Native
- Download the VAE and T5 model following CogVideo:\
- Tora t2v model weights: Link. Downloading this weight requires following the CogVideoX License.
🔄 Inference
Text to Video
It requires around 30 GiB GPU memory tested on NVIDIA A100.
cd sat
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU sample_video.py --base configs/tora/model/cogvideox_5b_tora.yaml configs/tora/inference_sparse.yaml --load ckpts/tora/t2v --output-dir samples --point_path trajs/coaster.txt --input-file assets/text/t2v/examples.txt
You can change the --input-file
and --point_path
to your own prompts and trajectory points files. Please note that the trajectory is drawn on a 256x256 canvas.
Replace $N_GPU
with the number of GPUs you want to use.
Image to Video
cd sat
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU sample_video.py --base configs/tora/model/cogvideox_5b_tora_i2v.yaml configs/tora/inference_sparse.yaml --load ckpts/tora/i2v --output-dir samples --point_path trajs/sawtooth.txt --input-file assets/text/i2v/examples.txt --img_dir assets/images --image2video
The first frame images should be placed in the --img_dir
. The names of these images should be specified in the corresponding text prompt in --input-file
, seperated by @@
.
Recommendations for Text Prompts
For text prompts, we highly recommend using GPT-4 to enhance the details. Simple prompts may negatively impact both visual quality and motion control effectiveness.
You can refer to the following resources for guidance:
🖥️ Gradio Demo
Usage:
cd sat
python app.py --load ckpts/tora/t2v
🧠 Training
Data Preparation
Following this guide https://github.com/THUDM/CogVideo/blob/main/sat/README.md#preparing-the-dataset, structure the datasets as follows:
.
├── labels
│ ├── 1.txt
│ ├── 2.txt
│ ├── ...
└── videos
├── 1.mp4
├── 2.mp4
├── ...
Training data examples are in sat/training_examples
Text to Video
It requires around 60 GiB GPU memory tested on NVIDIA A100.
Replace $N_GPU
with the number of GPUs you want to use.
- Stage 1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU train_video.py --base configs/tora/model/cogvideox_5b_tora.yaml configs/tora/train_dense.yaml --experiment-name "t2v-stage1"
- Stage 2
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU train_video.py --base configs/tora/model/cogvideox_5b_tora.yaml configs/tora/train_sparse.yaml --experiment-name "t2v-stage2"
🎯 Troubleshooting
1. ValueError: Non-consecutive added token...
Upgrade the transformers package to 4.44.2. See this issue.
🤝 Acknowledgements
We would like to express our gratitude to the following open-source projects that have been instrumental in the development of our project:
- CogVideo: An open source video generation framework by THUKEG.
- Open-Sora: An open source video generation framework by HPC-AI Tech.
- MotionCtrl: A video generation model supporting motion control by ARC Lab, Tencent PCG.
- ComfyUI-DragNUWA: An implementation of DragNUWA for ComfyUI.
Special thanks to the contributors of these libraries for their hard work and dedication!
📄 Our previous work
📚 Citation
@article{zhang2025tora2,
title={Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation},
author={Zhenghao Zhang and Junchao Liao and Menghao Li and Zuozhuo Dai and Bingxue Qiu and Siyu Zhu and Long Qin and Weizhi Wang},
journal={ACM Multimedia (MM)},
year={2025}
}