Rename READEME.md to README.md
Browse files- READEME.md β README.md +10 -14
READEME.md β README.md
RENAMED
|
@@ -5,7 +5,7 @@
|
|
| 5 |
<p>
|
| 6 |
|
| 7 |
<p align="center">
|
| 8 |
-
π₯οΈ <a href="https://github.com/HiDream-ai/MotionPro">GitHub</a>    ο½    π <a href="https://zhw-zhang.github.io/MotionPro-page/"><b>Project Page</b></a>    |   π€ <a href="https://huggingface.co/
|
| 9 |
<br>
|
| 10 |
|
| 11 |
[**MotionPro: A Precise Motion Controller for Image-to-Video Generation**](https://zhw-zhang.github.io/MotionPro-page/) <be>
|
|
@@ -27,18 +27,14 @@ Additionally, our repository provides more tools to benefit the research communi
|
|
| 27 |
|
| 28 |
## Video Demos
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
<div align="center">
|
| 31 |
-
<video src="assets/func_1.mp4" width="70%" autoplay loop muted playsinline poster="">
|
| 32 |
-
</video>
|
| 33 |
<p><em>Examples of different motion control types by our MotionPro.</em></p>
|
| 34 |
</div>
|
| 35 |
|
| 36 |
-
<!-- <div align="center">
|
| 37 |
-
<video src="assets/func_1.mp4" width="70%" autoplay loop muted playsinline poster="">
|
| 38 |
-
</video>
|
| 39 |
-
<p><em>Figure 2: Synchronized video generation and Video recapture.</em></p>
|
| 40 |
-
</div> -->
|
| 41 |
-
|
| 42 |
## π₯ Updates
|
| 43 |
- [x] **\[2025.03.26\]** Release inference and training code.
|
| 44 |
- [ ] **\[2025.03.27\]** Upload gradio demo usage video.
|
|
@@ -68,11 +64,11 @@ pip install -r requirements.txt
|
|
| 68 |
|
| 69 |
| Models | Download Link | Notes |
|
| 70 |
|-------------------|-------------------------------------------------------------------------------|--------------------------------------------|
|
| 71 |
-
| MotionPro | π€[Huggingface](https://huggingface.co/
|
| 72 |
-
| MotionPro-Dense | π€[Huggingface](https://huggingface.co/
|
| 73 |
|
| 74 |
|
| 75 |
-
Download the model from HuggingFace at high speeds (30-
|
| 76 |
```
|
| 77 |
cd tools/huggingface_down
|
| 78 |
bash download_hfd.sh
|
|
@@ -104,7 +100,7 @@ python demo_sparse_flex_wh_pure_camera.py
|
|
| 104 |
By combining MotionPro and MotionPro-Dense, we can achieve the following functionalities:
|
| 105 |
- Synchronized video generation. We assume that two videos, `pure_obj_motion.mp4` and `pure_camera_motion.mp4`, have been generated using the respective demos. By combining their motion flows and using the result as a condition for MotionPro-Dense, we obtain `final_video`. By pairing the same object motion with different camera motions, we can generate `synchronized videos` where the object motion remains consistent while the camera motion varies. [More Details](assets/README_syn.md)
|
| 106 |
|
| 107 |
-
Here, you need to first download the [model_weights](https://huggingface.co/
|
| 108 |
|
| 109 |
```
|
| 110 |
python inference_dense.py --ori_video 'assets/cases/dog_pure_obj_motion.mp4' --camera_video 'assets/cases/dog_pure_camera_motion_1.mp4' --save_name 'syn_video.mp4' --ckpt_path 'MotionPro-Dense CKPT-PATH'
|
|
@@ -117,7 +113,7 @@ python inference_dense.py --ori_video 'assets/cases/dog_pure_obj_motion.mp4' --c
|
|
| 117 |
<details open>
|
| 118 |
<summary><strong>Data Prepare</strong></summary>
|
| 119 |
|
| 120 |
-
We have packaged several demo videos to help users debug the training code. Simply π€[download](https://huggingface.co/
|
| 121 |
|
| 122 |
Additionally, `./data/dot_single_video` contains code for processing raw videos using [DOT](https://github.com/16lemoing/dot) to generate the necessary conditions for training, making it easier for the community to create training datasets.
|
| 123 |
|
|
|
|
| 5 |
<p>
|
| 6 |
|
| 7 |
<p align="center">
|
| 8 |
+
π₯οΈ <a href="https://github.com/HiDream-ai/MotionPro">GitHub</a>    ο½    π <a href="https://zhw-zhang.github.io/MotionPro-page/"><b>Project Page</b></a>    |   π€ <a href="https://huggingface.co/HiDream-ai/MotionPro/tree/main">Hugging Face</a>   |    π <a href="">Paper </a>    |    π <a href="">PDF</a>   
|
| 9 |
<br>
|
| 10 |
|
| 11 |
[**MotionPro: A Precise Motion Controller for Image-to-Video Generation**](https://zhw-zhang.github.io/MotionPro-page/) <be>
|
|
|
|
| 27 |
|
| 28 |
## Video Demos
|
| 29 |
|
| 30 |
+
|
| 31 |
+
https://github.com/user-attachments/assets/2af6d638-e09c-4e98-a565-43c8ca30f91b
|
| 32 |
+
|
| 33 |
+
|
| 34 |
<div align="center">
|
|
|
|
|
|
|
| 35 |
<p><em>Examples of different motion control types by our MotionPro.</em></p>
|
| 36 |
</div>
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
## π₯ Updates
|
| 39 |
- [x] **\[2025.03.26\]** Release inference and training code.
|
| 40 |
- [ ] **\[2025.03.27\]** Upload gradio demo usage video.
|
|
|
|
| 64 |
|
| 65 |
| Models | Download Link | Notes |
|
| 66 |
|-------------------|-------------------------------------------------------------------------------|--------------------------------------------|
|
| 67 |
+
| MotionPro | π€[Huggingface](https://huggingface.co/HiDream-ai/MotionPro/blob/main/MotionPro-gs_16k.pt) | Supports both object and camera control. This is the default model mentioned in the paper. |
|
| 68 |
+
| MotionPro-Dense | π€[Huggingface](https://huggingface.co/HiDream-ai/MotionPro/blob/main/MotionPro_Dense-gs_14k.pt) | Supports synchronized video generation when combined with MotionPro. MotionPro-Dense shares the same architecture as Motion, but the input conditions are modified to include: dense optical flow and per-frame visibility masks relative to the first frame. |
|
| 69 |
|
| 70 |
|
| 71 |
+
Download the model from HuggingFace at high speeds (30-75MB/s):
|
| 72 |
```
|
| 73 |
cd tools/huggingface_down
|
| 74 |
bash download_hfd.sh
|
|
|
|
| 100 |
By combining MotionPro and MotionPro-Dense, we can achieve the following functionalities:
|
| 101 |
- Synchronized video generation. We assume that two videos, `pure_obj_motion.mp4` and `pure_camera_motion.mp4`, have been generated using the respective demos. By combining their motion flows and using the result as a condition for MotionPro-Dense, we obtain `final_video`. By pairing the same object motion with different camera motions, we can generate `synchronized videos` where the object motion remains consistent while the camera motion varies. [More Details](assets/README_syn.md)
|
| 102 |
|
| 103 |
+
Here, you need to first download the [model_weights](https://huggingface.co/HiDream-ai/MotionPro/blob/main/tools/co-tracker/checkpoints/scaled_offline.pth) of cotracker and place them in the `tools/co-tracker/checkpoints` directory.
|
| 104 |
|
| 105 |
```
|
| 106 |
python inference_dense.py --ori_video 'assets/cases/dog_pure_obj_motion.mp4' --camera_video 'assets/cases/dog_pure_camera_motion_1.mp4' --save_name 'syn_video.mp4' --ckpt_path 'MotionPro-Dense CKPT-PATH'
|
|
|
|
| 113 |
<details open>
|
| 114 |
<summary><strong>Data Prepare</strong></summary>
|
| 115 |
|
| 116 |
+
We have packaged several demo videos to help users debug the training code. Simply π€[download](https://huggingface.co/HiDream-ai/MotionPro/tree/main/data), extract the files, and place them in the `./data` directory.
|
| 117 |
|
| 118 |
Additionally, `./data/dot_single_video` contains code for processing raw videos using [DOT](https://github.com/16lemoing/dot) to generate the necessary conditions for training, making it easier for the community to create training datasets.
|
| 119 |
|