Hugging Face's logo

jbilcke-hf
/
CogVideoX-Fun-V1.5-5b-for-InferenceEndpoints

Diffusers
Safetensors
CogVideoXPipeline
Inference Endpoints
Model card Files Community
jbilcke-hf HF staff commited on
Commit
c91ba9a
•
1 Parent(s): 10db43e

Upload 5 files

Browse files
Files changed (5) hide show
  1. LICENSE +71 -0
  2. README_en.md +464 -0
  3. configuration.json +1 -0
  4. gitattributes +35 -0
  5. model_index.json +24 -0
LICENSE ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The CogVideoX License
2
+
3
+ 1. Definitions
4
+
5
+ “Licensor” means the CogVideoX Model Team that distributes its Software.
6
+
7
+ “Software” means the CogVideoX model parameters made available under this license.
8
+
9
+ 2. License Grant
10
+
11
+ Under the terms and conditions of this license, the licensor hereby grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license. The intellectual property rights of the generated content belong to the user to the extent permitted by applicable local laws.
12
+ This license allows you to freely use all open-source models in this repository for academic research. Users who wish to use the models for commercial purposes must register and obtain a basic commercial license in https://open.bigmodel.cn/mla/form .
13
+ Users who have registered and obtained the basic commercial license can use the models for commercial activities for free, but must comply with all terms and conditions of this license. Additionally, the number of service users (visits) for your commercial activities must not exceed 1 million visits per month.
14
+ If the number of service users (visits) for your commercial activities exceeds 1 million visits per month, you need to contact our business team to obtain more commercial licenses.
15
+ The above copyright statement and this license statement should be included in all copies or significant portions of this software.
16
+
17
+ 3. Restriction
18
+
19
+ You will not use, copy, modify, merge, publish, distribute, reproduce, or create derivative works of the Software, in whole or in part, for any military, or illegal purposes.
20
+
21
+ You will not use the Software for any act that may undermine China's national security and national unity, harm the public interest of society, or infringe upon the rights and interests of human beings.
22
+
23
+ 4. Disclaimer
24
+
25
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
26
+
27
+ 5. Limitation of Liability
28
+
29
+ EXCEPT TO THE EXTENT PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER BASED IN TORT, NEGLIGENCE, CONTRACT, LIABILITY, OR OTHERWISE WILL ANY LICENSOR BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES, OR ANY OTHER COMMERCIAL LOSSES, EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
30
+
31
+ 6. Dispute Resolution
32
+
33
+ This license shall be governed and construed in accordance with the laws of People’s Republic of China. Any dispute arising from or in connection with this License shall be submitted to Haidian District People's Court in Beijing.
34
+
35
+ Note that the license is subject to update to a more comprehensive version. For any questions related to the license and copyright, please contact us at [email protected].
36
+
37
+ 1. 定义
38
+
39
+ “许可方”是指分发其软件的 CogVideoX 模型团队。
40
+
41
+ “软件”是指根据本许可提供的 CogVideoX 模型参数。
42
+
43
+ 2. 许可授予
44
+
45
+ 根据本许可的条款和条件，许可方特此授予您非排他性、全球性、不可转让、不可再许可、可撤销、免版税的版权许可。生成内容的知识产权所属，可根据适用当地法律的规定，在法律允许的范围内由用户享有生成内容的知识产权或其他权利。
46
+ 本许可允许您免费使用本仓库中的所有开源模型进行学术研究。对于希望将模型用于商业目的的用户，需在 https://open.bigmodel.cn/mla/form 完成登记并获得基础商用授权。
47
+
48
+ 经过登记并获得基础商用授权的用户可以免费使用本模型进行商业活动，但必须遵守本许可的所有条款和条件。
49
+ 在本许可证下，您的商业活动的服务用户数量（访问量）不得超过100万人次访问 / 每月。如果超过，您需要与我们的商业团队联系以获得更多的商业许可。
50
+ 上述版权声明和本许可声明应包含在本软件的所有副本或重要部分中。
51
+
52
+ 3.限制
53
+
54
+ 您不得出于任何军事或非法目的使用、复制、修改、合并、发布、分发、复制或创建本软件的全部或部分衍生作品。
55
+
56
+ 您不得利用本软件从事任何危害国家安全和国家统一、危害社会公共利益、侵犯人身权益的行为。
57
+
58
+ 4.免责声明
59
+
60
+ 本软件“按原样”提供，不提供任何明示或暗示的保证，包括但不限于对适销性、特定用途的适用性和非侵权性的保证。
61
+ 在任何情况下，作者或版权持有人均不对任何索赔、损害或其他责任负责，无论是在合同诉讼、侵权行为还是其他方面，由软件或软件的使用或其他交易引起、由软件引起或与之相关 软件。
62
+
63
+ 5. 责任限制
64
+
65
+ 除适用��律禁止的范围外，在任何情况下且根据任何法律理论，无论是基于侵权行为、疏忽、合同、责任或其他原因，任何许可方均不对您承担任何直接、间接、特殊、偶然、示范性、 或间接损害，或任何其他商业损失，即使许可人已被告知此类损害的可能性。
66
+
67
+ 6.争议解决
68
+
69
+ 本许可受中华人民共和国法律管辖并按其解释。 因本许可引起的或与本许可有关的任何争议应提交北京市海淀区人民法院。
70
+
71
+ 请注意，许可证可能会更新到更全面的版本。 有关许可和版权的任何问题，请通过 [email protected] 与我们联系。
README_en.md ADDED
@@ -0,0 +1,464 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # CogVideoX-Fun
2
+
3
+ 😊 Welcome!
4
+
5
+ [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https://huggingface.co/spaces/alibaba-pai/CogVideoX-Fun-5b)
6
+
7
+ English | [简体中文](./README.md)
8
+
9
+ # Table of Contents
10
+ - [Table of Contents](#table-of-contents)
11
+ - [Introduction](#introduction)
12
+ - [Quick Start](#quick-start)
13
+ - [Video Result](#video-result)
14
+ - [How to use](#how-to-use)
15
+ - [Model zoo](#model-zoo)
16
+ - [TODO List](#todo-list)
17
+ - [Reference](#reference)
18
+ - [License](#license)
19
+
20
+ # Introduction
21
+ CogVideoX-Fun is a modified pipeline based on the CogVideoX structure, designed to provide more flexibility in generation. It can be used to create AI images and videos, as well as to train baseline models and Lora models for Diffusion Transformer. We support predictions directly from the already trained CogVideoX-Fun model, allowing the generation of videos at different resolutions, approximately 6 seconds long with 8 fps (1 to 49 frames). Users can also train their own baseline models and Lora models to achieve certain style transformations.
22
+
23
+ We will support quick pull-ups from different platforms, refer to [Quick Start](#quick-start).
24
+
25
+ What's New:
26
+ - Use reward backpropagation to train Lora and optimize the video, aligning it better with human preferences, detailes in [here](scripts/README_TRAIN_REWARD.md). A new version of the control model supports various conditions (e.g., Canny, Depth, Pose, MLSD, etc.). [2024.11.21]
27
+ - CogVideoX-Fun Control is now supported in diffusers. Thanks to [a-r-r-o-w](https://github.com/a-r-r-o-w) who contributed the support in this [PR](https://github.com/huggingface/diffusers/pull/9671). Check out the [docs](https://huggingface.co/docs/diffusers/main/en/api/pipelines/cogvideox) to know more. [ 2024.10.16 ]
28
+ - Retrain the i2v model and add noise to increase the motion amplitude of the video. Upload the control model training code and control model. [ 2024.09.29 ]
29
+ - Create code! Now supporting Windows and Linux. Supports 2b and 5b models. Supports video generation at any resolution from 256x256x49 to 1024x1024x49. [ 2024.09.18 ]
30
+
31
+ Function：
32
+ - [Data Preprocessing](#data-preprocess)
33
+ - [Train DiT](#dit-train)
34
+ - [Video Generation](#video-gen)
35
+
36
+ Our UI interface is as follows:
37
+ ![ui](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/ui.jpg)
38
+
39
+ # Quick Start
40
+ ### 1. Cloud usage: AliyunDSW/Docker
41
+ #### a. From AliyunDSW
42
+ DSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.
43
+
44
+ Aliyun provide free GPU time in [Freetier](https://free.aliyun.com/?product=9602825&crowd=enterprise&spm=5176.28055625.J_5831864660.1.e939154aRgha4e&scm=20140722.M_9974135.P_110.MO_1806-ID_9974135-MID_9974135-CID_30683-ST_8512-V_1), get it and use in Aliyun PAI-DSW to start CogVideoX-Fun within 5min!
45
+
46
+ [![DSW Notebook](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/dsw.png)](https://gallery.pai-ml.com/#/preview/deepLearning/cv/cogvideox_fun)
47
+
48
+ #### b. From ComfyUI
49
+ Our ComfyUI is as follows, please refer to [ComfyUI README](comfyui/README.md) for details.
50
+ ![workflow graph](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/cogvideoxfunv1_workflow_i2v.jpg)
51
+
52
+ #### c. From docker
53
+ If you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.
54
+
55
+ Then execute the following commands in this way:
56
+
57
+ ```
58
+ # pull image
59
+ docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
60
+
61
+ # enter image
62
+ docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
63
+
64
+ # clone code
65
+ git clone https://github.com/aigc-apps/CogVideoX-Fun.git
66
+
67
+ # enter CogVideoX-Fun's dir
68
+ cd CogVideoX-Fun
69
+
70
+ # download weights
71
+ mkdir models/Diffusion_Transformer
72
+ mkdir models/Personalized_Model
73
+
74
+ wget https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/Diffusion_Transformer/CogVideoX-Fun-V1.1-2b-InP.tar.gz -O models/Diffusion_Transformer/CogVideoX-Fun-V1.1-2b-InP.tar.gz
75
+
76
+ cd models/Diffusion_Transformer/
77
+ tar -xvf CogVideoX-Fun-V1.1-2b-InP.tar.gz
78
+ cd ../../
79
+ ```
80
+
81
+ ### 2. Local install: Environment Check/Downloading/Installation
82
+ #### a. Environment Check
83
+ We have verified CogVideoX-Fun execution on the following environment:
84
+
85
+ The detailed of Windows:
86
+ - OS: Windows 10
87
+ - python: python3.10 & python3.11
88
+ - pytorch: torch2.2.0
89
+ - CUDA: 11.8 & 12.1
90
+ - CUDNN: 8+
91
+ - GPU： Nvidia-3060 12G & Nvidia-3090 24G
92
+
93
+ The detailed of Linux:
94
+ - OS: Ubuntu 20.04, CentOS
95
+ - python: python3.10 & python3.11
96
+ - pytorch: torch2.2.0
97
+ - CUDA: 11.8 & 12.1
98
+ - CUDNN: 8+
99
+ - GPU：Nvidia-V100 16G & Nvidia-A10 24G & Nvidia-A100 40G & Nvidia-A100 80G
100
+
101
+ We need about 60GB available on disk (for saving weights), please check!
102
+
103
+ #### b. Weights
104
+ We'd better place the [weights](#model-zoo) along the specified path:
105
+
106
+ ```
107
+ 📦 models/
108
+ ├─�� 📂 Diffusion_Transformer/
109
+ │ ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
110
+ │ └── 📂 CogVideoX-Fun-V1.1-5b-InP/
111
+ ├── 📂 Personalized_Model/
112
+ │ └── your trained trainformer model / your trained lora model (for UI load)
113
+ ```
114
+
115
+ # Video Result
116
+ The results displayed are all based on image.
117
+
118
+ ### CogVideoX-Fun-V1.1-5B
119
+
120
+ Resolution-1024
121
+
122
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
123
+ <tr>
124
+ <td>
125
+ <video src="https://github.com/user-attachments/assets/34e7ec8f-293e-4655-bb14-5e1ee476f788" width="100%" controls autoplay loop></video>
126
+ </td>
127
+ <td>
128
+ <video src="https://github.com/user-attachments/assets/7809c64f-eb8c-48a9-8bdc-ca9261fd5434" width="100%" controls autoplay loop></video>
129
+ </td>
130
+ <td>
131
+ <video src="https://github.com/user-attachments/assets/8e76aaa4-c602-44ac-bcb4-8b24b72c386c" width="100%" controls autoplay loop></video>
132
+ </td>
133
+ <td>
134
+ <video src="https://github.com/user-attachments/assets/19dba894-7c35-4f25-b15c-384167ab3b03" width="100%" controls autoplay loop></video>
135
+ </td>
136
+ </tr>
137
+ </table>
138
+
139
+
140
+ Resolution-768
141
+
142
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
143
+ <tr>
144
+ <td>
145
+ <video src="https://github.com/user-attachments/assets/0bc339b9-455b-44fd-8917-80272d702737" width="100%" controls autoplay loop></video>
146
+ </td>
147
+ <td>
148
+ <video src="https://github.com/user-attachments/assets/70a043b9-6721-4bd9-be47-78b7ec5c27e9" width="100%" controls autoplay loop></video>
149
+ </td>
150
+ <td>
151
+ <video src="https://github.com/user-attachments/assets/d5dd6c09-14f3-40f8-8b6d-91e26519b8ac" width="100%" controls autoplay loop></video>
152
+ </td>
153
+ <td>
154
+ <video src="https://github.com/user-attachments/assets/9327e8bc-4f17-46b0-b50d-38c250a9483a" width="100%" controls autoplay loop></video>
155
+ </td>
156
+ </tr>
157
+ </table>
158
+
159
+ Resolution-512
160
+
161
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
162
+ <tr>
163
+ <td>
164
+ <video src="https://github.com/user-attachments/assets/ef407030-8062-454d-aba3-131c21e6b58c" width="100%" controls autoplay loop></video>
165
+ </td>
166
+ <td>
167
+ <video src="https://github.com/user-attachments/assets/7610f49e-38b6-4214-aa48-723ae4d1b07e" width="100%" controls autoplay loop></video>
168
+ </td>
169
+ <td>
170
+ <video src="https://github.com/user-attachments/assets/1fff0567-1e15-415c-941e-53ee8ae2c841" width="100%" controls autoplay loop></video>
171
+ </td>
172
+ <td>
173
+ <video src="https://github.com/user-attachments/assets/bcec48da-b91b-43a0-9d50-cf026e00fa4f" width="100%" controls autoplay loop></video>
174
+ </td>
175
+ </tr>
176
+ </table>
177
+
178
+ ### CogVideoX-Fun-V1.1-5B with Reward Backpropagation
179
+
180
+ <table border="0" style="width: 100%; text-align: center; margin-top: 20px;">
181
+ <thead>
182
+ <tr>
183
+ <th style="text-align: center;" width="10%">Prompt</sup></th>
184
+ <th style="text-align: center;" width="30%">CogVideoX-Fun-V1.1-5B</th>
185
+ <th style="text-align: center;" width="30%">CogVideoX-Fun-V1.1-5B <br> HPSv2.1 Reward LoRA</th>
186
+ <th style="text-align: center;" width="30%">CogVideoX-Fun-V1.1-5B <br> MPS Reward LoRA</th>
187
+ </tr>
188
+ </thead>
189
+ <tr>
190
+ <td>
191
+ Pig with wings flying above a diamond mountain
192
+ </td>
193
+ <td>
194
+ <video src="https://github.com/user-attachments/assets/6682f507-4ca2-45e9-9d76-86e2d709efb3" width="100%" controls autoplay loop></video>
195
+ </td>
196
+ <td>
197
+ <video src="https://github.com/user-attachments/assets/ec9219a2-96b3-44dd-b918-8176b2beb3b0" width="100%" controls autoplay loop></video>
198
+ </td>
199
+ <td>
200
+ <video src="https://github.com/user-attachments/assets/a75c6a6a-0b69-4448-afc0-fda3c7955ba0" width="100%" controls autoplay loop></video>
201
+ </td>
202
+ </tr>
203
+ <tr>
204
+ <td>
205
+ A dog runs through a field while a cat climbs a tree
206
+ </td>
207
+ <td>
208
+ <video src="https://github.com/user-attachments/assets/0392d632-2ec3-46b4-8867-0da1db577b6d" width="100%" controls autoplay loop></video>
209
+ </td>
210
+ <td>
211
+ <video src="https://github.com/user-attachments/assets/7d8c729d-6afb-408e-b812-67c40c3aaa96" width="100%" controls autoplay loop></video>
212
+ </td>
213
+ <td>
214
+ <video src="https://github.com/user-attachments/assets/dcd1343c-7435-4558-b602-9c0fa08cbd59" width="100%" controls autoplay loop></video>
215
+ </td>
216
+ </tr>
217
+ </table>
218
+
219
+ ### CogVideoX-Fun-V1.1-5B-Control
220
+
221
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
222
+ <tr>
223
+ <td>
224
+ <video src="https://github.com/user-attachments/assets/53002ce2-dd18-4d4f-8135-b6f68364cabd" width="100%" controls autoplay loop></video>
225
+ </td>
226
+ <td>
227
+ <video src="https://github.com/user-attachments/assets/fce43c0b-81fa-4ab2-9ca7-78d786f520e6" width="100%" controls autoplay loop></video>
228
+ </td>
229
+ <td>
230
+ <video src="https://github.com/user-attachments/assets/b208b92c-5add-4ece-a200-3dbbe47b93c3" width="100%" controls autoplay loop></video>
231
+ </td>
232
+ <tr>
233
+ <td>
234
+ A young woman with beautiful clear eyes and blonde hair, wearing white clothes and twisting her body, with the camera focused on her face. High quality, masterpiece, best quality, high resolution, ultra-fine, dreamlike.
235
+ </td>
236
+ <td>
237
+ A young woman with beautiful clear eyes and blonde hair, wearing white clothes and twisting her body, with the camera focused on her face. High quality, masterpiece, best quality, high resolution, ultra-fine, dreamlike.
238
+ </td>
239
+ <td>
240
+ A young bear.
241
+ </td>
242
+ </tr>
243
+ <tr>
244
+ <td>
245
+ <video src="https://github.com/user-attachments/assets/ea908454-684b-4d60-b562-3db229a250a9" width="100%" controls autoplay loop></video>
246
+ </td>
247
+ <td>
248
+ <video src="https://github.com/user-attachments/assets/ffb7c6fc-8b69-453b-8aad-70dfae3899b9" width="100%" controls autoplay loop></video>
249
+ </td>
250
+ <td>
251
+ <video src="https://github.com/user-attachments/assets/d3f757a3-3551-4dcb-9372-7a61469813f5" width="100%" controls autoplay loop></video>
252
+ </td>
253
+ </tr>
254
+ </table>
255
+
256
+ ### CogVideoX-Fun-V1.1-5B-Pose
257
+
258
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
259
+ <tr>
260
+ <td>
261
+ Resolution-512
262
+ </td>
263
+ <td>
264
+ Resolution-768
265
+ </td>
266
+ <td>
267
+ Resolution-1024
268
+ </td>
269
+ <tr>
270
+ <td>
271
+ <video src="https://github.com/user-attachments/assets/a746df51-9eb7-4446-bee5-2ee30285c143" width="100%" controls autoplay loop></video>
272
+ </td>
273
+ <td>
274
+ <video src="https://github.com/user-attachments/assets/db295245-e6aa-43be-8c81-32cb411f1473" width="100%" controls autoplay loop></video>
275
+ </td>
276
+ <td>
277
+ <video src="https://github.com/user-attachments/assets/ec9875b2-fde0-48e1-ab7e-490cee51ef40" width="100%" controls autoplay loop></video>
278
+ </td>
279
+ </tr>
280
+ </table>
281
+
282
+ ### CogVideoX-Fun-V1.1-2B
283
+
284
+ Resolution-768
285
+
286
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
287
+ <tr>
288
+ <td>
289
+ <video src="https://github.com/user-attachments/assets/03235dea-980e-4fc5-9c41-e40a5bc1b6d0" width="100%" controls autoplay loop></video>
290
+ </td>
291
+ <td>
292
+ <video src="https://github.com/user-attachments/assets/f7302648-5017-47db-bdeb-4d893e620b37" width="100%" controls autoplay loop></video>
293
+ </td>
294
+ <td>
295
+ <video src="https://github.com/user-attachments/assets/cbadf411-28fa-4b87-813d-da63ff481904" width="100%" controls autoplay loop></video>
296
+ </td>
297
+ <td>
298
+ <video src="https://github.com/user-attachments/assets/87cc9d0b-b6fe-4d2d-b447-174513d169ab" width="100%" controls autoplay loop></video>
299
+ </td>
300
+ </tr>
301
+ </table>
302
+
303
+ ### CogVideoX-Fun-V1.1-2B-Pose
304
+
305
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
306
+ <tr>
307
+ <td>
308
+ Resolution-512
309
+ </td>
310
+ <td>
311
+ Resolution-768
312
+ </td>
313
+ <td>
314
+ Resolution-1024
315
+ </td>
316
+ <tr>
317
+ <td>
318
+ <video src="https://github.com/user-attachments/assets/487bcd7b-1b7f-4bb4-95b5-96a6b6548b3e" width="100%" controls autoplay loop></video>
319
+ </td>
320
+ <td>
321
+ <video src="https://github.com/user-attachments/assets/2710fd18-8489-46e4-8086-c237309ae7f6" width="100%" controls autoplay loop></video>
322
+ </td>
323
+ <td>
324
+ <video src="https://github.com/user-attachments/assets/b79513db-7747-4512-b86c-94f9ca447fe2" width="100%" controls autoplay loop></video>
325
+ </td>
326
+ </tr>
327
+ </table>
328
+
329
+ # How to use
330
+
331
+ <h3 id="video-gen">1. Inference </h3>
332
+
333
+ #### a. Using Python Code
334
+ - Step 1: Download the corresponding [weights](#model-zoo) and place them in the models folder.
335
+ - Step 2: Modify prompt, neg_prompt, guidance_scale, and seed in the predict_t2v.py file.
336
+ - Step 3: Run the predict_t2v.py file, wait for the generated results, and save the results in the samples/cogvideox-fun-videos-t2v folder.
337
+ - Step 4: If you want to combine other backbones you have trained with Lora, modify the predict_t2v.py and Lora_path in predict_t2v.py depending on the situation.
338
+
339
+ #### b. Using webui
340
+ - Step 1: Download the corresponding [weights](#model-zoo) and place them in the models folder.
341
+ - Step 2: Run the app.py file to enter the graph page.
342
+ - Step 3: Select the generated model based on the page, fill in prompt, neg_prompt, guidance_scale, and seed, click on generate, wait for the generated result, and save the result in the samples folder.
343
+
344
+ #### c. From ComfyUI
345
+ Please refer to [ComfyUI README](comfyui/README.md) for details.
346
+
347
+ ### 2. Model Training
348
+ A complete CogVideoX-Fun training pipeline should include data preprocessing, and Video DiT training.
349
+
350
+ <h4 id="data-preprocess">a. data preprocessing</h4>
351
+
352
+ We have provided a simple demo of training the Lora model through image data, which can be found in the [wiki](https://github.com/aigc-apps/CogVideoX-Fun/wiki/Training-Lora) for details.
353
+
354
+ A complete data preprocessing link for long video segmentation, cleaning, and description can refer to [README](cogvideox/video_caption/README.md) in the video captions section.
355
+
356
+ If you want to train a text to image and video generation model. You need to arrange the dataset in this format.
357
+
358
+ ```
359
+ 📦 project/
360
+ ├── 📂 datasets/
361
+ │ ├── 📂 internal_datasets/
362
+ │ ├── 📂 train/
363
+ │ │ ├── 📄 00000001.mp4
364
+ │ │ ├── 📄 00000002.jpg
365
+ │ │ └── 📄 .....
366
+ │ └── 📄 json_of_internal_datasets.json
367
+ ```
368
+
369
+ The json_of_internal_datasets.json is a standard JSON file. The file_path in the json can to be set as relative path, as shown in below:
370
+ ```json
371
+ [
372
+ {
373
+ "file_path": "train/00000001.mp4",
374
+ "text": "A group of young men in suits and sunglasses are walking down a city street.",
375
+ "type": "video"
376
+ },
377
+ {
378
+ "file_path": "train/00000002.jpg",
379
+ "text": "A group of young men in suits and sunglasses are walking down a city street.",
380
+ "type": "image"
381
+ },
382
+ .....
383
+ ]
384
+ ```
385
+
386
+ You can also set the path as absolute path as follow:
387
+ ```json
388
+ [
389
+ {
390
+ "file_path": "/mnt/data/videos/00000001.mp4",
391
+ "text": "A group of young men in suits and sunglasses are walking down a city street.",
392
+ "type": "video"
393
+ },
394
+ {
395
+ "file_path": "/mnt/data/train/00000001.jpg",
396
+ "text": "A group of young men in suits and sunglasses are walking down a city street.",
397
+ "type": "image"
398
+ },
399
+ .....
400
+ ]
401
+ ```
402
+
403
+ <h4 id="dit-train">b. Video DiT training </h4>
404
+
405
+ If the data format is relative path during data preprocessing, please set ```scripts/train.sh``` as follow.
406
+ ```
407
+ export DATASET_NAME="datasets/internal_datasets/"
408
+ export DATASET_META_NAME="datasets/internal_datasets/json_of_internal_datasets.json"
409
+ ```
410
+
411
+ If the data format is absolute path during data preprocessing, please set ```scripts/train.sh``` as follow.
412
+ ```
413
+ export DATASET_NAME=""
414
+ export DATASET_META_NAME="/mnt/data/json_of_internal_datasets.json"
415
+ ```
416
+
417
+ Then, we run scripts/train.sh.
418
+ ```sh
419
+ sh scripts/train.sh
420
+ ```
421
+
422
+ For details on setting some parameters, please refer to [Readme Train](scripts/README_TRAIN.md), [Readme Lora](scripts/README_TRAIN_LORA.md) and [Readme Control](scripts/README_TRAIN_CONTROL.md).
423
+
424
+
425
+ # Model zoo
426
+
427
+ V1.5:
428
+
429
+ | Name | Storage Space | Hugging Face | Model Scope | Description |
430
+ |--|--|--|--|--|
431
+ | CogVideoX-Fun-V1.5-5b-InP | 20.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.5-5b-InP) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.5-5b-InP) | Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024) and has been trained on 85 frames at a rate of 8 frames per second. |
432
+
433
+ V1.1:
434
+
435
+ | Name | Storage Space | Hugging Face | Model Scope | Description |
436
+ |--|--|--|--|--|
437
+ | CogVideoX-Fun-V1.1-2b-InP | 13.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-2b-InP) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-2b-InP) | Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second. |
438
+ | CogVideoX-Fun-V1.1-5b-InP | 20.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-InP) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-InP) | Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second. Noise has been added to the reference image, and the amplitude of motion is greater compared to V1.0. |
439
+ | CogVideoX-Fun-V1.1-2b-Pose | 13.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-2b-Pose) | Our official pose-control video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second.|
440
+ | CogVideoX-Fun-V1.1-2b-Control | 13.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-2b-Control) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-2b-Control) | Our official control video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second. Supporting various control conditions such as Canny, Depth, Pose, MLSD, etc.|
441
+ | CogVideoX-Fun-V1.1-5b-Pose | 20.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-Pose) | Our official pose-control video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second.|
442
+ | CogVideoX-Fun-V1.1-5b-Control | 20.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-Control) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-Control) | Our official control video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second. Supporting various control conditions such as Canny, Depth, Pose, MLSD, etc.|
443
+ | CogVideoX-Fun-V1.1-Reward-LoRAs | - | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-Reward-LoRAs) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-Reward-LoRAs) | The official reward backpropagation technology model optimizes the videos generated by CogVideoX-Fun-V1.1 to better match human preferences. ｜
444
+
445
+ V1.0:
446
+
447
+ | Name | Storage Space | Hugging Face | Model Scope | Description |
448
+ |--|--|--|--|--|
449
+ | CogVideoX-Fun-2b-InP | 13.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-2b-InP) | [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-2b-InP) | Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second. |
450
+ | CogVideoX-Fun-5b-InP | 20.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/CogVideoX-Fun-5b-InP)| [😄Link](https://modelscope.cn/models/PAI/CogVideoX-Fun-5b-InP)| Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 49 frames at a rate of 8 frames per second. |
451
+
452
+ # TODO List
453
+ - Support Chinese.
454
+
455
+ # Reference
456
+ - CogVideo: https://github.com/THUDM/CogVideo/
457
+ - EasyAnimate: https://github.com/aigc-apps/EasyAnimate
458
+
459
+ # License
460
+ This project is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).
461
+
462
+ The CogVideoX-2B model (including its corresponding Transformers module and VAE module) is released under the [Apache 2.0 License](LICENSE).
463
+
464
+ The CogVideoX-5B model (Transformers module) is released under the [CogVideoX LICENSE](https://huggingface.co/THUDM/CogVideoX-5b/blob/main/LICENSE).
configuration.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"framework":"Pytorch","task":"text-to-video-synthesis"}
gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
model_index.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "CogVideoXPipeline",
3
+ "_diffusers_version": "0.32.0.dev0",
4
+ "scheduler": [
5
+ "diffusers",
6
+ "CogVideoXDDIMScheduler"
7
+ ],
8
+ "text_encoder": [
9
+ "transformers",
10
+ "T5EncoderModel"
11
+ ],
12
+ "tokenizer": [
13
+ "transformers",
14
+ "T5Tokenizer"
15
+ ],
16
+ "transformer": [
17
+ "diffusers",
18
+ "CogVideoXTransformer3DModel"
19
+ ],
20
+ "vae": [
21
+ "diffusers",
22
+ "AutoencoderKLCogVideoX"
23
+ ]
24
+ }