title: Make-Your-Video: Customized Video Generation Using Textual and
    Structural Guidance
summary: Creating a vivid video from the event or scenario in our imagination is a
  truly fascinating experience. Recent advancements in text-to-video synthesis
  have unveiled the potential to achieve this with prompts only. While text is
  convenient in conveying the overall scene context, it may be insufficient to
  control precisely. In this paper, we explore customized video generation by
  utilizing text as context description and motion structure (e.g. frame-wise
  depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
  joint-conditional video generation using a Latent Diffusion Model that is
  pre-trained for still image synthesis and then promoted for video generation
  with the introduction of temporal modules. This two-stage learning scheme not
  only reduces the computing resources required, but also improves the
  performance by transferring the rich concepts available in image datasets
  solely into video generation. Moreover, we use a simple yet effective causal
  attention mask strategy to enable longer video synthesis, which mitigates the
  potential quality degradation effectively. Experimental results show the
  superiority of our method over existing baselines, particularly in terms of
  temporal coherence and fidelity to users' guidance. In addition, our model
  enables several intriguing applications that demonstrate potential for
  practical usage.
paper_authors._id: 64794f32a68454566355d7ce
paper_authors.name: Jinbo Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T12:46:24.334Z
paper_authors._id: 64770e86d7cf39f2e937ae9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
paper_authors.fullname: Jinbo Xing
paper_authors.type: user
paper_authors.user: Doubiiu
paper_authors._id: 64794f32a68454566355d7cf
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T11:03:28.227Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 64794f32a68454566355d7d0
paper_authors.name: Yuxin Liu
paper_authors._id: 64794f32a68454566355d7d1
paper_authors.name: Yuechen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T11:57:52.969Z
paper_authors._id: 6418554a0956be7233a1023e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
paper_authors.fullname: zhang yuechen
paper_authors.type: user
paper_authors.user: julianjuaner
paper_authors._id: 64794f32a68454566355d7d2
paper_authors.name: Yong Zhang
paper_authors._id: 64794f32a68454566355d7d3
paper_authors.name: Yingqing He
paper_authors._id: 64794f32a68454566355d7d4
paper_authors.name: Hanyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T11:59:05.215Z
paper_authors._id: 641297a1ac08ffb70793fd04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641297a1ac08ffb70793fd04/3XVQYehMCGwIJvqqcQ3Gf.png
paper_authors.fullname: Hanyuan Liu
paper_authors.type: user
paper_authors.user: hyliu
paper_authors._id: 64794f32a68454566355d7d5
paper_authors.name: Haoxin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:24:16.621Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 64794f32a68454566355d7d6
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T12:00:24.093Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 64794f32a68454566355d7d7
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T12:00:56.785Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 64794f32a68454566355d7d8
paper_authors.name: Ying Shan
paper_authors._id: 64794f32a68454566355d7d9
paper_authors.name: Tien-Tsin Wong
paper_title: Make-Your-Video: Customized Video Generation Using Textual and
    Structural Guidance
paper_summary: Creating a vivid video from the event or scenario in our imagination is a
  truly fascinating experience. Recent advancements in text-to-video synthesis
  have unveiled the potential to achieve this with prompts only. While text is
  convenient in conveying the overall scene context, it may be insufficient to
  control precisely. In this paper, we explore customized video generation by
  utilizing text as context description and motion structure (e.g. frame-wise
  depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
  joint-conditional video generation using a Latent Diffusion Model that is
  pre-trained for still image synthesis and then promoted for video generation
  with the introduction of temporal modules. This two-stage learning scheme not
  only reduces the computing resources required, but also improves the
  performance by transferring the rich concepts available in image datasets
  solely into video generation. Moreover, we use a simple yet effective causal
  attention mask strategy to enable longer video synthesis, which mitigates the
  potential quality degradation effectively. Experimental results show the
  superiority of our method over existing baselines, particularly in terms of
  temporal coherence and fidelity to users' guidance. In addition, our model
  enables several intriguing applications that demonstrate potential for
  practical usage.
paper_ai_summary: Make-Your-Video generates high-quality videos by combining text descriptions with motion structure using a Latent Diffusion Model with temporal modules, achieving better performance and temporal coherence than existing methods.
paper_ai_keywords: Latent Diffusion Model joint-conditional video generation pre-trained temporal modules causal attention mask