title: Dual-Stream Diffusion Net for Text-to-Video Generation
summary: With the emerging diffusion models, recently, text-to-video generation has
  aroused increasing attention. But an important bottleneck therein is that
  generative videos often tend to carry some flickers and artifacts. In this
  work, we propose a dual-stream diffusion net (DSDN) to improve the consistency
  of content variations in generating videos. In particular, the designed two
  diffusion streams, video content and motion branches, could not only run
  separately in their private spaces for producing personalized video variations
  as well as content, but also be well-aligned between the content and motion
  domains through leveraging our designed cross-transformer interaction module,
  which would benefit the smoothness of generated videos. Besides, we also
  introduce motion decomposer and combiner to faciliate the operation on video
  motion. Qualitative and quantitative experiments demonstrate that our method
  could produce amazing continuous videos with fewer flickers.
paper_authors._id: 64ddc0c04d6048abd76ecb9b
paper_authors.name: Binhui Liu
paper_authors._id: 64ddc0c04d6048abd76ecb9c
paper_authors.name: Xin Liu
paper_authors._id: 64ddc0c04d6048abd76ecb9d
paper_authors.name: Anbo Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-17T14:08:16.806Z
paper_authors._id: 628ba40d488ff0ea1c71d3cd
paper_authors.avatarUrl: /avatars/b6b99150dca9006aef54516347d06166.svg
paper_authors.fullname: daianbo
paper_authors.type: user
paper_authors.user: daianbo
paper_authors._id: 64ddc0c04d6048abd76ecb9e
paper_authors.name: Zhiyong Zeng
paper_authors._id: 64ddc0c04d6048abd76ecb9f
paper_authors.name: Zhen Cui
paper_authors._id: 64ddc0c04d6048abd76ecba0
paper_authors.name: Jian Yang
paper_title: Dual-Stream Diffusion Net for Text-to-Video Generation
paper_summary: With the emerging diffusion models, recently, text-to-video generation has
  aroused increasing attention. But an important bottleneck therein is that
  generative videos often tend to carry some flickers and artifacts. In this
  work, we propose a dual-stream diffusion net (DSDN) to improve the consistency
  of content variations in generating videos. In particular, the designed two
  diffusion streams, video content and motion branches, could not only run
  separately in their private spaces for producing personalized video variations
  as well as content, but also be well-aligned between the content and motion
  domains through leveraging our designed cross-transformer interaction module,
  which would benefit the smoothness of generated videos. Besides, we also
  introduce motion decomposer and combiner to faciliate the operation on video
  motion. Qualitative and quantitative experiments demonstrate that our method
  could produce amazing continuous videos with fewer flickers.
paper_ai_keywords: diffusion models text-to-video generation flickers artifacts dual-stream diffusion net (DSDN) diffusion streams video content motion branches private spaces personalized video variations content well-aligned content and motion domains cross-transformer interaction module smoothness generated videos motion decomposer motion combiner qualitative experiments quantitative experiments continuous videos