title: StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video
    Generation
summary: For recent diffusion-based generative models, maintaining consistent content
  across a series of generated images, especially those containing subjects and
  complex details, presents a significant challenge. In this paper, we propose a
  new way of self-attention calculation, termed Consistent Self-Attention, that
  significantly boosts the consistency between the generated images and augments
  prevalent pretrained diffusion-based text-to-image models in a zero-shot
  manner. To extend our method to long-range video generation, we further
  introduce a novel semantic space temporal motion prediction module, named
  Semantic Motion Predictor. It is trained to estimate the motion conditions
  between two provided images in the semantic spaces. This module converts the
  generated sequence of images into videos with smooth transitions and consistent
  subjects that are significantly more stable than the modules based on latent
  spaces only, especially in the context of long video generation. By merging
  these two novel components, our framework, referred to as StoryDiffusion, can
  describe a text-based story with consistent images or videos encompassing a
  rich variety of contents. The proposed StoryDiffusion encompasses pioneering
  explorations in visual story generation with the presentation of images and
  videos, which we hope could inspire more research from the aspect of
  architectural modifications. Our code is made publicly available at
  https://github.com/HVision-NKU/StoryDiffusion.
paper_authors._id: 663443af2adb91d8d9fa2b9a
paper_authors.name: Yupeng Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:39:20.759Z
paper_authors._id: 64b54d5799ba6b130245da20
paper_authors.avatarUrl: /avatars/c5281b17a4d4a565d6d189a007ac934e.svg
paper_authors.fullname: zhou
paper_authors.type: user
paper_authors.user: YupengZhou
paper_authors._id: 663443af2adb91d8d9fa2b9b
paper_authors.name: Daquan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:39:31.405Z
paper_authors._id: 63fe0b160c1bbe8e29d2dd32
paper_authors.avatarUrl: /avatars/bc574036287170a77057893efaa48e2d.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: DaQuan21
paper_authors._id: 663443af2adb91d8d9fa2b9c
paper_authors.name: Ming-Ming Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:39:51.152Z
paper_authors._id: 64e496ae0195913c7fa91c66
paper_authors.avatarUrl: /avatars/23f274f0a3b4ef6ded35205df9bfb564.svg
paper_authors.fullname: chengmingming
paper_authors.type: user
paper_authors.user: mingming8688
paper_authors._id: 663443af2adb91d8d9fa2b9d
paper_authors.name: Jiashi Feng
paper_authors._id: 663443af2adb91d8d9fa2b9e
paper_authors.name: Qibin Hou
paper_title: StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video
    Generation
paper_summary: For recent diffusion-based generative models, maintaining consistent content
  across a series of generated images, especially those containing subjects and
  complex details, presents a significant challenge. In this paper, we propose a
  new way of self-attention calculation, termed Consistent Self-Attention, that
  significantly boosts the consistency between the generated images and augments
  prevalent pretrained diffusion-based text-to-image models in a zero-shot
  manner. To extend our method to long-range video generation, we further
  introduce a novel semantic space temporal motion prediction module, named
  Semantic Motion Predictor. It is trained to estimate the motion conditions
  between two provided images in the semantic spaces. This module converts the
  generated sequence of images into videos with smooth transitions and consistent
  subjects that are significantly more stable than the modules based on latent
  spaces only, especially in the context of long video generation. By merging
  these two novel components, our framework, referred to as StoryDiffusion, can
  describe a text-based story with consistent images or videos encompassing a
  rich variety of contents. The proposed StoryDiffusion encompasses pioneering
  explorations in visual story generation with the presentation of images and
  videos, which we hope could inspire more research from the aspect of
  architectural modifications. Our code is made publicly available at
  https://github.com/HVision-NKU/StoryDiffusion.
paper_ai_summary: StoryDiffusion, combining consistent self-attention and semantic motion prediction, enables generation of coherent and stable images and videos from textual descriptions.
paper_ai_keywords: diffusion-based generative models self-attention Consistent Self-Attention latent spaces Semantic Motion Predictor visual story generation