title: Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive
    Content Generation
summary: This paper explores higher-resolution video outpainting with extensive
  content generation. We point out common issues faced by existing methods when
  attempting to largely outpaint videos: the generation of low-quality content
  and limitations imposed by GPU memory. To address these challenges, we propose
  a diffusion-based method called Follow-Your-Canvas. It builds upon two
  core designs. First, instead of employing the common practice of "single-shot"
  outpainting, we distribute the task across spatial windows and seamlessly merge
  them. It allows us to outpaint videos of any size and resolution without being
  constrained by GPU memory. Second, the source video and its relative positional
  relation are injected into the generation process of each window. It makes the
  generated spatial layout within each window harmonize with the source video.
  Coupling with these two designs enables us to generate higher-resolution
  outpainting videos with rich content while keeping spatial and temporal
  consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g.,
  from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically
  pleasing results. It achieves the best quantitative results across various
  resolution and scale setups. The code is released on
  https://github.com/mayuelala/FollowYourCanvas
paper_authors._id: 66d8050d0aaebff5ea43bcda
paper_authors.name: Qihua Chen
paper_authors._id: 66d8050d0aaebff5ea43bcdb
paper_authors.name: Yue Ma
paper_authors._id: 66d8050d0aaebff5ea43bcdc
paper_authors.name: Hongfa Wang
paper_authors._id: 66d8050d0aaebff5ea43bcdd
paper_authors.name: Junkun Yuan
paper_authors._id: 66d8050d0aaebff5ea43bcde
paper_authors.name: Wenzhe Zhao
paper_authors._id: 66d8050d0aaebff5ea43bcdf
paper_authors.name: Qi Tian
paper_authors._id: 66d8050d0aaebff5ea43bce0
paper_authors.name: Hongmei Wang
paper_authors._id: 66d8050d0aaebff5ea43bce1
paper_authors.name: Shaobo Min
paper_authors._id: 66d8050d0aaebff5ea43bce2
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T09:50:21.697Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 66d8050d0aaebff5ea43bce3
paper_authors.name: Wei Liu
paper_title: Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive
    Content Generation
paper_summary: This paper explores higher-resolution video outpainting with extensive
  content generation. We point out common issues faced by existing methods when
  attempting to largely outpaint videos: the generation of low-quality content
  and limitations imposed by GPU memory. To address these challenges, we propose
  a diffusion-based method called Follow-Your-Canvas. It builds upon two
  core designs. First, instead of employing the common practice of "single-shot"
  outpainting, we distribute the task across spatial windows and seamlessly merge
  them. It allows us to outpaint videos of any size and resolution without being
  constrained by GPU memory. Second, the source video and its relative positional
  relation are injected into the generation process of each window. It makes the
  generated spatial layout within each window harmonize with the source video.
  Coupling with these two designs enables us to generate higher-resolution
  outpainting videos with rich content while keeping spatial and temporal
  consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g.,
  from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically
  pleasing results. It achieves the best quantitative results across various
  resolution and scale setups. The code is released on
  https://github.com/mayuelala/FollowYourCanvas
paper_ai_summary: A diffusion-based method called Follow-Your-Canvas addresses low-quality content generation and GPU memory limitations in video outpainting by distributing the task across spatial windows and considering relative positional relation with the source video.
paper_ai_keywords: diffusion-based method Follow-Your-Canvas spatial windows GPU memory source video relative positional relation spatial consistency temporal consistency video outpainting