title: Hierarchical Patch Diffusion Models for High-Resolution Video Generation
summary: Diffusion models have demonstrated remarkable performance in image and video
  synthesis. However, scaling them to high-resolution inputs is challenging and
  requires restructuring the diffusion pipeline into multiple independent
  components, limiting scalability and complicating downstream applications. This
  makes it very efficient during training and unlocks end-to-end optimization on
  high-resolution videos. We improve PDMs in two principled ways. First, to
  enforce consistency between patches, we develop deep context fusion -- an
  architectural technique that propagates the context information from low-scale
  to high-scale patches in a hierarchical manner. Second, to accelerate training
  and inference, we propose adaptive computation, which allocates more network
  capacity and computation towards coarse image details. The resulting model sets
  a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in
  class-conditional video generation on UCF-101 256^2, surpassing recent
  methods by more than 100%. Then, we show that it can be rapidly fine-tuned from
  a base 36times 64 low-resolution generator for high-resolution 64 times
  288 times 512 text-to-video synthesis. To the best of our knowledge, our
  model is the first diffusion-based architecture which is trained on such high
  resolutions entirely end-to-end. Project webpage:
  https://snap-research.github.io/hpdm.
paper_authors._id: 666a53d7372850ab364c4034
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:17:42.648Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 666a53d7372850ab364c4035
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:17:48.025Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 666a53d7372850ab364c4036
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:17:54.235Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 666a53d7372850ab364c4037
paper_authors.name: Sergey Tulyakov
paper_title: Hierarchical Patch Diffusion Models for High-Resolution Video Generation
paper_summary: Diffusion models have demonstrated remarkable performance in image and video
  synthesis. However, scaling them to high-resolution inputs is challenging and
  requires restructuring the diffusion pipeline into multiple independent
  components, limiting scalability and complicating downstream applications. This
  makes it very efficient during training and unlocks end-to-end optimization on
  high-resolution videos. We improve PDMs in two principled ways. First, to
  enforce consistency between patches, we develop deep context fusion -- an
  architectural technique that propagates the context information from low-scale
  to high-scale patches in a hierarchical manner. Second, to accelerate training
  and inference, we propose adaptive computation, which allocates more network
  capacity and computation towards coarse image details. The resulting model sets
  a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in
  class-conditional video generation on UCF-101 256^2, surpassing recent
  methods by more than 100%. Then, we show that it can be rapidly fine-tuned from
  a base 36times 64 low-resolution generator for high-resolution 64 times
  288 times 512 text-to-video synthesis. To the best of our knowledge, our
  model is the first diffusion-based architecture which is trained on such high
  resolutions entirely end-to-end. Project webpage:
  https://snap-research.github.io/hpdm.
paper_ai_keywords: diffusion models image super-resolution video synthesis diffusion pipeline deep context fusion hierarchical propagation adaptive computation FVD score Inception Score class-conditional video generation text-to-video synthesis end-to-end training