title: LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation
summary: With the impressive progress in diffusion-based text-to-image generation,
  extending such powerful generative ability to text-to-video raises enormous
  attention. Existing methods either require large-scale text-video pairs and a
  large number of training resources or learn motions that are precisely aligned
  with template videos. It is non-trivial to balance a trade-off between the
  degree of generation freedom and the resource costs for video generation. In
  our study, we present a few-shot-based tuning framework, LAMP, which enables
  text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos
  on a single GPU. Specifically, we design a first-frame-conditioned pipeline
  that uses an off-the-shelf text-to-image model for content generation so that
  our tuned video diffusion model mainly focuses on motion learning. The
  well-developed text-to-image techniques can provide visually pleasing and
  diverse content as generation conditions, which highly improves video quality
  and generation freedom. To capture the features of temporal dimension, we
  expand the pretrained 2D convolution layers of the T2I model to our novel
  temporal-spatial motion learning layers and modify the attention blocks to the
  temporal level. Additionally, we develop an effective inference trick,
  shared-noise sampling, which can improve the stability of videos with
  computational costs. Our method can also be flexibly applied to other tasks,
  e.g. real-world image animation and video editing. Extensive experiments
  demonstrate that LAMP can effectively learn the motion pattern on limited data
  and generate high-quality videos. The code and models are available at
  https://rq-wu.github.io/projects/LAMP.
paper_authors._id: 652f52300af36fdc8745d25e
paper_authors.name: Ruiqi Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-19T08:10:21.057Z
paper_authors._id: 6530c9a5f4c60acf0cc9259a
paper_authors.avatarUrl: /avatars/a3f5452accdf62c6a0855878f0f82852.svg
paper_authors.fullname: Ruiqi Wu
paper_authors.type: user
paper_authors.user: wuruiqi0722
paper_authors._id: 652f52300af36fdc8745d25f
paper_authors.name: Liangyu Chen
paper_authors._id: 652f52300af36fdc8745d260
paper_authors.name: Tong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:24:50.109Z
paper_authors._id: 643d901e37ca9134b5888c51
paper_authors.avatarUrl: /avatars/a2f4a7959b2effed04fb193cee405da7.svg
paper_authors.fullname: Tony Yang
paper_authors.type: user
paper_authors.user: Tongyang
paper_authors._id: 652f52300af36fdc8745d261
paper_authors.name: Chunle Guo
paper_authors._id: 652f52300af36fdc8745d262
paper_authors.name: Chongyi Li
paper_authors._id: 652f52300af36fdc8745d263
paper_authors.name: Xiangyu Zhang
paper_title: LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation
paper_summary: With the impressive progress in diffusion-based text-to-image generation,
  extending such powerful generative ability to text-to-video raises enormous
  attention. Existing methods either require large-scale text-video pairs and a
  large number of training resources or learn motions that are precisely aligned
  with template videos. It is non-trivial to balance a trade-off between the
  degree of generation freedom and the resource costs for video generation. In
  our study, we present a few-shot-based tuning framework, LAMP, which enables
  text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos
  on a single GPU. Specifically, we design a first-frame-conditioned pipeline
  that uses an off-the-shelf text-to-image model for content generation so that
  our tuned video diffusion model mainly focuses on motion learning. The
  well-developed text-to-image techniques can provide visually pleasing and
  diverse content as generation conditions, which highly improves video quality
  and generation freedom. To capture the features of temporal dimension, we
  expand the pretrained 2D convolution layers of the T2I model to our novel
  temporal-spatial motion learning layers and modify the attention blocks to the
  temporal level. Additionally, we develop an effective inference trick,
  shared-noise sampling, which can improve the stability of videos with
  computational costs. Our method can also be flexibly applied to other tasks,
  e.g. real-world image animation and video editing. Extensive experiments
  demonstrate that LAMP can effectively learn the motion pattern on limited data
  and generate high-quality videos. The code and models are available at
  https://rq-wu.github.io/projects/LAMP.
paper_ai_keywords: diffusion-based text-to-image generation few-shot-based tuning framework LAMP text-to-image diffusion model first-frame-conditioned pipeline off-the-shelf text-to-image model content generation pretrained 2D convolution layers temporal-spatial motion learning layers attention blocks temporal level shared-noise sampling real-world image generation video editing motion pattern learning high-quality video generation