title: AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models
    without Specific Tuning
summary: With the advance of text-to-image models (e.g., Stable Diffusion) and
  corresponding personalization techniques such as DreamBooth and LoRA, everyone
  can manifest their imagination into high-quality images at an affordable cost.
  Subsequently, there is a great demand for image animation techniques to further
  combine generated static images with motion dynamics. In this report, we
  propose a practical framework to animate most of the existing personalized
  text-to-image models once and for all, saving efforts in model-specific tuning.
  At the core of the proposed framework is to insert a newly initialized motion
  modeling module into the frozen text-to-image model and train it on video clips
  to distill reasonable motion priors. Once trained, by simply injecting this
  motion modeling module, all personalized versions derived from the same base
  T2I readily become text-driven models that produce diverse and personalized
  animated images. We conduct our evaluation on several public representative
  personalized text-to-image models across anime pictures and realistic
  photographs, and demonstrate that our proposed framework helps these models
  generate temporally smooth animation clips while preserving the domain and
  diversity of their outputs. Code and pre-trained weights will be publicly
  available at https://animatediff.github.io/ .
paper_authors._id: 64accd7cb2f83dec96103610
paper_authors.name: Yuwei Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T07:29:21.009Z
paper_authors._id: 6371f83d5ffa922d638ef486
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
paper_authors.fullname: Yuwei Guo
paper_authors.type: user
paper_authors.user: guoyww
paper_authors._id: 64accd7cb2f83dec96103611
paper_authors.name: Ceyuan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-11T03:45:02.111Z
paper_authors._id: 64accf3fc1da7c4dbc69392d
paper_authors.avatarUrl: /avatars/ec8ad2498f8a49a7a00c77a7f70d34bb.svg
paper_authors.fullname: Ceyuan
paper_authors.type: user
paper_authors.user: Ceyuan
paper_authors._id: 64accd7cb2f83dec96103612
paper_authors.name: Anyi Rao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:48:56.805Z
paper_authors._id: 63f8130749569335b679af62
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f8130749569335b679af62/vgTu23-y0UKocwAGqNMwT.jpeg
paper_authors.fullname: Anyi Rao
paper_authors.type: user
paper_authors.user: anyirao
paper_authors._id: 64accd7cb2f83dec96103613
paper_authors.name: Yaohui Wang
paper_authors._id: 64accd7cb2f83dec96103614
paper_authors.name: Yu Qiao
paper_authors._id: 64accd7cb2f83dec96103615
paper_authors.name: Dahua Lin
paper_authors._id: 64accd7cb2f83dec96103616
paper_authors.name: Bo Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T07:29:15.798Z
paper_authors._id: 635f93577c05eb9f59966209
paper_authors.avatarUrl: /avatars/add48d7e3790a6b500d6c451ef8b0f75.svg
paper_authors.fullname: Intelligent Digital Creation
paper_authors.type: user
paper_authors.user: BoDai
paper_title: AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models
    without Specific Tuning
paper_summary: With the advance of text-to-image models (e.g., Stable Diffusion) and
  corresponding personalization techniques such as DreamBooth and LoRA, everyone
  can manifest their imagination into high-quality images at an affordable cost.
  Subsequently, there is a great demand for image animation techniques to further
  combine generated static images with motion dynamics. In this report, we
  propose a practical framework to animate most of the existing personalized
  text-to-image models once and for all, saving efforts in model-specific tuning.
  At the core of the proposed framework is to insert a newly initialized motion
  modeling module into the frozen text-to-image model and train it on video clips
  to distill reasonable motion priors. Once trained, by simply injecting this
  motion modeling module, all personalized versions derived from the same base
  T2I readily become text-driven models that produce diverse and personalized
  animated images. We conduct our evaluation on several public representative
  personalized text-to-image models across anime pictures and realistic
  photographs, and demonstrate that our proposed framework helps these models
  generate temporally smooth animation clips while preserving the domain and
  diversity of their outputs. Code and pre-trained weights will be publicly
  available at https://animatediff.github.io/ .
paper_ai_summary: A framework inserts a motion model into existing text-to-image models to enable animation, using video clips to learn motion priors while preserving model domain and diversity.
paper_ai_keywords: text-to-image models Stable Diffusion DreamBooth LoRA motion modeling module video clips motion priors animated images