title: Emu Video: Factorizing Text-to-Video Generation by Explicit Image
    Conditioning
summary: We present Emu Video, a text-to-video generation model that factorizes the
  generation into two steps: first generating an image conditioned on the text,
  and then generating a video conditioned on the text and the generated image. We
  identify critical design decisions--adjusted noise schedules for diffusion, and
  multi-stage training--that enable us to directly generate high quality and high
  resolution videos, without requiring a deep cascade of models as in prior work.
  In human evaluations, our generated videos are strongly preferred in quality
  compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's
  PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial
  solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing
  approach naturally lends itself to animating images based on a user's text
  prompt, where our generations are preferred 96% over prior work.
paper_authors._id: 655ac402deee83130a46b65b
paper_authors.name: Rohit Girdhar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T10:09:09.220Z
paper_authors._id: 63c6c737656e7822e23732d8
paper_authors.avatarUrl: /avatars/3cf062864dbf2b5423eb1f4beaeec1ea.svg
paper_authors.fullname: Rohit Girdhar
paper_authors.type: user
paper_authors.user: rgirdhar
paper_authors._id: 655ac402deee83130a46b65c
paper_authors.name: Mannat Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-23T10:49:19.549Z
paper_authors._id: 650fece35877b1c077298b81
paper_authors.avatarUrl: /avatars/be9c948ecf9c413ad4c8aa6598d3ba86.svg
paper_authors.fullname: Mannat Singh
paper_authors.type: user
paper_authors.user: mannatsingh
paper_authors._id: 655ac402deee83130a46b65d
paper_authors.name: Andrew Brown
paper_authors._id: 655ac402deee83130a46b65e
paper_authors.name: Quentin Duval
paper_authors._id: 655ac402deee83130a46b65f
paper_authors.name: Samaneh Azadi
paper_authors._id: 655ac402deee83130a46b660
paper_authors.name: Sai Saketh Rambhatla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T10:10:22.412Z
paper_authors._id: 6435a89b876d46610bf30b21
paper_authors.avatarUrl: /avatars/88dba6589b07ad310bad94108774d893.svg
paper_authors.fullname: Saketh Rambhatla
paper_authors.type: user
paper_authors.user: rssaketh
paper_authors._id: 655ac402deee83130a46b661
paper_authors.name: Akbar Shah
paper_authors._id: 655ac402deee83130a46b662
paper_authors.name: Xi Yin
paper_authors._id: 655ac402deee83130a46b663
paper_authors.name: Devi Parikh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T10:10:50.917Z
paper_authors._id: 6340753995c20b9447379c67
paper_authors.avatarUrl: /avatars/f793c00a98335cd8f7efce41d403da16.svg
paper_authors.fullname: Devi Parikh
paper_authors.type: user
paper_authors.user: deviparikh-genai
paper_authors._id: 655ac402deee83130a46b664
paper_authors.name: Ishan Misra
paper_title: Emu Video: Factorizing Text-to-Video Generation by Explicit Image
    Conditioning
paper_summary: We present Emu Video, a text-to-video generation model that factorizes the
  generation into two steps: first generating an image conditioned on the text,
  and then generating a video conditioned on the text and the generated image. We
  identify critical design decisions--adjusted noise schedules for diffusion, and
  multi-stage training--that enable us to directly generate high quality and high
  resolution videos, without requiring a deep cascade of models as in prior work.
  In human evaluations, our generated videos are strongly preferred in quality
  compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's
  PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial
  solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing
  approach naturally lends itself to animating images based on a user's text
  prompt, where our generations are preferred 96% over prior work.
paper_ai_keywords: text-to-video generation model diffusion noise schedules multi-stage training high quality and high resolution videos human evaluations Google's Imagen Video Nvidia's PYOCO Meta's Make-A-Video RunwayML's Gen2 Pika Labs animating images