title: VideoPoet: A Large Language Model for Zero-Shot Video Generation
summary: We present VideoPoet, a language model capable of synthesizing high-quality
  video, with matching audio, from a large variety of conditioning signals.
  VideoPoet employs a decoder-only transformer architecture that processes
  multimodal inputs -- including images, videos, text, and audio. The training
  protocol follows that of Large Language Models (LLMs), consisting of two
  stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
  incorporates a mixture of multimodal generative objectives within an
  autoregressive Transformer framework. The pretrained LLM serves as a foundation
  that can be adapted for a range of video generation tasks. We present empirical
  results demonstrating the model's state-of-the-art capabilities in zero-shot
  video generation, specifically highlighting VideoPoet's ability to generate
  high-fidelity motions. Project page: http://sites.research.google/videopoet/
paper_authors._id: 6584e931e07563d421f56939
paper_authors.name: Dan Kondratyuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:47:50.146Z
paper_authors._id: 63351623259c518276f0c63f
paper_authors.avatarUrl: /avatars/77a49a1addeec11c9a799f9590f181b1.svg
paper_authors.fullname: Dan Kondratyuk
paper_authors.type: user
paper_authors.user: hyperparticle
paper_authors._id: 6584e931e07563d421f5693a
paper_authors.name: Lijun Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:48:51.547Z
paper_authors._id: 639886ba65a34085ae29abb7
paper_authors.avatarUrl: /avatars/51f4d4d6fbcc07ab8b7a4c1c2f7da53d.svg
paper_authors.fullname: Lijun Yu
paper_authors.type: user
paper_authors.user: lijunyu
paper_authors._id: 6584e931e07563d421f5693b
paper_authors.name: Xiuye Gu
paper_authors._id: 6584e931e07563d421f5693c
paper_authors.name: José Lezama
paper_authors._id: 6584e931e07563d421f5693d
paper_authors.name: Jonathan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:49:20.267Z
paper_authors._id: 654afe44a2de0e36e4594675
paper_authors.avatarUrl: /avatars/7d0a908b9560bf990c5feb1fb3567e5a.svg
paper_authors.fullname: Jonathan Huang
paper_authors.type: user
paper_authors.user: jhuang11
paper_authors._id: 6584e931e07563d421f5693e
paper_authors.name: Rachel Hornung
paper_authors._id: 6584e931e07563d421f5693f
paper_authors.name: Hartwig Adam
paper_authors._id: 6584e931e07563d421f56940
paper_authors.name: Hassan Akbari
paper_authors._id: 6584e931e07563d421f56941
paper_authors.name: Yair Alon
paper_authors._id: 6584e931e07563d421f56942
paper_authors.name: Vighnesh Birodkar
paper_authors._id: 6584e931e07563d421f56943
paper_authors.name: Yong Cheng
paper_authors._id: 6584e931e07563d421f56944
paper_authors.name: Ming-Chang Chiu
paper_authors._id: 6584e931e07563d421f56945
paper_authors.name: Josh Dillon
paper_authors._id: 6584e931e07563d421f56946
paper_authors.name: Irfan Essa
paper_authors._id: 6584e931e07563d421f56947
paper_authors.name: Agrim Gupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:51:12.598Z
paper_authors._id: 63defb9015266dd945f9f772
paper_authors.avatarUrl: /avatars/b304ac6c3b45553358f8b6cd5c3df293.svg
paper_authors.fullname: Agrim
paper_authors.type: user
paper_authors.user: agrimgupta
paper_authors._id: 6584e931e07563d421f56948
paper_authors.name: Meera Hahn
paper_authors._id: 6584e931e07563d421f56949
paper_authors.name: Anja Hauth
paper_authors._id: 6584e931e07563d421f5694a
paper_authors.name: David Hendon
paper_authors._id: 6584e931e07563d421f5694b
paper_authors.name: Alonso Martinez
paper_authors._id: 6584e931e07563d421f5694c
paper_authors.name: David Minnen
paper_authors._id: 6584e931e07563d421f5694d
paper_authors.name: David Ross
paper_authors._id: 6584e931e07563d421f5694e
paper_authors.name: Grant Schindler
paper_authors._id: 6584e931e07563d421f5694f
paper_authors.name: Mikhail Sirotenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:52:11.389Z
paper_authors._id: 655cd825d246e013225191c6
paper_authors.avatarUrl: /avatars/2b376ed5be43c448d1cc385446f59613.svg
paper_authors.fullname: Mikhail Sirotenko
paper_authors.type: user
paper_authors.user: msirotenko
paper_authors._id: 6584e931e07563d421f56950
paper_authors.name: Kihyuk Sohn
paper_authors._id: 6584e931e07563d421f56951
paper_authors.name: Krishna Somandepalli
paper_authors._id: 6584e931e07563d421f56952
paper_authors.name: Huisheng Wang
paper_authors._id: 6584e931e07563d421f56953
paper_authors.name: Jimmy Yan
paper_authors._id: 6584e931e07563d421f56954
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 6584e931e07563d421f56955
paper_authors.name: Xuan Yang
paper_authors._id: 6584e931e07563d421f56956
paper_authors.name: Bryan Seybold
paper_authors._id: 6584e931e07563d421f56957
paper_authors.name: Lu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-23T11:17:53.789Z
paper_authors._id: 64a5cba3bea0116f8f7187a7
paper_authors.avatarUrl: /avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
paper_authors.fullname: Lu Jiang
paper_authors.type: user
paper_authors.user: roadjiang
paper_title: VideoPoet: A Large Language Model for Zero-Shot Video Generation
paper_summary: We present VideoPoet, a language model capable of synthesizing high-quality
  video, with matching audio, from a large variety of conditioning signals.
  VideoPoet employs a decoder-only transformer architecture that processes
  multimodal inputs -- including images, videos, text, and audio. The training
  protocol follows that of Large Language Models (LLMs), consisting of two
  stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
  incorporates a mixture of multimodal generative objectives within an
  autoregressive Transformer framework. The pretrained LLM serves as a foundation
  that can be adapted for a range of video generation tasks. We present empirical
  results demonstrating the model's state-of-the-art capabilities in zero-shot
  video generation, specifically highlighting VideoPoet's ability to generate
  high-fidelity motions. Project page: http://sites.research.google/videopoet/
paper_ai_keywords: decoder-only transformer architecture multimodal inputs autoregressive Transformer framework pretraining task-specific adaptation multimodal generative objectives zero-shot video generation high-fidelity motions