VideoPoet: A Large Language Model for Zero-Shot Video Generation
We present VideoPoet, a language model capable of synthesizing high-quality
video, with matching audio, from a large variety of conditioning signals.
VideoPoet employs a decoder-only transformer architecture that processes
multimodal inputs -- including images, videos, text, and audio. The training
protocol follows that of Large Language Models (LLMs), consisting of two
stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
incorporates a mixture of multimodal generative objectives within an
autoregressive Transformer framework. The pretrained LLM serves as a foundation
that can be adapted for a range of video generation tasks. We present empirical
results demonstrating the model's state-of-the-art capabilities in zero-shot
video generation, specifically highlighting VideoPoet's ability to generate
high-fidelity motions. Project page: http://sites.research.google/videopoet/
6584e931e07563d421f56939
Dan Kondratyuk
admin_assigned
2023-12-22T10:47:50.146Z
63351623259c518276f0c63f
/avatars/77a49a1addeec11c9a799f9590f181b1.svg
Dan Kondratyuk
user
hyperparticle
6584e931e07563d421f5693a
Lijun Yu
admin_assigned
2023-12-22T10:48:51.547Z
639886ba65a34085ae29abb7
/avatars/51f4d4d6fbcc07ab8b7a4c1c2f7da53d.svg
Lijun Yu
user
lijunyu
6584e931e07563d421f5693b
Xiuye Gu
6584e931e07563d421f5693c
José Lezama
6584e931e07563d421f5693d
Jonathan Huang
admin_assigned
2023-12-22T10:49:20.267Z
654afe44a2de0e36e4594675
/avatars/7d0a908b9560bf990c5feb1fb3567e5a.svg
Jonathan Huang
user
jhuang11
6584e931e07563d421f5693e
Rachel Hornung
6584e931e07563d421f5693f
Hartwig Adam
6584e931e07563d421f56940
Hassan Akbari
6584e931e07563d421f56941
Yair Alon
6584e931e07563d421f56942
Vighnesh Birodkar
6584e931e07563d421f56943
Yong Cheng
6584e931e07563d421f56944
Ming-Chang Chiu
6584e931e07563d421f56945
Josh Dillon
6584e931e07563d421f56946
Irfan Essa
6584e931e07563d421f56947
Agrim Gupta
admin_assigned
2023-12-22T10:51:12.598Z
63defb9015266dd945f9f772
/avatars/b304ac6c3b45553358f8b6cd5c3df293.svg
Agrim
user
agrimgupta
6584e931e07563d421f56948
Meera Hahn
claimed_verified
2025-07-29T07:13:58.393Z
6887d4b7d01ee133939109b9
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bCUDzOvi2FFQVzOzmGBaW.png
Meera Hahn
user
meerahahn
6584e931e07563d421f56949
Anja Hauth
6584e931e07563d421f5694a
David Hendon
6584e931e07563d421f5694b
Alonso Martinez
6584e931e07563d421f5694c
David Minnen
6584e931e07563d421f5694d
David Ross
6584e931e07563d421f5694e
Grant Schindler
6584e931e07563d421f5694f
Mikhail Sirotenko
admin_assigned
2023-12-22T10:52:11.389Z
655cd825d246e013225191c6
/avatars/2b376ed5be43c448d1cc385446f59613.svg
Mikhail Sirotenko
user
msirotenko
6584e931e07563d421f56950
Kihyuk Sohn
6584e931e07563d421f56951
Krishna Somandepalli
6584e931e07563d421f56952
Huisheng Wang
6584e931e07563d421f56953
Jimmy Yan
6584e931e07563d421f56954
Ming-Hsuan Yang
6584e931e07563d421f56955
Xuan Yang
6584e931e07563d421f56956
Bryan Seybold
6584e931e07563d421f56957
Lu Jiang
claimed_verified
2023-12-23T11:17:53.789Z
64a5cba3bea0116f8f7187a7
/avatars/97b3bad82e359c57d1ab7d6e3aa7748b.svg
Lu Jiang
user
roadjiang
VideoPoet: A Large Language Model for Zero-Shot Video Generation
We present VideoPoet, a language model capable of synthesizing high-quality
video, with matching audio, from a large variety of conditioning signals.
VideoPoet employs a decoder-only transformer architecture that processes
multimodal inputs -- including images, videos, text, and audio. The training
protocol follows that of Large Language Models (LLMs), consisting of two
stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
incorporates a mixture of multimodal generative objectives within an
autoregressive Transformer framework. The pretrained LLM serves as a foundation
that can be adapted for a range of video generation tasks. We present empirical
results demonstrating the model's state-of-the-art capabilities in zero-shot
video generation, specifically highlighting VideoPoet's ability to generate
high-fidelity motions. Project page: http://sites.research.google/videopoet/
VideoPoet is a language model that synthesizes high-quality video and audio from diverse inputs using a transformer architecture, showcasing superior zero-shot video generation capabilities.
decoder-only transformer multimodal inputs autoregressive Transformer framework pretraining task-specific adaptation multimodal generative objectives zero-shot video generation high-fidelity motions