iVideoGPT: Interactive VideoGPTs are Scalable World Models
World models empower model-based agents to interactively explore, reason, and
plan within imagined environments for real-world decision-making. However, the
high demand for interactivity poses challenges in harnessing recent
advancements in video generative models for developing world models at scale.
This work introduces Interactive VideoGPT (iVideoGPT), a scalable
autoregressive transformer framework that integrates multimodal signals--visual
observations, actions, and rewards--into a sequence of tokens, facilitating an
interactive experience of agents via next-token prediction. iVideoGPT features
a novel compressive tokenization technique that efficiently discretizes
high-dimensional visual observations. Leveraging its scalable architecture, we
are able to pre-train iVideoGPT on millions of human and robotic manipulation
trajectories, establishing a versatile foundation that is adaptable to serve as
interactive world models for a wide range of downstream tasks. These include
action-conditioned video prediction, visual planning, and model-based
reinforcement learning, where iVideoGPT achieves competitive performance
compared with state-of-the-art methods. Our work advances the development of
interactive general world models, bridging the gap between generative video
models and practical model-based reinforcement learning applications.
6653f2d561422fc897228332
Jialong Wu
admin_assigned
2024-05-27T09:11:24.622Z
643b866bff50448bcfc7d1d1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q12v-BatVKimoi8q-coi-.jpeg
Jialong Wu
user
manchery
6653f2d561422fc897228333
Shaofeng Yin
claimed_verified
2024-05-28T10:01:15.462Z
64e01150607e093241a208e6
/avatars/fd24699fc000778696092383ac3d9e99.svg
Shaofeng Yin
user
operator22
6653f2d561422fc897228334
Ningya Feng
admin_assigned
2024-05-27T09:11:45.223Z
65c25931c36307b08566c746
/avatars/363aa6ed7c55fa77e72f6e6bc1391afb.svg
fengningya
user
fengningya
6653f2d561422fc897228335
Xu He
6653f2d561422fc897228336
Dong Li
6653f2d561422fc897228337
Jianye Hao
6653f2d561422fc897228338
Mingsheng Long
iVideoGPT: Interactive VideoGPTs are Scalable World Models
World models empower model-based agents to interactively explore, reason, and
plan within imagined environments for real-world decision-making. However, the
high demand for interactivity poses challenges in harnessing recent
advancements in video generative models for developing world models at scale.
This work introduces Interactive VideoGPT (iVideoGPT), a scalable
autoregressive transformer framework that integrates multimodal signals--visual
observations, actions, and rewards--into a sequence of tokens, facilitating an
interactive experience of agents via next-token prediction. iVideoGPT features
a novel compressive tokenization technique that efficiently discretizes
high-dimensional visual observations. Leveraging its scalable architecture, we
are able to pre-train iVideoGPT on millions of human and robotic manipulation
trajectories, establishing a versatile foundation that is adaptable to serve as
interactive world models for a wide range of downstream tasks. These include
action-conditioned video prediction, visual planning, and model-based
reinforcement learning, where iVideoGPT achieves competitive performance
compared with state-of-the-art methods. Our work advances the development of
interactive general world models, bridging the gap between generative video
models and practical model-based reinforcement learning applications.
Interactive VideoGPT is a scalable multimodal autoregressive transformer for world modeling that integrates visual observations, actions, and rewards, enabling competitive video prediction, planning, and reinforcement learning.
autoregressive transformer tokenization technique multimodal signals video generative models world models interactive experience pre-train manipulation trajectories action-conditioned video prediction visual planning model-based reinforcement learning