title: iVideoGPT: Interactive VideoGPTs are Scalable World Models
summary: World models empower model-based agents to interactively explore, reason, and
  plan within imagined environments for real-world decision-making. However, the
  high demand for interactivity poses challenges in harnessing recent
  advancements in video generative models for developing world models at scale.
  This work introduces Interactive VideoGPT (iVideoGPT), a scalable
  autoregressive transformer framework that integrates multimodal signals--visual
  observations, actions, and rewards--into a sequence of tokens, facilitating an
  interactive experience of agents via next-token prediction. iVideoGPT features
  a novel compressive tokenization technique that efficiently discretizes
  high-dimensional visual observations. Leveraging its scalable architecture, we
  are able to pre-train iVideoGPT on millions of human and robotic manipulation
  trajectories, establishing a versatile foundation that is adaptable to serve as
  interactive world models for a wide range of downstream tasks. These include
  action-conditioned video prediction, visual planning, and model-based
  reinforcement learning, where iVideoGPT achieves competitive performance
  compared with state-of-the-art methods. Our work advances the development of
  interactive general world models, bridging the gap between generative video
  models and practical model-based reinforcement learning applications.
paper_authors._id: 6653f2d561422fc897228332
paper_authors.name: Jialong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:11:24.622Z
paper_authors._id: 643b866bff50448bcfc7d1d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q12v-BatVKimoi8q-coi-.jpeg
paper_authors.fullname: Jialong Wu
paper_authors.type: user
paper_authors.user: manchery
paper_authors._id: 6653f2d561422fc897228333
paper_authors.name: Shaofeng Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T10:01:15.462Z
paper_authors._id: 64e01150607e093241a208e6
paper_authors.avatarUrl: /avatars/fd24699fc000778696092383ac3d9e99.svg
paper_authors.fullname: Shaofeng Yin
paper_authors.type: user
paper_authors.user: operator22
paper_authors._id: 6653f2d561422fc897228334
paper_authors.name: Ningya Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:11:45.223Z
paper_authors._id: 65c25931c36307b08566c746
paper_authors.avatarUrl: /avatars/363aa6ed7c55fa77e72f6e6bc1391afb.svg
paper_authors.fullname: fengningya
paper_authors.type: user
paper_authors.user: fengningya
paper_authors._id: 6653f2d561422fc897228335
paper_authors.name: Xu He
paper_authors._id: 6653f2d561422fc897228336
paper_authors.name: Dong Li
paper_authors._id: 6653f2d561422fc897228337
paper_authors.name: Jianye Hao
paper_authors._id: 6653f2d561422fc897228338
paper_authors.name: Mingsheng Long
paper_title: iVideoGPT: Interactive VideoGPTs are Scalable World Models
paper_summary: World models empower model-based agents to interactively explore, reason, and
  plan within imagined environments for real-world decision-making. However, the
  high demand for interactivity poses challenges in harnessing recent
  advancements in video generative models for developing world models at scale.
  This work introduces Interactive VideoGPT (iVideoGPT), a scalable
  autoregressive transformer framework that integrates multimodal signals--visual
  observations, actions, and rewards--into a sequence of tokens, facilitating an
  interactive experience of agents via next-token prediction. iVideoGPT features
  a novel compressive tokenization technique that efficiently discretizes
  high-dimensional visual observations. Leveraging its scalable architecture, we
  are able to pre-train iVideoGPT on millions of human and robotic manipulation
  trajectories, establishing a versatile foundation that is adaptable to serve as
  interactive world models for a wide range of downstream tasks. These include
  action-conditioned video prediction, visual planning, and model-based
  reinforcement learning, where iVideoGPT achieves competitive performance
  compared with state-of-the-art methods. Our work advances the development of
  interactive general world models, bridging the gap between generative video
  models and practical model-based reinforcement learning applications.
paper_ai_summary: Interactive VideoGPT is a scalable multimodal autoregressive transformer for world modeling that integrates visual observations, actions, and rewards, enabling competitive video prediction, planning, and reinforcement learning.
paper_ai_keywords: autoregressive transformer tokenization technique multimodal signals video generative models world models interactive experience pre-train manipulation trajectories action-conditioned video prediction visual planning model-based reinforcement learning