title: Video as the New Language for Real-World Decision Making
summary: Both text and video data are abundant on the internet and support large-scale
  self-supervised learning through next token or frame prediction. However, they
  have not been equally leveraged: language models have had significant
  real-world impact, whereas video generation has remained largely limited to
  media entertainment. Yet video data captures important information about the
  physical world that is difficult to express in language. To address this gap,
  we discuss an under-appreciated opportunity to extend video generation to solve
  tasks in the real world. We observe how, akin to language, video can serve as a
  unified interface that can absorb internet knowledge and represent diverse
  tasks. Moreover, we demonstrate how, like language models, video generation can
  serve as planners, agents, compute engines, and environment simulators through
  techniques such as in-context learning, planning and reinforcement learning. We
  identify major impact opportunities in domains such as robotics, self-driving,
  and science, supported by recent work that demonstrates how such advanced
  capabilities in video generation are plausibly within reach. Lastly, we
  identify key challenges in video generation that mitigate progress. Addressing
  these challenges will enable video generation models to demonstrate unique
  value alongside language models in a wider array of AI applications.
paper_authors._id: 65dea45f6efc2a4ebb07c3b5
paper_authors.name: Sherry Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:26:49.890Z
paper_authors._id: 636a05908d250800ca6de12b
paper_authors.avatarUrl: /avatars/549a764e6cbdf2a410329dea0b726d23.svg
paper_authors.fullname: Sherry Yang
paper_authors.type: user
paper_authors.user: sherryy
paper_authors._id: 65dea45f6efc2a4ebb07c3b6
paper_authors.name: Jacob Walker
paper_authors._id: 65dea45f6efc2a4ebb07c3b7
paper_authors.name: Jack Parker-Holder
paper_authors._id: 65dea45f6efc2a4ebb07c3b8
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:27:41.132Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 65dea45f6efc2a4ebb07c3b9
paper_authors.name: Jake Bruce
paper_authors._id: 65dea45f6efc2a4ebb07c3ba
paper_authors.name: Andre Barreto
paper_authors._id: 65dea45f6efc2a4ebb07c3bb
paper_authors.name: Pieter Abbeel
paper_authors._id: 65dea45f6efc2a4ebb07c3bc
paper_authors.name: Dale Schuurmans
paper_title: Video as the New Language for Real-World Decision Making
paper_summary: Both text and video data are abundant on the internet and support large-scale
  self-supervised learning through next token or frame prediction. However, they
  have not been equally leveraged: language models have had significant
  real-world impact, whereas video generation has remained largely limited to
  media entertainment. Yet video data captures important information about the
  physical world that is difficult to express in language. To address this gap,
  we discuss an under-appreciated opportunity to extend video generation to solve
  tasks in the real world. We observe how, akin to language, video can serve as a
  unified interface that can absorb internet knowledge and represent diverse
  tasks. Moreover, we demonstrate how, like language models, video generation can
  serve as planners, agents, compute engines, and environment simulators through
  techniques such as in-context learning, planning and reinforcement learning. We
  identify major impact opportunities in domains such as robotics, self-driving,
  and science, supported by recent work that demonstrates how such advanced
  capabilities in video generation are plausibly within reach. Lastly, we
  identify key challenges in video generation that mitigate progress. Addressing
  these challenges will enable video generation models to demonstrate unique
  value alongside language models in a wider array of AI applications.
paper_ai_summary: Video generation can serve as a unified interface for diverse tasks and advance real-world AI applications through techniques like in-context learning and reinforcement learning, with opportunities in robotics, self-driving, and science.
paper_ai_keywords: in-context learning planning reinforcement learning