title: Unified World Models: Coupling Video and Action Diffusion for
    Pretraining on Large Robotic Datasets
summary: Imitation learning has emerged as a promising approach towards building
  generalist robots. However, scaling imitation learning for large robot
  foundation models remains challenging due to its reliance on high-quality
  expert demonstrations. Meanwhile, large amounts of video data depicting a wide
  range of environments and diverse behaviors are readily available. This data
  provides a rich source of information about real-world dynamics and
  agent-environment interactions. Leveraging this data directly for imitation
  learning, however, has proven difficult due to the lack of action annotation
  required for most contemporary methods. In this work, we present Unified World
  Models (UWM), a framework that allows for leveraging both video and action data
  for policy learning. Specifically, a UWM integrates an action diffusion process
  and a video diffusion process within a unified transformer architecture, where
  independent diffusion timesteps govern each modality. We show that by simply
  controlling each diffusion timestep, UWM can flexibly represent a policy, a
  forward dynamics, an inverse dynamics, and a video generator. Through simulated
  and real-world experiments, we show that: (1) UWM enables effective pretraining
  on large-scale multitask robot datasets with both dynamics and action
  predictions, resulting in more generalizable and robust policies than imitation
  learning, (2) UWM naturally facilitates learning from action-free video data
  through independent control of modality-specific diffusion timesteps, further
  improving the performance of finetuned policies. Our results suggest that UWM
  offers a promising step toward harnessing large, heterogeneous datasets for
  scalable robot learning, and provides a simple unification between the often
  disparate paradigms of imitation learning and world modeling. Videos and code
  are available at https://weirdlabuw.github.io/uwm/.
paper_authors._id: 67f713dbb27f2c52f771fcb5
paper_authors.name: Chuning Zhu
paper_authors._id: 67f713dbb27f2c52f771fcb6
paper_authors.name: Raymond Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:26:05.361Z
paper_authors._id: 66fc85c98f3b78428e37be61
paper_authors.avatarUrl: /avatars/4b026e4d13e3d6f910533e84c2de040b.svg
paper_authors.fullname: Raymond Yu
paper_authors.type: user
paper_authors.user: raymondyu
paper_authors._id: 67f713dbb27f2c52f771fcb7
paper_authors.name: Siyuan Feng
paper_authors._id: 67f713dbb27f2c52f771fcb8
paper_authors.name: Benjamin Burchfiel
paper_authors._id: 67f713dbb27f2c52f771fcb9
paper_authors.name: Paarth Shah
paper_authors._id: 67f713dbb27f2c52f771fcba
paper_authors.name: Abhishek Gupta
paper_title: Unified World Models: Coupling Video and Action Diffusion for
    Pretraining on Large Robotic Datasets
paper_summary: Imitation learning has emerged as a promising approach towards building
  generalist robots. However, scaling imitation learning for large robot
  foundation models remains challenging due to its reliance on high-quality
  expert demonstrations. Meanwhile, large amounts of video data depicting a wide
  range of environments and diverse behaviors are readily available. This data
  provides a rich source of information about real-world dynamics and
  agent-environment interactions. Leveraging this data directly for imitation
  learning, however, has proven difficult due to the lack of action annotation
  required for most contemporary methods. In this work, we present Unified World
  Models (UWM), a framework that allows for leveraging both video and action data
  for policy learning. Specifically, a UWM integrates an action diffusion process
  and a video diffusion process within a unified transformer architecture, where
  independent diffusion timesteps govern each modality. We show that by simply
  controlling each diffusion timestep, UWM can flexibly represent a policy, a
  forward dynamics, an inverse dynamics, and a video generator. Through simulated
  and real-world experiments, we show that: (1) UWM enables effective pretraining
  on large-scale multitask robot datasets with both dynamics and action
  predictions, resulting in more generalizable and robust policies than imitation
  learning, (2) UWM naturally facilitates learning from action-free video data
  through independent control of modality-specific diffusion timesteps, further
  improving the performance of finetuned policies. Our results suggest that UWM
  offers a promising step toward harnessing large, heterogeneous datasets for
  scalable robot learning, and provides a simple unification between the often
  disparate paradigms of imitation learning and world modeling. Videos and code
  are available at https://weirdlabuw.github.io/uwm/.
paper_ai_summary: Unified World Models leverage both video and action data within a unified transformer architecture to enable effective pretraining on large-scale multitask robot datasets and improve policy generalization and robustness.
paper_ai_keywords: Unified World Models (UWM) action diffusion process video diffusion process unified transformer architecture diffusion timesteps policy learning forward dynamics inverse dynamics video generator pretraining multitask robot datasets policy generalization policy robustness