title: DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot
    Planning
summary: The ability to predict future outcomes given control actions is fundamental
  for physical reasoning. However, such predictive models, often called world
  models, have proven challenging to learn and are typically developed for
  task-specific solutions with online policy learning. We argue that the true
  potential of world models lies in their ability to reason and plan across
  diverse problems using only passive data. Concretely, we require world models
  to have the following three properties: 1) be trainable on offline,
  pre-collected trajectories, 2) support test-time behavior optimization, and 3)
  facilitate task-agnostic reasoning. To realize this, we present DINO World
  Model (DINO-WM), a new method to model visual dynamics without reconstructing
  the visual world. DINO-WM leverages spatial patch features pre-trained with
  DINOv2, enabling it to learn from offline behavioral trajectories by predicting
  future patch features. This design allows DINO-WM to achieve observational
  goals through action sequence optimization, facilitating task-agnostic behavior
  planning by treating desired goal patch features as prediction targets. We
  evaluate DINO-WM across various domains, including maze navigation, tabletop
  pushing, and particle manipulation. Our experiments demonstrate that DINO-WM
  can generate zero-shot behavioral solutions at test time without relying on
  expert demonstrations, reward modeling, or pre-learned inverse models. Notably,
  DINO-WM exhibits strong generalization capabilities compared to prior
  state-of-the-art work, adapting to diverse task families such as arbitrarily
  configured mazes, push manipulation with varied object shapes, and
  multi-particle scenarios.
paper_authors._id: 67a0783a1b24595484396c4d
paper_authors.name: Gaoyue Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T12:58:14.740Z
paper_authors._id: 63486560e0bf88ccd36fe568
paper_authors.avatarUrl: /avatars/934cffbd9f5c699abad20dcf86745382.svg
paper_authors.fullname: Gaoyue Zhou
paper_authors.type: user
paper_authors.user: gaoyuezhou
paper_authors._id: 67a0783a1b24595484396c4e
paper_authors.name: Hengkai Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T12:58:20.238Z
paper_authors._id: 634236fc8d8089ebaefb8180
paper_authors.avatarUrl: /avatars/e40518ff3f0d0a58ba4f46048c84640d.svg
paper_authors.fullname: Hengkai Pan
paper_authors.type: user
paper_authors.user: garyphk
paper_authors._id: 67a0783a1b24595484396c4f
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T12:58:26.526Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 67a0783a1b24595484396c50
paper_authors.name: Lerrel Pinto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T12:58:41.008Z
paper_authors._id: 66fffa0766bc54f4e532e3d2
paper_authors.avatarUrl: /avatars/9db8b2183097bcaddded06d1b800cf77.svg
paper_authors.fullname: Lerrel Pinto
paper_authors.type: user
paper_authors.user: LerrelPinto
paper_title: DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot
    Planning
paper_summary: The ability to predict future outcomes given control actions is fundamental
  for physical reasoning. However, such predictive models, often called world
  models, have proven challenging to learn and are typically developed for
  task-specific solutions with online policy learning. We argue that the true
  potential of world models lies in their ability to reason and plan across
  diverse problems using only passive data. Concretely, we require world models
  to have the following three properties: 1) be trainable on offline,
  pre-collected trajectories, 2) support test-time behavior optimization, and 3)
  facilitate task-agnostic reasoning. To realize this, we present DINO World
  Model (DINO-WM), a new method to model visual dynamics without reconstructing
  the visual world. DINO-WM leverages spatial patch features pre-trained with
  DINOv2, enabling it to learn from offline behavioral trajectories by predicting
  future patch features. This design allows DINO-WM to achieve observational
  goals through action sequence optimization, facilitating task-agnostic behavior
  planning by treating desired goal patch features as prediction targets. We
  evaluate DINO-WM across various domains, including maze navigation, tabletop
  pushing, and particle manipulation. Our experiments demonstrate that DINO-WM
  can generate zero-shot behavioral solutions at test time without relying on
  expert demonstrations, reward modeling, or pre-learned inverse models. Notably,
  DINO-WM exhibits strong generalization capabilities compared to prior
  state-of-the-art work, adapting to diverse task families such as arbitrarily
  configured mazes, push manipulation with varied object shapes, and
  multi-particle scenarios.
paper_ai_keywords: world models DINO World Model (DINO-WM) spatial patch features DINOv2 observational goals action sequence optimization behavioral solutions zero-shot behavioral solutions reward modeling pre-learned inverse models generalization capabilities maze navigation tabletop pushing particle manipulation arbitrarily configured mazes push manipulation multi-particle scenarios