title: Learning and Leveraging World Models in Visual Representation Learning
summary: Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising
  self-supervised approach that learns by leveraging a world model. While
  previously limited to predicting missing parts of an input, we explore how to
  generalize the JEPA prediction task to a broader set of corruptions. We
  introduce Image World Models, an approach that goes beyond masked image
  modeling and learns to predict the effect of global photometric transformations
  in latent space. We study the recipe of learning performant IWMs and show that
  it relies on three key aspects: conditioning, prediction difficulty, and
  capacity. Additionally, we show that the predictive world model learned by IWM
  can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM
  world model matches or surpasses the performance of previous self-supervised
  methods. Finally, we show that learning with an IWM allows one to control the
  abstraction level of the learned representations, learning invariant
  representations such as contrastive methods, or equivariant representations
  such as masked image modelling.
paper_authors._id: 65e5548b75b43b925e538c9b
paper_authors.name: Quentin Garrido
paper_authors._id: 65e5548b75b43b925e538c9c
paper_authors.name: Mahmoud Assran
paper_authors._id: 65e5548b75b43b925e538c9d
paper_authors.name: Nicolas Ballas
paper_authors._id: 65e5548b75b43b925e538c9e
paper_authors.name: Adrien Bardes
paper_authors._id: 65e5548b75b43b925e538c9f
paper_authors.name: Laurent Najman
paper_authors._id: 65e5548b75b43b925e538ca0
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-04T09:54:03.248Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_title: Learning and Leveraging World Models in Visual Representation Learning
paper_summary: Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising
  self-supervised approach that learns by leveraging a world model. While
  previously limited to predicting missing parts of an input, we explore how to
  generalize the JEPA prediction task to a broader set of corruptions. We
  introduce Image World Models, an approach that goes beyond masked image
  modeling and learns to predict the effect of global photometric transformations
  in latent space. We study the recipe of learning performant IWMs and show that
  it relies on three key aspects: conditioning, prediction difficulty, and
  capacity. Additionally, we show that the predictive world model learned by IWM
  can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM
  world model matches or surpasses the performance of previous self-supervised
  methods. Finally, we show that learning with an IWM allows one to control the
  abstraction level of the learned representations, learning invariant
  representations such as contrastive methods, or equivariant representations
  such as masked image modelling.
paper_ai_keywords: Joint-Embedding Predictive Architecture (JEPA) self-supervised approach world model global photometric transformations latent space Image World Models (IWM) masked image modeling predictive world model fine-tuning self-supervised methods contrastive methods equivariant representations