Learning and Leveraging World Models in Visual Representation Learning
Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising
self-supervised approach that learns by leveraging a world model. While
previously limited to predicting missing parts of an input, we explore how to
generalize the JEPA prediction task to a broader set of corruptions. We
introduce Image World Models, an approach that goes beyond masked image
modeling and learns to predict the effect of global photometric transformations
in latent space. We study the recipe of learning performant IWMs and show that
it relies on three key aspects: conditioning, prediction difficulty, and
capacity. Additionally, we show that the predictive world model learned by IWM
can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM
world model matches or surpasses the performance of previous self-supervised
methods. Finally, we show that learning with an IWM allows one to control the
abstraction level of the learned representations, learning invariant
representations such as contrastive methods, or equivariant representations
such as masked image modelling.
65e5548b75b43b925e538c9b
Quentin Garrido
extracted_pending
2025-05-12T16:14:44.734Z
63049022412a1b9d381b9dcb
/avatars/cac2a540fda40c5ae558968c50385f46.svg
Quentin Garrido
user
garridoq
65e5548b75b43b925e538c9c
Mahmoud Assran
65e5548b75b43b925e538c9d
Nicolas Ballas
65e5548b75b43b925e538c9e
Adrien Bardes
65e5548b75b43b925e538c9f
Laurent Najman
65e5548b75b43b925e538ca0
Yann LeCun
admin_assigned
2024-03-04T09:54:03.248Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
Learning and Leveraging World Models in Visual Representation Learning
Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising
self-supervised approach that learns by leveraging a world model. While
previously limited to predicting missing parts of an input, we explore how to
generalize the JEPA prediction task to a broader set of corruptions. We
introduce Image World Models, an approach that goes beyond masked image
modeling and learns to predict the effect of global photometric transformations
in latent space. We study the recipe of learning performant IWMs and show that
it relies on three key aspects: conditioning, prediction difficulty, and
capacity. Additionally, we show that the predictive world model learned by IWM
can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM
world model matches or surpasses the performance of previous self-supervised
methods. Finally, we show that learning with an IWM allows one to control the
abstraction level of the learned representations, learning invariant
representations such as contrastive methods, or equivariant representations
such as masked image modelling.
Image World Models extend Joint-Embedding Predictive Architecture to predict global photometric transformations, offering adaptable and high-performance representations through fine-tuning.
Joint-Embedding Predictive Architecture Image World Models masked image modeling global photometric transformations latent space conditioning prediction difficulty capacity fine-tuning contrastive methods equivariant representations