Intuitive physics understanding emerges from self-supervised pretraining
  on natural videos
We investigate the emergence of intuitive physics understanding in
general-purpose deep neural network models trained to predict masked regions in
natural videos. Leveraging the violation-of-expectation framework, we find that
video prediction models trained to predict outcomes in a learned representation
space demonstrate an understanding of various intuitive physics properties,
such as object permanence and shape consistency. In contrast, video prediction
in pixel space and multimodal large language models, which reason through text,
achieve performance closer to chance. Our comparisons of these architectures
reveal that jointly learning an abstract representation space while predicting
missing parts of sensory input, akin to predictive coding, is sufficient to
acquire an understanding of intuitive physics, and that even models trained on
one week of unique video achieve above chance performance. This challenges the
idea that core knowledge -- a set of innate systems to help understand the
world -- needs to be hardwired to develop an understanding of intuitive
physics.
67b450cf315f7b69956df3d6
Quentin Garrido
admin_assigned
2025-02-19T15:28:09.217Z
63049022412a1b9d381b9dcb
/avatars/cac2a540fda40c5ae558968c50385f46.svg
Quentin Garrido
user
garridoq
67b450cf315f7b69956df3d7
Nicolas Ballas
67b450cf315f7b69956df3d8
Mahmoud Assran
67b450cf315f7b69956df3d9
Adrien Bardes
67b450cf315f7b69956df3da
Laurent Najman
67b450cf315f7b69956df3db
Michael Rabbat
67b450cf315f7b69956df3dc
Emmanuel Dupoux
admin_assigned
2025-02-19T15:44:00.610Z
63317d2118711776b4663c3a
/avatars/7dedd1934c1000b6f81a2a37ec348347.svg
Emmanuel Dupoux
user
edupoux
67b450cf315f7b69956df3dd
Yann LeCun
admin_assigned
2025-02-19T15:43:53.902Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
Intuitive physics understanding emerges from self-supervised pretraining
  on natural videos
We investigate the emergence of intuitive physics understanding in
general-purpose deep neural network models trained to predict masked regions in
natural videos. Leveraging the violation-of-expectation framework, we find that
video prediction models trained to predict outcomes in a learned representation
space demonstrate an understanding of various intuitive physics properties,
such as object permanence and shape consistency. In contrast, video prediction
in pixel space and multimodal large language models, which reason through text,
achieve performance closer to chance. Our comparisons of these architectures
reveal that jointly learning an abstract representation space while predicting
missing parts of sensory input, akin to predictive coding, is sufficient to
acquire an understanding of intuitive physics, and that even models trained on
one week of unique video achieve above chance performance. This challenges the
idea that core knowledge -- a set of innate systems to help understand the
world -- needs to be hardwired to develop an understanding of intuitive
physics.
Deep neural network models trained to predict masked regions in natural videos learn intuitive physics properties in a learned representation space, contrasting with models operating in pixel space or using language.
video prediction models learned representation space object permanence shape consistency pixel space multimodal large language models predictive coding intuitive physics core knowledge