title: Intuitive physics understanding emerges from self-supervised pretraining
    on natural videos
summary: We investigate the emergence of intuitive physics understanding in
  general-purpose deep neural network models trained to predict masked regions in
  natural videos. Leveraging the violation-of-expectation framework, we find that
  video prediction models trained to predict outcomes in a learned representation
  space demonstrate an understanding of various intuitive physics properties,
  such as object permanence and shape consistency. In contrast, video prediction
  in pixel space and multimodal large language models, which reason through text,
  achieve performance closer to chance. Our comparisons of these architectures
  reveal that jointly learning an abstract representation space while predicting
  missing parts of sensory input, akin to predictive coding, is sufficient to
  acquire an understanding of intuitive physics, and that even models trained on
  one week of unique video achieve above chance performance. This challenges the
  idea that core knowledge -- a set of innate systems to help understand the
  world -- needs to be hardwired to develop an understanding of intuitive
  physics.
paper_authors._id: 67b450cf315f7b69956df3d6
paper_authors.name: Quentin Garrido
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:28:09.217Z
paper_authors._id: 63049022412a1b9d381b9dcb
paper_authors.avatarUrl: /avatars/7382c0a0e3f5609b754ec09a309d33f6.svg
paper_authors.fullname: Quentin Garrido
paper_authors.type: user
paper_authors.user: garridoq
paper_authors._id: 67b450cf315f7b69956df3d7
paper_authors.name: Nicolas Ballas
paper_authors._id: 67b450cf315f7b69956df3d8
paper_authors.name: Mahmoud Assran
paper_authors._id: 67b450cf315f7b69956df3d9
paper_authors.name: Adrien Bardes
paper_authors._id: 67b450cf315f7b69956df3da
paper_authors.name: Laurent Najman
paper_authors._id: 67b450cf315f7b69956df3db
paper_authors.name: Michael Rabbat
paper_authors._id: 67b450cf315f7b69956df3dc
paper_authors.name: Emmanuel Dupoux
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:44:00.610Z
paper_authors._id: 63317d2118711776b4663c3a
paper_authors.avatarUrl: /avatars/7dedd1934c1000b6f81a2a37ec348347.svg
paper_authors.fullname: Emmanuel Dupoux
paper_authors.type: user
paper_authors.user: edupoux
paper_authors._id: 67b450cf315f7b69956df3dd
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:43:53.902Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_title: Intuitive physics understanding emerges from self-supervised pretraining
    on natural videos
paper_summary: We investigate the emergence of intuitive physics understanding in
  general-purpose deep neural network models trained to predict masked regions in
  natural videos. Leveraging the violation-of-expectation framework, we find that
  video prediction models trained to predict outcomes in a learned representation
  space demonstrate an understanding of various intuitive physics properties,
  such as object permanence and shape consistency. In contrast, video prediction
  in pixel space and multimodal large language models, which reason through text,
  achieve performance closer to chance. Our comparisons of these architectures
  reveal that jointly learning an abstract representation space while predicting
  missing parts of sensory input, akin to predictive coding, is sufficient to
  acquire an understanding of intuitive physics, and that even models trained on
  one week of unique video achieve above chance performance. This challenges the
  idea that core knowledge -- a set of innate systems to help understand the
  world -- needs to be hardwired to develop an understanding of intuitive
  physics.
paper_ai_keywords: video prediction models learned representation space pixel space multimodal large language models