title: MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised
    Learning of Motion and Content Features
summary: Self-supervised learning of visual representations has been focusing on
  learning content features, which do not capture object motion or location, and
  focus on identifying and differentiating objects in images and videos. On the
  other hand, optical flow estimation is a task that does not involve
  understanding the content of the images on which it is estimated. We unify the
  two approaches and introduce MC-JEPA, a joint-embedding predictive architecture
  and self-supervised learning approach to jointly learn optical flow and content
  features within a shared encoder, demonstrating that the two associated
  objectives; the optical flow estimation objective and the self-supervised
  learning objective; benefit from each other and thus learn content features
  that incorporate motion information. The proposed approach achieves performance
  on-par with existing unsupervised optical flow benchmarks, as well as with
  common self-supervised learning approaches on downstream tasks such as semantic
  segmentation of images and videos.
paper_authors._id: 64bf8f8e4561d0aca2e04d49
paper_authors.name: Adrien Bardes
paper_authors._id: 64bf8f8e4561d0aca2e04d4a
paper_authors.name: Jean Ponce
paper_authors._id: 64bf8f8e4561d0aca2e04d4b
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T11:41:25.445Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_title: MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised
    Learning of Motion and Content Features
paper_summary: Self-supervised learning of visual representations has been focusing on
  learning content features, which do not capture object motion or location, and
  focus on identifying and differentiating objects in images and videos. On the
  other hand, optical flow estimation is a task that does not involve
  understanding the content of the images on which it is estimated. We unify the
  two approaches and introduce MC-JEPA, a joint-embedding predictive architecture
  and self-supervised learning approach to jointly learn optical flow and content
  features within a shared encoder, demonstrating that the two associated
  objectives; the optical flow estimation objective and the self-supervised
  learning objective; benefit from each other and thus learn content features
  that incorporate motion information. The proposed approach achieves performance
  on-par with existing unsupervised optical flow benchmarks, as well as with
  common self-supervised learning approaches on downstream tasks such as semantic
  segmentation of images and videos.
paper_ai_summary: MC-JEPA, a joint-embedding predictive architecture, simultaneously learns optical flow and content features through self-supervised learning, improving both optical flow estimation and downstream tasks like semantic segmentation.
paper_ai_keywords: self-supervised learning visual representations content features object motion optical flow estimation joint-embedding predictive architecture shared encoder semantic segmentation