title: Predicting masked tokens in stochastic locations improves masked image
    modeling
summary: Self-supervised learning is a promising paradigm in deep learning that
  enables learning from unlabeled data by constructing pretext tasks that require
  learning useful representations. In natural language processing, the dominant
  pretext task has been masked language modeling (MLM), while in computer vision
  there exists an equivalent called Masked Image Modeling (MIM). However, MIM is
  challenging because it requires predicting semantic content in accurate
  locations. E.g, given an incomplete picture of a dog, we can guess that there
  is a tail, but we cannot determine its exact location. In this work, we propose
  FlexPredict, a stochastic model that addresses this challenge by incorporating
  location uncertainty into the model. Specifically, we condition the model on
  stochastic masked token positions to guide the model toward learning features
  that are more robust to location uncertainties. Our approach improves
  downstream performance on a range of tasks, e.g, compared to MIM baselines,
  FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for
  semi-supervised video segmentation using ViT-L.
paper_authors._id: 64c9c380bd5f839f2fe4b198
paper_authors.name: Amir Bar
paper_authors._id: 64c9c380bd5f839f2fe4b199
paper_authors.name: Florian Bordes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-02T08:09:12.241Z
paper_authors._id: 63166336894404e2506b8811
paper_authors.avatarUrl: /avatars/ec51a1dd86511127cf83afd4bf7c0f52.svg
paper_authors.fullname: Florian Bordes
paper_authors.type: user
paper_authors.user: Fbordes
paper_authors._id: 64c9c380bd5f839f2fe4b19a
paper_authors.name: Assaf Shocher
paper_authors._id: 64c9c380bd5f839f2fe4b19b
paper_authors.name: Mahmoud Assran
paper_authors._id: 64c9c380bd5f839f2fe4b19c
paper_authors.name: Pascal Vincent
paper_authors._id: 64c9c380bd5f839f2fe4b19d
paper_authors.name: Nicolas Ballas
paper_authors._id: 64c9c380bd5f839f2fe4b19e
paper_authors.name: Trevor Darrell
paper_authors._id: 64c9c380bd5f839f2fe4b19f
paper_authors.name: Amir Globerson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-02T08:09:45.848Z
paper_authors._id: 62cd956bc589e4a9e23ea347
paper_authors.avatarUrl: /avatars/dbdccd9258c9d18173d312a8bf14dd6e.svg
paper_authors.fullname: Amir Globerson
paper_authors.type: user
paper_authors.user: amirgloberson
paper_authors._id: 64c9c380bd5f839f2fe4b1a0
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T11:40:45.141Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_title: Predicting masked tokens in stochastic locations improves masked image
    modeling
paper_summary: Self-supervised learning is a promising paradigm in deep learning that
  enables learning from unlabeled data by constructing pretext tasks that require
  learning useful representations. In natural language processing, the dominant
  pretext task has been masked language modeling (MLM), while in computer vision
  there exists an equivalent called Masked Image Modeling (MIM). However, MIM is
  challenging because it requires predicting semantic content in accurate
  locations. E.g, given an incomplete picture of a dog, we can guess that there
  is a tail, but we cannot determine its exact location. In this work, we propose
  FlexPredict, a stochastic model that addresses this challenge by incorporating
  location uncertainty into the model. Specifically, we condition the model on
  stochastic masked token positions to guide the model toward learning features
  that are more robust to location uncertainties. Our approach improves
  downstream performance on a range of tasks, e.g, compared to MIM baselines,
  FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for
  semi-supervised video segmentation using ViT-L.
paper_ai_summary: FlexPredict enhances self-supervised learning in image processing by incorporating stochastic location uncertainty into masked image modeling, leading to improved performance on tasks like linear probing and video segmentation.
paper_ai_keywords: self-supervised learning pretext tasks masked language modeling masked image modeling FlexPredict stochastic model location uncertainty stochastic masked token positions downstream performance ImageNet linear probing semi-supervised video segmentation ViT-B ViT-L