title: Scaling Language-Free Visual Representation Learning
summary: Visual Self-Supervised Learning (SSL) currently underperforms Contrastive
  Language-Image Pretraining (CLIP) in multimodal settings such as Visual
  Question Answering (VQA). This multimodal gap is often attributed to the
  semantics introduced by language supervision, even though visual SSL and CLIP
  models are often trained on different data. In this work, we ask the question:
  "Do visual self-supervised approaches lag behind CLIP due to the lack of
  language supervision, or differences in the training data?" We study this
  question by training both visual SSL and CLIP models on the same MetaCLIP data,
  and leveraging VQA as a diverse testbed for vision encoders. In this controlled
  setup, visual SSL models scale better than CLIP models in terms of data and
  model capacity, and visual SSL performance does not saturate even after scaling
  up to 7B parameters. Consequently, we observe visual SSL methods achieve
  CLIP-level performance on a wide range of VQA and classic vision benchmarks.
  These findings demonstrate that pure visual SSL can match language-supervised
  visual pretraining at scale, opening new opportunities for vision-centric
  representation learning.
paper_authors._id: 67ecebc51f669fb5591616cd
paper_authors.name: David Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:18:13.746Z
paper_authors._id: 66e9df8c38f5bae5a91785ca
paper_authors.avatarUrl: /avatars/6f49429fc34553903309872ec6f122b8.svg
paper_authors.fullname: David Fan
paper_authors.type: user
paper_authors.user: davidfan97
paper_authors._id: 67ecebc51f669fb5591616ce
paper_authors.name: Shengbang Tong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:18:31.173Z
paper_authors._id: 6374cbb7255276f3a22b4b35
paper_authors.avatarUrl: /avatars/7cf1bbb83447441e5fa2e1e4fcf7617b.svg
paper_authors.fullname: Peter Tong
paper_authors.type: user
paper_authors.user: tsbpp
paper_authors._id: 67ecebc51f669fb5591616cf
paper_authors.name: Jiachen Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:54:32.112Z
paper_authors._id: 6552126dd8a8835b66653767
paper_authors.avatarUrl: /avatars/0b1dad9ebaeada8f5e7ebe453123960b.svg
paper_authors.fullname: Jiachen Zhu
paper_authors.type: user
paper_authors.user: JiachenZhu
paper_authors._id: 67ecebc51f669fb5591616d0
paper_authors.name: Koustuv Sinha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:54:39.909Z
paper_authors._id: 622b93067b9143726fbedc37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647023505150-622b93067b9143726fbedc37.jpeg
paper_authors.fullname: Koustuv Sinha
paper_authors.type: user
paper_authors.user: koustuvs
paper_authors._id: 67ecebc51f669fb5591616d1
paper_authors.name: Zhuang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:55:01.118Z
paper_authors._id: 647d3a9383cdb93baf76dce2
paper_authors.avatarUrl: /avatars/56efca912a66ef9d4f2f3679aac31bae.svg
paper_authors.fullname: Zhuang Liu
paper_authors.type: user
paper_authors.user: liuzhuang13
paper_authors._id: 67ecebc51f669fb5591616d2
paper_authors.name: Xinlei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:55:07.362Z
paper_authors._id: 63e58e3a006a775275e59e41
paper_authors.avatarUrl: /avatars/75262a35b27a2ae1939df9118120d99e.svg
paper_authors.fullname: Xinlei Chen
paper_authors.type: user
paper_authors.user: endernewton
paper_authors._id: 67ecebc51f669fb5591616d3
paper_authors.name: Michael Rabbat
paper_authors._id: 67ecebc51f669fb5591616d4
paper_authors.name: Nicolas Ballas
paper_authors._id: 67ecebc51f669fb5591616d5
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:55:22.112Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 67ecebc51f669fb5591616d6
paper_authors.name: Amir Bar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:55:47.451Z
paper_authors._id: 632bad473690fb57e70c9d42
paper_authors.avatarUrl: /avatars/8c09bc614a7e62a7bd30c4cbb97d4122.svg
paper_authors.fullname: Amir Bar
paper_authors.type: user
paper_authors.user: amirbar1
paper_authors._id: 67ecebc51f669fb5591616d7
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:55:54.586Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_title: Scaling Language-Free Visual Representation Learning
paper_summary: Visual Self-Supervised Learning (SSL) currently underperforms Contrastive
  Language-Image Pretraining (CLIP) in multimodal settings such as Visual
  Question Answering (VQA). This multimodal gap is often attributed to the
  semantics introduced by language supervision, even though visual SSL and CLIP
  models are often trained on different data. In this work, we ask the question:
  "Do visual self-supervised approaches lag behind CLIP due to the lack of
  language supervision, or differences in the training data?" We study this
  question by training both visual SSL and CLIP models on the same MetaCLIP data,
  and leveraging VQA as a diverse testbed for vision encoders. In this controlled
  setup, visual SSL models scale better than CLIP models in terms of data and
  model capacity, and visual SSL performance does not saturate even after scaling
  up to 7B parameters. Consequently, we observe visual SSL methods achieve
  CLIP-level performance on a wide range of VQA and classic vision benchmarks.
  These findings demonstrate that pure visual SSL can match language-supervised
  visual pretraining at scale, opening new opportunities for vision-centric
  representation learning.
paper_ai_summary: Visual self-supervised learning matches language-supervised visual pretraining performance on VQA and vision benchmarks when both are trained on the same dataset and scaled appropriately.
paper_ai_keywords: visual self-supervised learning contrastive language-image pretraining CLIP visual question answering VQA MetaCLIP vision encoders data and model capacity