title: Automatic Data Curation for Self-Supervised Learning: A Clustering-Based
    Approach
summary: Self-supervised features are the cornerstone of modern machine learning
  systems. They are typically pre-trained on data collections whose construction
  and curation typically require extensive human effort. This manual process has
  some limitations similar to those encountered in supervised learning, e.g., the
  crowd-sourced selection of data is costly and time-consuming, preventing
  scaling the dataset size. In this work, we consider the problem of automatic
  curation of high-quality datasets for self-supervised pre-training. We posit
  that such datasets should be large, diverse and balanced, and propose a
  clustering-based approach for building ones satisfying all these criteria. Our
  method involves successive and hierarchical applications of k-means on a
  large and diverse data repository to obtain clusters that distribute uniformly
  among data concepts, followed by a hierarchical, balanced sampling step from
  these clusters. Extensive experiments on three different data domains including
  web-based images, satellite images and text show that features trained on our
  automatically curated datasets outperform those trained on uncurated data while
  being on par or better than ones trained on manually curated data.
paper_authors._id: 6653f376607894ea80490fca
paper_authors.name: Huy V. Vo
paper_authors._id: 6653f376607894ea80490fcb
paper_authors.name: Vasil Khalidov
paper_authors._id: 6653f376607894ea80490fcc
paper_authors.name: Timothée Darcet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:46:20.299Z
paper_authors._id: 63047da67424d937fa31b2b0
paper_authors.avatarUrl: /avatars/cfdbb2054da0af37834a619f6c03db52.svg
paper_authors.fullname: Timothée Darcet
paper_authors.type: user
paper_authors.user: TimDarcet
paper_authors._id: 6653f376607894ea80490fcd
paper_authors.name: Théo Moutakanni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:46:36.347Z
paper_authors._id: 639111acb73a0f8c02a92844
paper_authors.avatarUrl: /avatars/58766701471be66fd784bedd319741a4.svg
paper_authors.fullname: Theo Moutakanni
paper_authors.type: user
paper_authors.user: TheoM
paper_authors._id: 6653f376607894ea80490fce
paper_authors.name: Nikita Smetanin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:46:41.954Z
paper_authors._id: 5ecaa091968f6028e0559f80
paper_authors.avatarUrl: /avatars/e7e49ebe9d5755d2e976c9553161ce9b.svg
paper_authors.fullname: Nikita Smetanin
paper_authors.type: user
paper_authors.user: nikitos9000
paper_authors._id: 6653f376607894ea80490fcf
paper_authors.name: Marc Szafraniec
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:46:53.822Z
paper_authors._id: 6479d0aabfaa9e96b8514eef
paper_authors.avatarUrl: /avatars/d7aa2d38c81219b9a8af9e331a2916a0.svg
paper_authors.fullname: Marc Szafraniec
paper_authors.type: user
paper_authors.user: mszafraniec
paper_authors._id: 6653f376607894ea80490fd0
paper_authors.name: Hugo Touvron
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:46:59.632Z
paper_authors._id: 66002a56fc78eace53a1010d
paper_authors.avatarUrl: /avatars/330ca2deeca778f1f0e3deb1b80b5049.svg
paper_authors.fullname: Hugo Touvron
paper_authors.type: user
paper_authors.user: htouvron
paper_authors._id: 6653f376607894ea80490fd1
paper_authors.name: Camille Couprie
paper_authors._id: 6653f376607894ea80490fd2
paper_authors.name: Maxime Oquab
paper_authors._id: 6653f376607894ea80490fd3
paper_authors.name: Armand Joulin
paper_authors._id: 6653f376607894ea80490fd4
paper_authors.name: Hervé Jégou
paper_authors._id: 6653f376607894ea80490fd5
paper_authors.name: Patrick Labatut
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:47:34.268Z
paper_authors._id: 6446a623cffecebe8b5da0b4
paper_authors.avatarUrl: /avatars/bce1b49b5042d178bd8affcb7fce5704.svg
paper_authors.fullname: Patrick Labatut
paper_authors.type: user
paper_authors.user: patricklabatut
paper_authors._id: 6653f376607894ea80490fd6
paper_authors.name: Piotr Bojanowski
paper_title: Automatic Data Curation for Self-Supervised Learning: A Clustering-Based
    Approach
paper_summary: Self-supervised features are the cornerstone of modern machine learning
  systems. They are typically pre-trained on data collections whose construction
  and curation typically require extensive human effort. This manual process has
  some limitations similar to those encountered in supervised learning, e.g., the
  crowd-sourced selection of data is costly and time-consuming, preventing
  scaling the dataset size. In this work, we consider the problem of automatic
  curation of high-quality datasets for self-supervised pre-training. We posit
  that such datasets should be large, diverse and balanced, and propose a
  clustering-based approach for building ones satisfying all these criteria. Our
  method involves successive and hierarchical applications of k-means on a
  large and diverse data repository to obtain clusters that distribute uniformly
  among data concepts, followed by a hierarchical, balanced sampling step from
  these clusters. Extensive experiments on three different data domains including
  web-based images, satellite images and text show that features trained on our
  automatically curated datasets outperform those trained on uncurated data while
  being on par or better than ones trained on manually curated data.
paper_ai_summary: A clustering-based approach for automatic curation of high-quality datasets for self-supervised pre-training yields features that outperform those from uncurated data and match manually curated data.
paper_ai_keywords: self-supervised features k-means clustering-based approach hierarchical applications balanced sampling web-based images satellite images text