title: SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image
    Classification
summary: Data curation is the problem of how to collect and organize samples into a
  dataset that supports efficient learning. Despite the centrality of the task,
  little work has been devoted towards a large-scale, systematic comparison of
  various curation methods. In this work, we take steps towards a formal
  evaluation of data curation strategies and introduce SELECT, the first
  large-scale benchmark of curation strategies for image classification.
    In order to generate baseline methods for the SELECT benchmark, we create a
  new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K
  to date. Our dataset extends ImageNet with 5 new training-data shifts, each
  approximately the size of ImageNet-1K itself, and each assembled using a
  distinct curation strategy. We evaluate our data curation baselines in two
  ways: (i) using each training-data shift to train identical image
  classification models from scratch (ii) using the data itself to fit a
  pretrained self-supervised representation.
    Our findings show interesting trends, particularly pertaining to recent
  methods for data curation such as synthetic data generation and lookup based on
  CLIP embeddings. We show that although these strategies are highly competitive
  for certain tasks, the curation strategy used to assemble the original
  ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark
  can illuminate the path for new methods to further reduce the gap. We release
  our checkpoints, code, documentation, and a link to our dataset at
  https://github.com/jimmyxu123/SELECT.
paper_authors._id: 670535523c11ec9fe7f35755
paper_authors.name: Benjamin Feuer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:16:35.375Z
paper_authors._id: 62f7f4efe7c1c9bf10c81465
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f7f4efe7c1c9bf10c81465/AYlOg0fkP1o4GAP-8Y3xt.jpeg
paper_authors.fullname: Benjamin Feuer
paper_authors.type: user
paper_authors.user: penfever
paper_authors._id: 670535523c11ec9fe7f35756
paper_authors.name: Jiawei Xu
paper_authors._id: 670535523c11ec9fe7f35757
paper_authors.name: Niv Cohen
paper_authors._id: 670535523c11ec9fe7f35758
paper_authors.name: Patrick Yubeaton
paper_authors._id: 670535523c11ec9fe7f35759
paper_authors.name: Govind Mittal
paper_authors._id: 670535523c11ec9fe7f3575a
paper_authors.name: Chinmay Hegde
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T16:07:46.798Z
paper_authors._id: 631620f6894404e25068856f
paper_authors.avatarUrl: /avatars/52c30caa0ee11347f82420a14ec19996.svg
paper_authors.fullname: Chinmay Hegde
paper_authors.type: user
paper_authors.user: chegde
paper_title: SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image
    Classification
paper_summary: Data curation is the problem of how to collect and organize samples into a
  dataset that supports efficient learning. Despite the centrality of the task,
  little work has been devoted towards a large-scale, systematic comparison of
  various curation methods. In this work, we take steps towards a formal
  evaluation of data curation strategies and introduce SELECT, the first
  large-scale benchmark of curation strategies for image classification.
    In order to generate baseline methods for the SELECT benchmark, we create a
  new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K
  to date. Our dataset extends ImageNet with 5 new training-data shifts, each
  approximately the size of ImageNet-1K itself, and each assembled using a
  distinct curation strategy. We evaluate our data curation baselines in two
  ways: (i) using each training-data shift to train identical image
  classification models from scratch (ii) using the data itself to fit a
  pretrained self-supervised representation.
    Our findings show interesting trends, particularly pertaining to recent
  methods for data curation such as synthetic data generation and lookup based on
  CLIP embeddings. We show that although these strategies are highly competitive
  for certain tasks, the curation strategy used to assemble the original
  ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark
  can illuminate the path for new methods to further reduce the gap. We release
  our checkpoints, code, documentation, and a link to our dataset at
  https://github.com/jimmyxu123/SELECT.
paper_ai_summary: A benchmark named SELECT evaluates various large-scale data curation strategies for image classification, showing that traditional ImageNet-1K curation remains superior to recent methods like synthetic data and CLIP-based lookup.
paper_ai_keywords: SELECT data curation ImageNet++ image classification synthetic data generation CLIP embeddings self-supervised representation