title: An accurate detection is not all you need to combat label noise in
    web-noisy datasets
summary: Training a classifier on web-crawled data demands learning algorithms that
  are robust to annotation errors and irrelevant examples. This paper builds upon
  the recent empirical observation that applying unsupervised contrastive
  learning to noisy, web-crawled datasets yields a feature representation under
  which the in-distribution (ID) and out-of-distribution (OOD) samples are
  linearly separable. We show that direct estimation of the separating hyperplane
  can indeed offer an accurate detection of OOD samples, and yet, surprisingly,
  this detection does not translate into gains in classification accuracy.
  Digging deeper into this phenomenon, we discover that the near-perfect
  detection misses a type of clean examples that are valuable for supervised
  learning. These examples often represent visually simple images, which are
  relatively easy to identify as clean examples using standard loss- or
  distance-based methods despite being poorly separated from the OOD distribution
  using unsupervised learning. Because we further observe a low correlation with
  SOTA metrics, this urges us to propose a hybrid solution that alternates
  between noise detection using linear separation and a state-of-the-art (SOTA)
  small-loss approach. When combined with the SOTA algorithm PLS, we
  substantially improve SOTA results for real-world image classification in the
  presence of web noise github.com/PaulAlbert31/LSA
paper_authors._id: 668dda5f963fdbcecc82a151
paper_authors.name: Paul Albert
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:19:49.399Z
paper_authors._id: 65f26ed18404ac0e4cfe7d83
paper_authors.avatarUrl: /avatars/9c72744836f86a8e355a45700b10e393.svg
paper_authors.fullname: Paul Albert
paper_authors.type: user
paper_authors.user: PAlbert31
paper_authors._id: 668dda5f963fdbcecc82a152
paper_authors.name: Jack Valmadre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:58:56.498Z
paper_authors._id: 643632f2aaef013d1af27e54
paper_authors.avatarUrl: /avatars/156a4257f3fce1e40f89b42f1ea3522d.svg
paper_authors.fullname: Jack Valmadre
paper_authors.type: user
paper_authors.user: jvlmdr
paper_authors._id: 668dda5f963fdbcecc82a153
paper_authors.name: Eric Arazo
paper_authors._id: 668dda5f963fdbcecc82a154
paper_authors.name: Tarun Krishna
paper_authors._id: 668dda5f963fdbcecc82a155
paper_authors.name: Noel E. O'Connor
paper_authors._id: 668dda5f963fdbcecc82a156
paper_authors.name: Kevin McGuinness
paper_title: An accurate detection is not all you need to combat label noise in
    web-noisy datasets
paper_summary: Training a classifier on web-crawled data demands learning algorithms that
  are robust to annotation errors and irrelevant examples. This paper builds upon
  the recent empirical observation that applying unsupervised contrastive
  learning to noisy, web-crawled datasets yields a feature representation under
  which the in-distribution (ID) and out-of-distribution (OOD) samples are
  linearly separable. We show that direct estimation of the separating hyperplane
  can indeed offer an accurate detection of OOD samples, and yet, surprisingly,
  this detection does not translate into gains in classification accuracy.
  Digging deeper into this phenomenon, we discover that the near-perfect
  detection misses a type of clean examples that are valuable for supervised
  learning. These examples often represent visually simple images, which are
  relatively easy to identify as clean examples using standard loss- or
  distance-based methods despite being poorly separated from the OOD distribution
  using unsupervised learning. Because we further observe a low correlation with
  SOTA metrics, this urges us to propose a hybrid solution that alternates
  between noise detection using linear separation and a state-of-the-art (SOTA)
  small-loss approach. When combined with the SOTA algorithm PLS, we
  substantially improve SOTA results for real-world image classification in the
  presence of web noise github.com/PaulAlbert31/LSA
paper_ai_keywords: contrastive learning feature representation in-distribution (ID) out-of-distribution (OOD) separating hyperplane clean examples loss-based methods distance-based methods state-of-the-art (SOTA) small-loss approach