title: I-Con: A Unifying Framework for Representation Learning
summary: As the field of representation learning grows, there has been a proliferation
  of different loss functions to solve different classes of problems. We
  introduce a single information-theoretic equation that generalizes a large
  collection of modern loss functions in machine learning. In particular, we
  introduce a framework that shows that several broad classes of machine learning
  methods are precisely minimizing an integrated KL divergence between two
  conditional distributions: the supervisory and learned representations. This
  viewpoint exposes a hidden information geometry underlying clustering, spectral
  methods, dimensionality reduction, contrastive learning, and supervised
  learning. This framework enables the development of new loss functions by
  combining successful techniques from across the literature. We not only present
  a wide array of proofs, connecting over 23 different approaches, but we also
  leverage these theoretical results to create state-of-the-art unsupervised
  image classifiers that achieve a +8% improvement over the prior
  state-of-the-art on unsupervised classification on ImageNet-1K. We also
  demonstrate that I-Con can be used to derive principled debiasing methods which
  improve contrastive representation learners.
paper_authors._id: 6809ba7976a4f4f7268546a7
paper_authors.name: Shaden Alshammari
paper_authors._id: 6809ba7976a4f4f7268546a8
paper_authors.name: John Hershey
paper_authors._id: 6809ba7976a4f4f7268546a9
paper_authors.name: Axel Feldmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-24T11:27:03.416Z
paper_authors._id: 63124ad25e2531edb9edca45
paper_authors.avatarUrl: /avatars/9cc2e352ba3f79347e588a3e4a814987.svg
paper_authors.fullname: Axel Feldmann
paper_authors.type: user
paper_authors.user: axelf
paper_authors._id: 6809ba7976a4f4f7268546aa
paper_authors.name: William T. Freeman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-24T11:27:14.536Z
paper_authors._id: 654e866a85a5e608059a9b4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O0T97EupMHH_iXcv1xYFr.png
paper_authors.fullname: William Freeman
paper_authors.type: user
paper_authors.user: mrpuppt
paper_authors._id: 6809ba7976a4f4f7268546ab
paper_authors.name: Mark Hamilton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-24T11:27:36.305Z
paper_authors._id: 62dae3734398e21bf7f53443
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658512237806-noauth.jpeg
paper_authors.fullname: Mark Hamilton
paper_authors.type: user
paper_authors.user: mhamilton723
paper_title: I-Con: A Unifying Framework for Representation Learning
paper_summary: As the field of representation learning grows, there has been a proliferation
  of different loss functions to solve different classes of problems. We
  introduce a single information-theoretic equation that generalizes a large
  collection of modern loss functions in machine learning. In particular, we
  introduce a framework that shows that several broad classes of machine learning
  methods are precisely minimizing an integrated KL divergence between two
  conditional distributions: the supervisory and learned representations. This
  viewpoint exposes a hidden information geometry underlying clustering, spectral
  methods, dimensionality reduction, contrastive learning, and supervised
  learning. This framework enables the development of new loss functions by
  combining successful techniques from across the literature. We not only present
  a wide array of proofs, connecting over 23 different approaches, but we also
  leverage these theoretical results to create state-of-the-art unsupervised
  image classifiers that achieve a +8% improvement over the prior
  state-of-the-art on unsupervised classification on ImageNet-1K. We also
  demonstrate that I-Con can be used to derive principled debiasing methods which
  improve contrastive representation learners.
paper_ai_summary: A unified framework using KL divergence between supervisory and learned representations generalizes multiple machine learning loss functions and improves unsupervised image classification and debiasing.
paper_ai_keywords: information-theoretic equation KL divergence conditional distributions information geometry clustering spectral methods dimensionality reduction contrastive learning supervised learning unsupervised image classification ImageNet-1K debiasing methods