title: To Compress or Not to Compress- Self-Supervised Learning and Information
    Theory: A Review
summary: Deep neural networks have demonstrated remarkable performance in supervised
  learning tasks but require large amounts of labeled data. Self-supervised
  learning offers an alternative paradigm, enabling the model to learn from data
  without explicit labels. Information theory has been instrumental in
  understanding and optimizing deep neural networks. Specifically, the
  information bottleneck principle has been applied to optimize the trade-off
  between compression and relevant information preservation in supervised
  settings. However, the optimal information objective in self-supervised
  learning remains unclear. In this paper, we review various approaches to
  self-supervised learning from an information-theoretic standpoint and present a
  unified framework that formalizes the self-supervised information-theoretic
  learning problem. We integrate existing research into a coherent framework,
  examine recent self-supervised methods, and identify research opportunities and
  challenges. Moreover, we discuss empirical measurement of information-theoretic
  quantities and their estimators. This paper offers a comprehensive review of
  the intersection between information theory, self-supervised learning, and deep
  neural networks.
paper_authors._id: 645b23661a145d2ee9bc5c1b
paper_authors.name: Ravid Shwartz-Ziv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-10T10:36:43.299Z
paper_authors._id: 6303f7333926de1f7ec454f3
paper_authors.avatarUrl: /avatars/1677a517b8635872a7a905aac727ff59.svg
paper_authors.fullname: shwartz ziv
paper_authors.type: user
paper_authors.user: ravid
paper_authors._id: 645b23661a145d2ee9bc5c1c
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T11:42:43.707Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_title: To Compress or Not to Compress- Self-Supervised Learning and Information
    Theory: A Review
paper_summary: Deep neural networks have demonstrated remarkable performance in supervised
  learning tasks but require large amounts of labeled data. Self-supervised
  learning offers an alternative paradigm, enabling the model to learn from data
  without explicit labels. Information theory has been instrumental in
  understanding and optimizing deep neural networks. Specifically, the
  information bottleneck principle has been applied to optimize the trade-off
  between compression and relevant information preservation in supervised
  settings. However, the optimal information objective in self-supervised
  learning remains unclear. In this paper, we review various approaches to
  self-supervised learning from an information-theoretic standpoint and present a
  unified framework that formalizes the self-supervised information-theoretic
  learning problem. We integrate existing research into a coherent framework,
  examine recent self-supervised methods, and identify research opportunities and
  challenges. Moreover, we discuss empirical measurement of information-theoretic
  quantities and their estimators. This paper offers a comprehensive review of
  the intersection between information theory, self-supervised learning, and deep
  neural networks.
paper_ai_summary: A unified framework integrating information theory to formalize the self-supervised learning problem is presented, exploring the intersection with deep neural networks.
paper_ai_keywords: information theory self-supervised learning information bottleneck principle information-theoretic learning problem information-theoretic quantities estimators