To Compress or Not to Compress- Self-Supervised Learning and Information
  Theory: A Review
Deep neural networks have demonstrated remarkable performance in supervised
learning tasks but require large amounts of labeled data. Self-supervised
learning offers an alternative paradigm, enabling the model to learn from data
without explicit labels. Information theory has been instrumental in
understanding and optimizing deep neural networks. Specifically, the
information bottleneck principle has been applied to optimize the trade-off
between compression and relevant information preservation in supervised
settings. However, the optimal information objective in self-supervised
learning remains unclear. In this paper, we review various approaches to
self-supervised learning from an information-theoretic standpoint and present a
unified framework that formalizes the self-supervised information-theoretic
learning problem. We integrate existing research into a coherent framework,
examine recent self-supervised methods, and identify research opportunities and
challenges. Moreover, we discuss empirical measurement of information-theoretic
quantities and their estimators. This paper offers a comprehensive review of
the intersection between information theory, self-supervised learning, and deep
neural networks.
645b23661a145d2ee9bc5c1b
Ravid Shwartz-Ziv
claimed_verified
2023-05-10T10:36:43.299Z
6303f7333926de1f7ec454f3
/avatars/1677a517b8635872a7a905aac727ff59.svg
shwartz ziv
user
ravid
645b23661a145d2ee9bc5c1c
Yann LeCun
admin_assigned
2023-11-23T11:42:43.707Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
To Compress or Not to Compress- Self-Supervised Learning and Information
  Theory: A Review
Deep neural networks have demonstrated remarkable performance in supervised
learning tasks but require large amounts of labeled data. Self-supervised
learning offers an alternative paradigm, enabling the model to learn from data
without explicit labels. Information theory has been instrumental in
understanding and optimizing deep neural networks. Specifically, the
information bottleneck principle has been applied to optimize the trade-off
between compression and relevant information preservation in supervised
settings. However, the optimal information objective in self-supervised
learning remains unclear. In this paper, we review various approaches to
self-supervised learning from an information-theoretic standpoint and present a
unified framework that formalizes the self-supervised information-theoretic
learning problem. We integrate existing research into a coherent framework,
examine recent self-supervised methods, and identify research opportunities and
challenges. Moreover, we discuss empirical measurement of information-theoretic
quantities and their estimators. This paper offers a comprehensive review of
the intersection between information theory, self-supervised learning, and deep
neural networks.
A unified framework integrating information theory to formalize the self-supervised learning problem is presented, exploring the intersection with deep neural networks.
information theory self-supervised learning information bottleneck principle information-theoretic learning problem information-theoretic quantities estimators