The Role of Entropy and Reconstruction in Multi-View Self-Supervised
  Learning
The mechanisms behind the success of multi-view self-supervised learning
(MVSSL) are not yet fully understood. Contrastive MVSSL methods have been
studied through the lens of InfoNCE, a lower bound of the Mutual Information
(MI). However, the relation between other MVSSL methods and MI remains unclear.
We consider a different lower bound on the MI consisting of an entropy and a
reconstruction term (ER), and analyze the main MVSSL families through its lens.
Through this ER bound, we show that clustering-based methods such as
DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of
distillation-based approaches such as BYOL and DINO, showing that they
explicitly maximize the reconstruction term and implicitly encourage a stable
entropy, and we confirm this empirically. We show that replacing the objectives
of common MVSSL methods with this ER bound achieves competitive performance,
while making them stable when training with smaller batch sizes or smaller
exponential moving average (EMA) coefficients.
  Github repo: https://github.com/apple/ml-entropy-reconstruction.
64ba0cfd9d8360edd98d83af
Borja Rodríguez-Gálvez
claimed_verified
2023-07-21T09:54:59.146Z
64ba5252997dd1af6d4dcec9
/avatars/08ce2850253d944f98e67e99905dff54.svg
Borja Rodríguez Gálvez
user
burklight
64ba0cfd9d8360edd98d83b0
Arno Blaas
64ba0cfd9d8360edd98d83b1
Pau Rodríguez
claimed_verified
2023-07-21T09:54:55.600Z
64ba512141078fd9a059afaf
/avatars/6095fa8cca5e3ae8a4699a9595fc8f71.svg
Pau Rodriguez
user
prlz77
64ba0cfd9d8360edd98d83b2
Adam Goliński
64ba0cfd9d8360edd98d83b3
Xavier Suau
64ba0cfd9d8360edd98d83b4
Jason Ramapuram
64ba0cfd9d8360edd98d83b5
Dan Busbridge
claimed_verified
2025-02-13T20:36:50.179Z
64c3726f2a5eaefd000cdedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64c3726f2a5eaefd000cdedd/iwFifH1sWQy7agW3eTmNQ.png
Dan Busbridge
user
dbusbridge
64ba0cfd9d8360edd98d83b6
Luca Zappella
The Role of Entropy and Reconstruction in Multi-View Self-Supervised
  Learning
The mechanisms behind the success of multi-view self-supervised learning
(MVSSL) are not yet fully understood. Contrastive MVSSL methods have been
studied through the lens of InfoNCE, a lower bound of the Mutual Information
(MI). However, the relation between other MVSSL methods and MI remains unclear.
We consider a different lower bound on the MI consisting of an entropy and a
reconstruction term (ER), and analyze the main MVSSL families through its lens.
Through this ER bound, we show that clustering-based methods such as
DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of
distillation-based approaches such as BYOL and DINO, showing that they
explicitly maximize the reconstruction term and implicitly encourage a stable
entropy, and we confirm this empirically. We show that replacing the objectives
of common MVSSL methods with this ER bound achieves competitive performance,
while making them stable when training with smaller batch sizes or smaller
exponential moving average (EMA) coefficients.
  Github repo: https://github.com/apple/ml-entropy-reconstruction.
A new entropy and reconstruction (ER) bound is introduced to analyze multi-view self-supervised learning methods, showing its relation to clustering and distillation approaches, and demonstrating competitive performance with enhanced stability.
multi-view self-supervised learning (MVSSL) InfoNCE Mutual Information (MI) entropy reconstruction term (ER) DeepCluster SwAV BYOL DINO exponential moving average (EMA)