title: The Role of Entropy and Reconstruction in Multi-View Self-Supervised
    Learning
summary: The mechanisms behind the success of multi-view self-supervised learning
  (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been
  studied through the lens of InfoNCE, a lower bound of the Mutual Information
  (MI). However, the relation between other MVSSL methods and MI remains unclear.
  We consider a different lower bound on the MI consisting of an entropy and a
  reconstruction term (ER), and analyze the main MVSSL families through its lens.
  Through this ER bound, we show that clustering-based methods such as
  DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of
  distillation-based approaches such as BYOL and DINO, showing that they
  explicitly maximize the reconstruction term and implicitly encourage a stable
  entropy, and we confirm this empirically. We show that replacing the objectives
  of common MVSSL methods with this ER bound achieves competitive performance,
  while making them stable when training with smaller batch sizes or smaller
  exponential moving average (EMA) coefficients.
    Github repo: https://github.com/apple/ml-entropy-reconstruction.
paper_authors._id: 64ba0cfd9d8360edd98d83af
paper_authors.name: Borja Rodríguez-Gálvez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T09:54:59.146Z
paper_authors._id: 64ba5252997dd1af6d4dcec9
paper_authors.avatarUrl: /avatars/08ce2850253d944f98e67e99905dff54.svg
paper_authors.fullname: Borja Rodríguez Gálvez
paper_authors.type: user
paper_authors.user: burklight
paper_authors._id: 64ba0cfd9d8360edd98d83b0
paper_authors.name: Arno Blaas
paper_authors._id: 64ba0cfd9d8360edd98d83b1
paper_authors.name: Pau Rodríguez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-21T09:54:55.600Z
paper_authors._id: 64ba512141078fd9a059afaf
paper_authors.avatarUrl: /avatars/6095fa8cca5e3ae8a4699a9595fc8f71.svg
paper_authors.fullname: Pau Rodriguez
paper_authors.type: user
paper_authors.user: prlz77
paper_authors._id: 64ba0cfd9d8360edd98d83b2
paper_authors.name: Adam Goliński
paper_authors._id: 64ba0cfd9d8360edd98d83b3
paper_authors.name: Xavier Suau
paper_authors._id: 64ba0cfd9d8360edd98d83b4
paper_authors.name: Jason Ramapuram
paper_authors._id: 64ba0cfd9d8360edd98d83b5
paper_authors.name: Dan Busbridge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:50.179Z
paper_authors._id: 64c3726f2a5eaefd000cdedd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c3726f2a5eaefd000cdedd/iwFifH1sWQy7agW3eTmNQ.png
paper_authors.fullname: Dan Busbridge
paper_authors.type: user
paper_authors.user: dbusbridge
paper_authors._id: 64ba0cfd9d8360edd98d83b6
paper_authors.name: Luca Zappella
paper_title: The Role of Entropy and Reconstruction in Multi-View Self-Supervised
    Learning
paper_summary: The mechanisms behind the success of multi-view self-supervised learning
  (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been
  studied through the lens of InfoNCE, a lower bound of the Mutual Information
  (MI). However, the relation between other MVSSL methods and MI remains unclear.
  We consider a different lower bound on the MI consisting of an entropy and a
  reconstruction term (ER), and analyze the main MVSSL families through its lens.
  Through this ER bound, we show that clustering-based methods such as
  DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of
  distillation-based approaches such as BYOL and DINO, showing that they
  explicitly maximize the reconstruction term and implicitly encourage a stable
  entropy, and we confirm this empirically. We show that replacing the objectives
  of common MVSSL methods with this ER bound achieves competitive performance,
  while making them stable when training with smaller batch sizes or smaller
  exponential moving average (EMA) coefficients.
    Github repo: https://github.com/apple/ml-entropy-reconstruction.
paper_ai_keywords: multi-view self-supervised learning (MVSSL) Contrastive MVSSL InfoNCE Mutual Information (MI) Entropy and a reconstruction term (ER) DeepCluster SwAV BYOL DINO navigation term exponential moving average (EMA) coefficients