Answers.Body: <p><strong>Contrastive learning</strong> is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a <strong>dictionary look-up</strong> problem.</p>
  <p>If I conceptually compare the loss mechanisms for:</p>
  <ul>
  <li>SimCLR: A Simple Framework for Contrastive Learning of Visual Representations @ <a href="https://arxiv.org/abs/2002.05709" rel="nofollow noreferrer">https://arxiv.org/abs/2002.05709</a></li>
  <li>MoCo: Momentum Contrast for Unsupervised Visual Representation Learning @ <a href="https://arxiv.org/abs/1911.05722" rel="nofollow noreferrer">https://arxiv.org/abs/1911.05722</a></li>
  </ul>
  <p>Both <strong>MoCo</strong> and <strong>SimCLR</strong> use varients of a contrastive loss function, like InfoNCE from the paper <em>Representation Learning with Contrastive Predictive Coding</em>
  <span class="math-container">\begin{eqnarray*}
  \mathcal{L}_{q,k^+,\{k^-\}}=-log\frac{exp(q\cdot k^+/\tau)}{exp(q\cdot k^+/\tau)+\sum\limits_{k^-}exp(q\cdot k^-/\tau)}
  \end{eqnarray*}</span></p>
  <p>Here q is a query representation, <span class="math-container">$k^+$</span> is a representation of the positive (similar) key sample, and <span class="math-container">${k^−}$</span> are representations of the negative (dissimilar) key samples. <span class="math-container">$\tau$</span> is a temperature hyper-parameter. In the instance discrimination pretext task (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.
  <a href="https://i.sstatic.net/7F6DD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7F6DD.png" alt="SimCLR vs MoCo" /></a></p>
  <p>The contrastive loss can be minimized by various mechanisms that differ in <strong>how the keys are maintained</strong>.</p>
  <p>In an <strong>end-to-end</strong> mechanism (Fig. 1a), the negative keys are from the same batch and updated end-to-end by back-propagation. <strong>SimCLR</strong>, is based on this mechanism and requires a large batch to provide a large set of negatives.</p>
  <p>In the <strong>MoCo</strong> mechanism i.e. <strong>Momentum Contrast</strong> (Fig. 1b), the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-19T21:45:58.350Z
Answers.LastActivityDate: 2020-06-19T21:51:50.307Z
Answers.LastEditDate: 2020-06-19T21:51:50.307Z
Answers.UserDisplayName: Mayank Bhaskar
Body: <p>Recently, I have seen a surge of papers w.r.t contrastive learning (a subset of semi-supervised learning). </p>
  
  <p>Can anyone give a detailed explanation of this approach with its advantages/disadvantages and what are the cases in which it gives better results?</p>
  
  <p>Also, why it's gaining traction amongst the ML research community?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-08T15:15:49.650Z
Comments.Text: As I understand it contrastive learning is a type of data augmentation. For classification you have labeled dataset, two augmentation of the same sample should have the same label. Idea of contrastive learning: not only   two different augmentation of the same sample should have the same label, but the label should be different from the sample augmented from different label. Contrastive learning use loss which encourage that difference (therefore "contrastive") Advantages and disadvantages are the same as for common augmentation. Based on https://arxiv.org/abs/2002.05709
Comments.UserDisplayName: mirror2image
Tags: <machine-learning><deep-learning><comparison><semi-supervised-learning>
Title: What's the intuition behind contrastive learning?
UserDisplayName: Mayank Bhaskar
ThreadText: # What's the intuition behind contrastive learning?
  
  (Asked by: username_0 on 2020-03-07)
  
  Recently, I have seen a surge of papers w.r.t contrastive learning (a subset of semi-supervised learning).
  
  Can anyone give a detailed explanation of this approach with its advantages/disadvantages and what are the cases in which it gives better results?
  
  Also, why it's gaining traction amongst the ML research community?
  
  **Tags:** machine-learning, deep-learning, comparison, semi-supervised-learning
  
  > **Comments:**
  > **username_1**: As I understand it contrastive learning is a type of data augmentation. For classification you have labeled dataset, two augmentation of the same sample should have the same label. Idea of contrastive learning: not only   two different augmentation of the same sample should have the same label, but the label should be different from the sample augmented from different label. Contrastive learning use loss which encourage that difference (therefore "contrastive") Advantages and disadvantages are the same as for common augmentation. Based on https://arxiv.org/abs/2002.05709  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-06-19. Score: 1
  
  **Contrastive learning** is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a **dictionary look-up** problem.
  
  If I conceptually compare the loss mechanisms for:
  
  * SimCLR: A Simple Framework for Contrastive Learning of Visual Representations @ <https://arxiv.org/abs/2002.05709>
  * MoCo: Momentum Contrast for Unsupervised Visual Representation Learning @ <https://arxiv.org/abs/1911.05722>
  
  Both **MoCo** and **SimCLR** use varients of a contrastive loss function, like InfoNCE from the paper *Representation Learning with Contrastive Predictive Coding*
  \begin{eqnarray\*}
  \mathcal{L}\_{q,k^+,\{k^-\}}=-log\frac{exp(q\cdot k^+/\tau)}{exp(q\cdot k^+/\tau)+\sum\limits\_{k^-}exp(q\cdot k^-/\tau)}
  \end{eqnarray\*}
  
  Here q is a query representation, $k^+$ is a representation of the positive (similar) key sample, and ${k^−}$ are representations of the negative (dissimilar) key samples. $\tau$ is a temperature hyper-parameter. In the instance discrimination pretext task (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair.
  [![SimCLR vs MoCo](https://i.sstatic.net/7F6DD.png)](https://i.sstatic.net/7F6DD.png)
  
  The contrastive loss can be minimized by various mechanisms that differ in **how the keys are maintained**.
  
  In an **end-to-end** mechanism (Fig. 1a), the negative keys are from the same batch and updated end-to-end by back-propagation. **SimCLR**, is based on this mechanism and requires a large batch to provide a large set of negatives.
  
  In the **MoCo** mechanism i.e. **Momentum Contrast** (Fig. 1b), the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch.
  
  
  ---
  
  