Answers.Body: <p>To train the model, you take <span class="math-container">$N$</span> samples then build <span class="math-container">$N$</span> pairs by means of applying two different augmentations. So the total num. of samples is <span class="math-container">$2N$</span>. Now, you take one pair <span class="math-container">$(i,j)$</span> and consider that to be <em>positive</em>. The remaining <span class="math-container">$N-1$</span> pairs (so <span class="math-container">$2(N-1)=2N-2$</span> samples) are <em>negatives</em>.</p>
  <p>For each positive pair <span class="math-container">$(i,j)$</span> you compute the similarity of the positive pair at the numerator, and normalize it by the sum of similarities between the anchor <span class="math-container">$i$</span> and the negative <span class="math-container">$k$</span>: if you reason on pairs, instead of single samples, you have that <span class="math-container">$k\neq j$</span>. Maybe in the paper is not so clear, but <span class="math-container">$i$</span> and <span class="math-container">$j$</span> are positive because they are two different views of the same input sample (say <span class="math-container">$x$</span>), instead <span class="math-container">$(i, k)$</span> and <span class="math-container">$(k, j)$</span> are negatives because <span class="math-container">$k$</span> is a view from another input (say <span class="math-container">$x'$</span>, where <span class="math-container">$x\neq x'$</span>.) So, when computing the denominator you do <em>not</em> include <span class="math-container">$j$</span>. I found a <a href="https://theaisummer.com/simclr/" rel="nofollow noreferrer">blog post</a> that explains that too.</p>
  <p>Lastly, the loss is also computed on <span class="math-container">$(j,i)$</span> pairs, so you sum it and divide by 2 times the batch size <span class="math-container">$N$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-30T15:56:38.837Z
Answers.Text: mmmh ok but to have normalized probabilities shouldn't I also include the index $i$ itself? Thanks for your answer, but could you elaborate a bit more please? I would like to have this point clear.
Answers.UserDisplayName: James Arten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-30T17:54:57.833Z
Answers.Text: @JamesArten I've edited my answer, I hope is clearer now
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-18T14:30:39.900Z
Answers.Text: `For the denominator we need both the positive and negative pairs.`
  The blog post you cited explicitly says this, where did you read that you do _not_ include j in the denominator?
Answers.UserDisplayName: Doch88
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-30T15:17:40.027Z
Answers.LastActivityDate: 2023-05-31T06:51:14.093Z
Answers.LastEditDate: 2023-05-31T06:51:14.093Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>If you look at this part: <span class="math-container">$$\frac{\exp(sim(z_i,z_j)/\tau)}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq i]} \exp(sim(z_i,z_k)/\tau)}$$</span>
  you will notice that it's similar to:
  <span class="math-container">$$\frac{\exp(x_i)}{\sum_{i=1}^{N}\exp(x_j)}$$</span>
  Which is basically a softmax function.
  It's essentially a way to normalize the loss function so that all the probabilities sum up to 1. In the SimCLR case, you only have to exclude k=i, because of course the similarity of a sample with itself will always be 1.</p>
  <p>Essentially this normalization is the reason why the positive samples (similarity close to 1.0) are pushed away from negative samples (similarity close to 0).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-18T15:38:49.007Z
Answers.LastActivityDate: 2023-10-19T07:24:23.937Z
Answers.LastEditDate: 2023-10-19T07:24:23.937Z
Answers.UserDisplayName: Doch88
Body: <p>I'm trying to understand few details about NT-Xent loss defined in SimCLR paper(<a href="https://arxiv.org/abs/2002.05709" rel="nofollow noreferrer">link</a>). The loss is defined as</p>
  <p><span class="math-container">$$\mathcal{l}_{i,j} = -\log\frac{\exp(sim(z_i,z_j)/\tau)}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq i]} \exp(sim(z_i,z_k)/\tau)}$$</span></p>
  <p>Where <span class="math-container">$z_i$</span> and <span class="math-container">$z_j$</span> represent two augmentations for the same image. What I don't understand is: at the denominator, I understand that we want to exclude the point <span class="math-container">$z_i$</span> using the indicator function but shouldn't we exclude also <span class="math-container">$z_j$</span>? Otherwise we will have <span class="math-container">$k=j$</span> for some <span class="math-container">$k$</span>. Essentially, why we do let the positive sample at the denominator?</p>
  
Tags: <self-supervised-learning><contrastive-learning>
Title: Definition of negatives in NT-Xent loss
UserDisplayName: James Arten
ThreadText: # Definition of negatives in NT-Xent loss
  
  (Asked by: username_0 on 2023-05-30)
  
  I'm trying to understand few details about NT-Xent loss defined in SimCLR paper([link](https://arxiv.org/abs/2002.05709)). The loss is defined as
  
  $$\mathcal{l}\_{i,j} = -\log\frac{\exp(sim(z\_i,z\_j)/\tau)}{\sum\_{k=1}^{2N}\mathbb{1}\_{[k\neq i]} \exp(sim(z\_i,z\_k)/\tau)}$$
  
  Where $z\_i$ and $z\_j$ represent two augmentations for the same image. What I don't understand is: at the denominator, I understand that we want to exclude the point $z\_i$ using the indicator function but shouldn't we exclude also $z\_j$? Otherwise we will have $k=j$ for some $k$. Essentially, why we do let the positive sample at the denominator?
  
  **Tags:** self-supervised-learning, contrastive-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-05-30. Score: 3
  
  To train the model, you take $N$ samples then build $N$ pairs by means of applying two different augmentations. So the total num. of samples is $2N$. Now, you take one pair $(i,j)$ and consider that to be *positive*. The remaining $N-1$ pairs (so $2(N-1)=2N-2$ samples) are *negatives*.
  
  For each positive pair $(i,j)$ you compute the similarity of the positive pair at the numerator, and normalize it by the sum of similarities between the anchor $i$ and the negative $k$: if you reason on pairs, instead of single samples, you have that $k\neq j$. Maybe in the paper is not so clear, but $i$ and $j$ are positive because they are two different views of the same input sample (say $x$), instead $(i, k)$ and $(k, j)$ are negatives because $k$ is a view from another input (say $x'$, where $x\neq x'$.) So, when computing the denominator you do *not* include $j$. I found a [blog post](https://theaisummer.com/simclr/) that explains that too.
  
  Lastly, the loss is also computed on $(j,i)$ pairs, so you sum it and divide by 2 times the batch size $N$.
  
  > **Comments:**
  > **username_0**: mmmh ok but to have normalized probabilities shouldn't I also include the index $i$ itself? Thanks for your answer, but could you elaborate a bit more please? I would like to have this point clear.  
  > **username_1**: @username_0 I've edited my answer, I hope is clearer now  
  > **username_2**: `For the denominator we need both the positive and negative pairs.` The blog post you cited explicitly says this, where did you read that you do _not_ include j in the denominator?  
  
  ---
  
  ### Answer by username_2 on 2023-10-18. Score: 0
  
  If you look at this part: $$\frac{\exp(sim(z\_i,z\_j)/\tau)}{\sum\_{k=1}^{2N}\mathbb{1}\_{[k\neq i]} \exp(sim(z\_i,z\_k)/\tau)}$$
  you will notice that it's similar to:
  $$\frac{\exp(x\_i)}{\sum\_{i=1}^{N}\exp(x\_j)}$$
  Which is basically a softmax function.
  It's essentially a way to normalize the loss function so that all the probabilities sum up to 1. In the SimCLR case, you only have to exclude k=i, because of course the similarity of a sample with itself will always be 1.
  
  Essentially this normalization is the reason why the positive samples (similarity close to 1.0) are pushed away from negative samples (similarity close to 0).
  
  
  ---
  
  