Answers.Body: <p>Yes, the In-batch negative training improves the results but not objectively, it is more helpful in reducing the confusion of semantics of various pattern which I guess tends to happen in deep-layers.</p>
  <p>In the context of Blip-2, By including these hard negative samples in the training batch, the model is forced to learn to distinguish between positive and negative samples, which was helpful in Visual Knowledge Reasoning, but it didn't improve accuracy significantly.</p>
  <p>For your last question regarding the link to gradient, yes it has to do with gradient, I guess.</p>
  <pre><code>        # sample way to generate hard negative samples
          with torch.no_grad():
              outputs = model(inputs)
              _, predicted = torch.max(outputs.data, 1)
              negatives = inputs[predicted != labels]
  
          #  hard negative samples with the positive samples
          inputs = torch.cat((inputs, negatives), dim=0)
          labels = torch.cat((labels, torch.zeros(negatives.shape[0], dtype=torch.long)), dim=0)
  
          # Zero the parameter gradients
          optimizer.zero_grad()
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T08:16:18.147Z
Answers.LastActivityDate: 2023-04-04T07:37:28.843Z
Answers.LastEditDate: 2023-04-04T07:37:28.843Z
Answers.UserDisplayName: TheExorcist
Body: <p>I have read <a href="https://arxiv.org/abs/2004.04906" rel="nofollow noreferrer">Dense passage retrieval for Open Domain Question Answering</a>, and in page 6 it talks about in-batch negative training, it states the following:</p>
  <blockquote>
  <p>We find that using a similar
  configuration (7 gold negative passages), in-batch
  negative training improves the results substantially.
  The key difference between the two is whether the
  gold negative passages come from the same batch
  or from the whole training set. Effectively, in-batch
  negative training is an easy and memory-efficient
  way to reuse the negative examples already in the
  batch rather than creating new ones. It produces
  more pairs and thus increases the number of train-
  ing examples, which might contribute to the good
  model performance. As a result, accuracy consis-
  tently improves as the batch size grows.</p>
  </blockquote>
  <p>I know that it is more efficient and hence we can increase the number of negatives by increasing the batch size (or we can use other techniques if we cannot do that), but in the paper they have the following table:</p>
  <p><a href="https://i.sstatic.net/G868r.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/G868r.png" alt="table 3" /></a></p>
  <p>How having the same number of negatives (3rd &amp; 4th rows) gave different results?</p>
  <p>Is that something realted to the gradient or something, since we sample from the same examples that contribute to calculating the gradient in one step?</p>
  
Tags: <machine-learning><natural-language-processing><papers><question-answering>
Title: In-batch negative training Improves the results
UserDisplayName: Kais Hasan
ThreadText: # In-batch negative training Improves the results
  
  (Asked by: username_0 on 2022-06-30)
  
  I have read [Dense passage retrieval for Open Domain Question Answering](https://arxiv.org/abs/2004.04906), and in page 6 it talks about in-batch negative training, it states the following:
  
  > We find that using a similar
  > configuration (7 gold negative passages), in-batch
  > negative training improves the results substantially.
  > The key difference between the two is whether the
  > gold negative passages come from the same batch
  > or from the whole training set. Effectively, in-batch
  > negative training is an easy and memory-efficient
  > way to reuse the negative examples already in the
  > batch rather than creating new ones. It produces
  > more pairs and thus increases the number of train-
  > ing examples, which might contribute to the good
  > model performance. As a result, accuracy consis-
  > tently improves as the batch size grows.
  
  I know that it is more efficient and hence we can increase the number of negatives by increasing the batch size (or we can use other techniques if we cannot do that), but in the paper they have the following table:
  
  [![table 3](https://i.sstatic.net/G868r.png)](https://i.sstatic.net/G868r.png)
  
  How having the same number of negatives (3rd & 4th rows) gave different results?
  
  Is that something realted to the gradient or something, since we sample from the same examples that contribute to calculating the gradient in one step?
  
  **Tags:** machine-learning, natural-language-processing, papers, question-answering
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-31. Score: 0
  
  Yes, the In-batch negative training improves the results but not objectively, it is more helpful in reducing the confusion of semantics of various pattern which I guess tends to happen in deep-layers.
  In the context of Blip-2, By including these hard negative samples in the training batch, the model is forced to learn to distinguish between positive and negative samples, which was helpful in Visual Knowledge Reasoning, but it didn't improve accuracy significantly.
  For your last question regarding the link to gradient, yes it has to do with gradient, I guess.
          # sample way to generate hard negative samples
          with torch.no_grad():
              outputs = model(inputs)
              _, predicted = torch.max(outputs.data, 1)
              negatives = inputs[predicted != labels]
  
          #  hard negative samples with the positive samples
          inputs = torch.cat((inputs, negatives), dim=0)
          labels = torch.cat((labels, torch.zeros(negatives.shape[0], dtype=torch.long)), dim=0)
  
          # Zero the parameter gradients
          optimizer.zero_grad()
  
  
  ---
  
  