Answers.Body: <p>In the original word2vec implementation, there was an initial pass on the entire dataset to build up the vocabulary. During training the negatives are randomly sampled from the entire vocabulary. The sampling strategy matters quite a bit. If we just sample every word with equal probability, we treat rare and frequent words alike. If we sample based on their frequency, we update the vectors of the frequent words far more often. The answer is to balance the two - subsample the frequent words. The authors report metrics with different such heuristics.</p>
  <p>Recommender systems (using two tower DNN's) are usually trained using libraries like TF or Pytorch where training data is always batched. In this setting it's natural to get negatives from only within that batch. Fetching items from the entire dataset would be very very computationally inefficient. The same issue of oversampling frequent items occurs here too. Although we don't have global item frequency counts, sampling uniformly from every batch mimics sampling from the entire dataset weighted by frequency.There are approaches to correct for this.</p>
  <p>So in summary</p>
  <ol>
  <li>Sampling randomly from every batch has the same effect as sampling from the entire dataset weighted by frequency</li>
  <li>Usually it helps to subsample frequent items or words.</li>
  <li>TF Recommenders allows specifying candidate sampling probability. Check the <a href="https://www.tensorflow.org/recommenders/api_docs/python/tfrs/tasks/Retrieval" rel="nofollow noreferrer">API</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T04:14:40.457Z
Answers.LastActivityDate: 2023-02-26T04:14:40.457Z
Answers.UserDisplayName: Satyajit
Body: <p>For nlp task like word2vec, we do negative sampling through the entire dataset</p>
  <p>But in some cases like candidate generation in recommendation system, we do in batch negative sampling.</p>
  <p>So my question are:</p>
  <ol>
  <li>What's the difference between two method?</li>
  <li>Did module like <code>tensorflow</code> has already realized  <code>in batch NEG</code> and <code>globally NEG</code>?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-11T12:08:21.087Z
Comments.Text: Can you provide more details (e.g. links to references) that mention this "negative sampling" and this "batch negative sampling"?
Comments.UserDisplayName: nbro
Tags: <machine-learning><natural-language-processing><recommender-system>
Title: Negative sampling in batch or entire dataset?
UserDisplayName: IMAPOTATO
ThreadText: # Negative sampling in batch or entire dataset?
  
  (Asked by: username_0 on 2022-12-06)
  
  For nlp task like word2vec, we do negative sampling through the entire dataset
  
  But in some cases like candidate generation in recommendation system, we do in batch negative sampling.
  
  So my question are:
  
  1. What's the difference between two method?
  2. Did module like `tensorflow` has already realized `in batch NEG` and `globally NEG`?
  
  **Tags:** machine-learning, natural-language-processing, recommender-system
  
  > **Comments:**
  > **username_1**: Can you provide more details (e.g. links to references) that mention this "negative sampling" and this "batch negative sampling"?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-26. Score: 0 | Accepted answer
  
  In the original word2vec implementation, there was an initial pass on the entire dataset to build up the vocabulary. During training the negatives are randomly sampled from the entire vocabulary. The sampling strategy matters quite a bit. If we just sample every word with equal probability, we treat rare and frequent words alike. If we sample based on their frequency, we update the vectors of the frequent words far more often. The answer is to balance the two - subsample the frequent words. The authors report metrics with different such heuristics.
  
  Recommender systems (using two tower DNN's) are usually trained using libraries like TF or Pytorch where training data is always batched. In this setting it's natural to get negatives from only within that batch. Fetching items from the entire dataset would be very very computationally inefficient. The same issue of oversampling frequent items occurs here too. Although we don't have global item frequency counts, sampling uniformly from every batch mimics sampling from the entire dataset weighted by frequency.There are approaches to correct for this.
  
  So in summary
  
  1. Sampling randomly from every batch has the same effect as sampling from the entire dataset weighted by frequency
  2. Usually it helps to subsample frequent items or words.
  3. TF Recommenders allows specifying candidate sampling probability. Check the [API](https://www.tensorflow.org/recommenders/api_docs/python/tfrs/tasks/Retrieval)
  
  
  ---
  
  