Answers.Body: <p>If you look at the words in your dictionary (vocab) before/after pruning, most likely you'd see there isn't a lot of difference, not so much to affect your model performance. </p>
  
  <p>In fact, creating a dictionary and model training are two more or less indpendent processes. To make your life easier, you could find the largest dev set you can find for building your vocab (excluding test set), and freeze it for all subsequent ETL/modeling.  This way you don't have to deal with dictionary versioning, for example, after choosing different subsets of your training data. </p>
  
  <p>Also if you have compute capacity, I'd suggest to upsample positive/negative classes instead, because those neutral samples you're dropping do have signal on the use of language and which borderline ambiguous samples should be treated as neutral. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T09:56:53.107Z
Answers.Text: Thanks for this response! I never thought about just finding a large dev set but that makes a lot of sense. Could you explain what you mean by upsampling though? Thanks!
Answers.UserDisplayName: Nore
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T10:30:03.693Z
Answers.Text: Sure. So say your class distribution is `pos=5, neg=5, neutral=10`, you do sampling with replacement from `pos` and `neg` until you have 10 samples for all classes. In pandas you can easily do this with `df.sample` and if you do a bit of search you'll find other ways/libs to help you with that.
Answers.UserDisplayName: olix20
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T10:32:01.323Z
Answers.Text: Ah okay, so this would create duplicate sentences in the training correct? Have you also heard of the `class_weights` parameter of most sklearn models? It seems to balance classes by assigning weights. Someone else pointed it out to me and I think it does what I want but much nicer. Wanted to ask if you knew it as well? Thanks!
Answers.UserDisplayName: Nore
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T12:05:40.967Z
Answers.Text: yes correct. Indeed an alternative is `class_weights`, but depends if the model you want to use implements it.
Answers.UserDisplayName: olix20
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T07:05:12.023Z
Answers.Text: Hey I had a further ponder over finding a large dev set but something occurred to me. Wouldn't this encode various words that exist in the dev set and testing set but not in the training set? I'm wondering what the point would be of keeping those words in the test set if the classifier isn't even trained on them. At the moment, when I use BoW I drop the words not in the training set from the test set. What's the intuition behind keeping them using a dev set? Would they just be ignored by the classifier anyway? Or rather, how do trained classifiers deal with those kinds of words? Thanks!
Answers.UserDisplayName: Nore
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T14:11:31.707Z
Answers.Text: You don't really have to go the extra mile for finding the largest dev set. Using your raw training set (tokenized) would be enough. Your observation is currect, when train corpus doens't contain a word, there will be no signal learned from that single word. In practice this isn't a problem, because you look at more than one word for making a decision (n-grams, conv nets, RNNs, transformers, etc). If a word doesn't appear in the training set, most likely it's not significant enough (rare) to have an effect. In practice a vocab of 30k-50k words is enough to make a decent classifier.
Answers.UserDisplayName: olix20
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T14:15:37.907Z
Answers.Text: You freeze the vocab mostly for convenience. You shouldn't be doing tokenization and vocab building by hand. Most of the tools out there will take care of picking the most common N words, will tokenize the text properly, and map any unseen (unknown) word to a special `<unk>` token. That will be the placeholder to represent all rare/out-of-vocabulary (oov) tokens and your model will also learn to play nicely with it (usually treat it as noise).
Answers.UserDisplayName: olix20
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T08:26:21.530Z
Answers.LastActivityDate: 2020-04-07T08:26:21.530Z
Answers.UserDisplayName: olix20
Body: <p>My question is about when to balance training data for sentiment analysis. </p>
  
  <p>Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral labels as the other 2 combined, so I used a function to drop neutral labels randomly. </p>
  
  <p>However, I wasn't sure if I should do this before or after creating the <code>vocab2index</code> mappings. </p>
  
  <p>To explain, I am numericizing my text data by creating a vocabulary of words in the training data and linking them to numbers using enumerate. I think to use that dictionary of <code>vocab2index</code> values to numericise the training data. I also use that same dictionary to numericise the testing data, dropping any words that do not exist in the dictionary. </p>
  
  <p>When I took a class on this, they had balanced the training data AFTER creating the vocab2index dictionary. However, when I thought about this in my own implementation, it did not make sense. What if some words from the original vocabulary are gone completely, then we aren't training the machine learning classifier on those words, but they would not be dropping from the testing data either (since words are dropping from <code>X_test</code> based on whether they are in the <code>vocab2index</code> dictionary). </p>
  
  <p>So should I be balancing the data BEFORE creating the vocab2index dictionary? </p>
  
  <p>I linked the code to create <code>X_train</code> and <code>X_test</code> below in case it help.</p>
  
  <pre><code>def create_X_train(training_data='Sentences_75Agree_csv.csv'):
      data_csv = pd.read_csv(filepath_or_buffer=training_data, sep='.@', header=None, names=['sentence','sentiment'], engine='python')
      list_data = []
      for index, row in data_csv.iterrows():
          dictionary_data = {}
          dictionary_data['message_body'] = row['sentence']
          if row['sentiment'] == 'positive':
               dictionary_data['sentiment'] = 2
          elif row['sentiment'] == 'negative':
               dictionary_data['sentiment'] = 0
          else:
               dictionary_data['sentiment'] = 1 # For neutral sentiment
          list_data.append(dictionary_data)
      dictionary_data = {}
      dictionary_data['data'] = list_data
      messages = [sentence['message_body'] for sentence in dictionary_data['data']]
      sentiments = [sentence['sentiment'] for sentence in dictionary_data['data']]
  
      tokenized = [preprocess(sentence) for sentence in messages]
      bow = Counter([word for sentence in tokenized for word in sentence]) 
      freqs = {key: value/len(tokenized) for key, value in bow.items()} #keys are the words in the vocab, values are the count of those words
  
      # Removing 5 most common words from data
      high_cutoff = 5
      K_most_common = [x[0] for x in bow.most_common(high_cutoff)] 
      filtered_words = [word for word in freqs if word not in K_most_common]
  
      # Create vocab2index dictionary:
      vocab = {word: i for i, word in enumerate(filtered_words, 1)}
      id2vocab = {i: word for word, i in vocab.items()}
      filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] 
  
      # Balancing training data due to large number of neutral sentences
      balanced = {'messages': [], 'sentiments':[]}
      n_neutral = sum(1 for each in sentiments if each == 1)
      N_examples = len(sentiments)
      # print(n_neutral/N_examples)
      keep_prob = (N_examples - n_neutral)/2/n_neutral
      # print(keep_prob)
      for idx, sentiment in enumerate(sentiments):
          message = filtered[idx]
          if len(message) == 0:
              # skip this sentence because it has length 0
              continue
          elif sentiment != 1 or random.random() &lt; keep_prob:
              balanced['messages'].append(message)
              balanced['sentiments'].append(sentiment)
  
      token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
      sentiments_balanced = balanced['sentiments']
  
      # Unit test:
      unique, counts = np.unique(sentiments_balanced, return_counts=True)
      print(np.asarray((unique, counts)).T)
      print(np.mean(sentiments_balanced))
      ##################
  
      # Left padding and truncating to the same length 
      X_train = token_ids
      for i, sentence in enumerate(X_train):
          if len(sentence) &lt;=30:
              X_train[i] = ((30-len(sentence)) * [0] + sentence)
          elif len(sentence) &gt; 30:
              X_train[i] = sentence[:30]
      return vocab, X_train, sentiments_balanced
  </code></pre>
  
  <pre><code>def create_X_test(test_sentences, vocab):
      tokenized = [preprocess(sentence) for sentence in test_sentences]
      filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] # X_test filtered to only words in training vocab
      # Alternate method with functional programming:
      # filtered = [list(filter(lambda a: a in vocab, sentence)) for sentence in tokenized]
      token_ids = [[vocab[word] for word in sentence] for sentence in filtered] # Numericise data
  
      # Remove short sentences in X_test
      token_ids_filtered = [sentence for sentence in token_ids if len(sentence)&gt;10]
      X_test = token_ids_filtered
      for i, sentence in enumerate(X_test):
          if len(sentence) &lt;=30:
              X_test[i] = ((30-len(sentence)) * [0] + sentence)
          elif len(sentence) &gt; 30:
              X_test[i] = sentence[:30]
      return X_test
  </code></pre>
  
Tags: <natural-language-processing><python><training><sentiment-analysis>
Title: Should I be balancing the data before creating the vocab-to-index dictionary?
UserDisplayName: Nore
ThreadText: # Should I be balancing the data before creating the vocab-to-index dictionary?
  
  (Asked by: username_0 on 2020-04-07)
  
  My question is about when to balance training data for sentiment analysis. 
  Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral labels as the other 2 combined, so I used a function to drop neutral labels randomly. 
  However, I wasn't sure if I should do this before or after creating the vocab2index mappings. 
  To explain, I am numericizing my text data by creating a vocabulary of words in the training data and linking them to numbers using enumerate. I think to use that dictionary of vocab2index values to numericise the training data. I also use that same dictionary to numericise the testing data, dropping any words that do not exist in the dictionary. 
  When I took a class on this, they had balanced the training data AFTER creating the vocab2index dictionary. However, when I thought about this in my own implementation, it did not make sense. What if some words from the original vocabulary are gone completely, then we aren't training the machine learning classifier on those words, but they would not be dropping from the testing data either (since words are dropping from X_test based on whether they are in the vocab2index dictionary). 
  So should I be balancing the data BEFORE creating the vocab2index dictionary? 
  I linked the code to create X_train and X_test below in case it help.
  def create_X_train(training_data='Sentences_75Agree_csv.csv'):
      data_csv = pd.read_csv(filepath_or_buffer=training_data, sep='.@', header=None, names=['sentence','sentiment'], engine='python')
      list_data = []
      for index, row in data_csv.iterrows():
          dictionary_data = {}
          dictionary_data['message_body'] = row['sentence']
          if row['sentiment'] == 'positive':
               dictionary_data['sentiment'] = 2
          elif row['sentiment'] == 'negative':
               dictionary_data['sentiment'] = 0
          else:
               dictionary_data['sentiment'] = 1 # For neutral sentiment
          list_data.append(dictionary_data)
      dictionary_data = {}
      dictionary_data['data'] = list_data
      messages = [sentence['message_body'] for sentence in dictionary_data['data']]
      sentiments = [sentence['sentiment'] for sentence in dictionary_data['data']]
  
      tokenized = [preprocess(sentence) for sentence in messages]
      bow = Counter([word for sentence in tokenized for word in sentence]) 
      freqs = {key: value/len(tokenized) for key, value in bow.items()} #keys are the words in the vocab, values are the count of those words
  
      # Removing 5 most common words from data
      high_cutoff = 5
      K_most_common = [x[0] for x in bow.most_common(high_cutoff)] 
      filtered_words = [word for word in freqs if word not in K_most_common]
  
      # Create vocab2index dictionary:
      vocab = {word: i for i, word in enumerate(filtered_words, 1)}
      id2vocab = {i: word for word, i in vocab.items()}
      filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] 
  
      # Balancing training data due to large number of neutral sentences
      balanced = {'messages': [], 'sentiments':[]}
      n_neutral = sum(1 for each in sentiments if each == 1)
      N_examples = len(sentiments)
      # print(n_neutral/N_examples)
      keep_prob = (N_examples - n_neutral)/2/n_neutral
      # print(keep_prob)
      for idx, sentiment in enumerate(sentiments):
          message = filtered[idx]
          if len(message) == 0:
              # skip this sentence because it has length 0
              continue
          elif sentiment != 1 or random.random() < keep_prob:
              balanced['messages'].append(message)
              balanced['sentiments'].append(sentiment)
  
      token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
      sentiments_balanced = balanced['sentiments']
  
      # Unit test:
      unique, counts = np.unique(sentiments_balanced, return_counts=True)
      print(np.asarray((unique, counts)).T)
      print(np.mean(sentiments_balanced))
      ##################
  
      # Left padding and truncating to the same length 
      X_train = token_ids
      for i, sentence in enumerate(X_train):
          if len(sentence) <=30:
              X_train[i] = ((30-len(sentence)) * [0] + sentence)
          elif len(sentence) > 30:
              X_train[i] = sentence[:30]
      return vocab, X_train, sentiments_balanced
  
  def create_X_test(test_sentences, vocab):
      tokenized = [preprocess(sentence) for sentence in test_sentences]
      filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] # X_test filtered to only words in training vocab
      # Alternate method with functional programming:
      # filtered = [list(filter(lambda a: a in vocab, sentence)) for sentence in tokenized]
      token_ids = [[vocab[word] for word in sentence] for sentence in filtered] # Numericise data
  
      # Remove short sentences in X_test
      token_ids_filtered = [sentence for sentence in token_ids if len(sentence)>10]
      X_test = token_ids_filtered
      for i, sentence in enumerate(X_test):
          if len(sentence) <=30:
              X_test[i] = ((30-len(sentence)) * [0] + sentence)
          elif len(sentence) > 30:
              X_test[i] = sentence[:30]
      return X_test
  
  **Tags:** natural-language-processing, python, training, sentiment-analysis
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-07. Score: 0 | Accepted answer
  
  If you look at the words in your dictionary (vocab) before/after pruning, most likely you'd see there isn't a lot of difference, not so much to affect your model performance.
  
  In fact, creating a dictionary and model training are two more or less indpendent processes. To make your life easier, you could find the largest dev set you can find for building your vocab (excluding test set), and freeze it for all subsequent ETL/modeling. This way you don't have to deal with dictionary versioning, for example, after choosing different subsets of your training data.
  
  Also if you have compute capacity, I'd suggest to upsample positive/negative classes instead, because those neutral samples you're dropping do have signal on the use of language and which borderline ambiguous samples should be treated as neutral.
  
  > **Comments:**
  > **username_0**: Thanks for this response! I never thought about just finding a large dev set but that makes a lot of sense. Could you explain what you mean by upsampling though? Thanks!  
  > **username_1**: Sure. So say your class distribution is `pos=5, neg=5, neutral=10`, you do sampling with replacement from `pos` and `neg` until you have 10 samples for all classes. In pandas you can easily do this with `df.sample` and if you do a bit of search you'll find other ways/libs to help you with that.  
  > **username_0**: Ah okay, so this would create duplicate sentences in the training correct? Have you also heard of the `class_weights` parameter of most sklearn models? It seems to balance classes by assigning weights. Someone else pointed it out to me and I think it does what I want but much nicer. Wanted to ask if you knew it as well? Thanks!  
  > **username_1**: yes correct. Indeed an alternative is `class_weights`, but depends if the model you want to use implements it.  
  > **username_0**: Hey I had a further ponder over finding a large dev set but something occurred to me. Wouldn't this encode various words that exist in the dev set and testing set but not in the training set? I'm wondering what the point would be of keeping those words in the test set if the classifier isn't even trained on them. At the moment, when I use BoW I drop the words not in the training set from the test set. What's the intuition behind keeping them using a dev set? Would they just be ignored by the classifier anyway? Or rather, how do trained classifiers deal with those kinds of words? Thanks!  
  > **username_1**: You don't really have to go the extra mile for finding the largest dev set. Using your raw training set (tokenized) would be enough. Your observation is currect, when train corpus doens't contain a word, there will be no signal learned from that single word. In practice this isn't a problem, because you look at more than one word for making a decision (n-grams, conv nets, RNNs, transformers, etc). If a word doesn't appear in the training set, most likely it's not significant enough (rare) to have an effect. In practice a vocab of 30k-50k words is enough to make a decent classifier.  
  > **username_1**: You freeze the vocab mostly for convenience. You shouldn't be doing tokenization and vocab building by hand. Most of the tools out there will take care of picking the most common N words, will tokenize the text properly, and map any unseen (unknown) word to a special `<unk>` token. That will be the placeholder to represent all rare/out-of-vocabulary (oov) tokens and your model will also learn to play nicely with it (usually treat it as noise).  
  
  ---
  
  