Answers.Body: <p>Your problem is that TFIDF is cutting out around 90 terms.</p>
  
  <p>Since you aren't using the <code>min_df</code> or <code>max_df</code> parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.</p>
  
  <p>There are two things I'd try out if I were you:</p>
  
  <ol>
  <li>Try explicitly converting the word2vec corpus to lowercase. <code>TfidfVectorizer</code> does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.</li>
  <li>Try using the <code>nltk.WordPunctTokenizer()</code> with the <code>TfidfVectorizer</code>. You can do this like this:</li>
  </ol>
  
  
  
  <pre><code>wpt = nltk.WordPunctTokenizer()
  tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                       tokenizer=wpt.tokenize)
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T09:59:57.023Z
Answers.Text: Thank you very much. I need to do the second thing for fix it.
Answers.UserDisplayName: B612
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T09:22:04.123Z
Answers.LastActivityDate: 2020-03-03T09:22:04.123Z
Answers.UserDisplayName: Djib2011
Body: <p>I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. <strong>I am using the same corpus for get both matrix, I don't know why each matrix have a different number of terms
  .</strong></p>
  
  <p>My problem is that I have a matrix TFIDF with the shape <code>(56096, 15500)</code> (corresponding to: number of terms, number of documents) and matrix Word2vec with the shape <code>(300, 56184)</code> (corresponding to : number of word-embeddings, number of terms).<br>
  And I need the same numbers of terms in both matrix. </p>
  
  <p>I use this code for get the matrix of word-embeddings Word2vec: </p>
  
  <pre><code>def w2vec_gensim(norm_corpus):
      wpt = nltk.WordPunctTokenizer()
      tokenized_corpus = [wpt.tokenize(document) for document in norm_corpus]
      # Set values for various parameters
      feature_size = 300
      # Word vector dimensionality
      window_context = 10
      # Context window size
      min_word_count = 1
      # Minimum word count
      sample = 1e-3
      # Downsample setting for frequent words
      w2v_model = word2vec.Word2Vec(tokenized_corpus, size=feature_size, window=window_context, min_count =  min_word_count, sample=sample, iter=100)
      words = list(w2v_model.wv.vocab)
      vectors=[]
      for w in words:
          vectors.append(w2v_model[w].tolist())
      embedding_matrix= np.array(vectors)
      embedding_matrix= embedding_matrix.T
      print(embedding_matrix.shape)
  
      return embedding_matrix
  </code></pre>
  
  <p>And this code for get the TFIDF matrix: </p>
  
  <pre><code>tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True)
  
  
  def matriz_tf_idf(datos, tv):
      tv_matrix = tv.fit_transform(datos)
      tv_matrix = tv_matrix.toarray()
      tv_matrix = tv_matrix.T
      return tv_matrix
  </code></pre>
  
  <p>And I need the same number of terms in each matrix. For example, if I have 56096 terms in TFIDF, I need the same number in embeddings matrix, I mean matrix TFIDF with the shape <code>(56096, 1550)</code> and matrix of embeddings Word2vec with the shape <code>(300, 56096)</code>. How I can get the same number of terms in both matrix? 
  Because I can't delete without more data, due to I need the multiplication to make sense because my goal is to get the embeddings from the documents. </p>
  
  <p>Thank you very much in advance.</p>
  
Tags: <natural-language-processing><python><word-embedding><word2vec><weights>
Title: Why I have a different number of terms in word2vec and TFIDF? How I can fix it?
UserDisplayName: B612
ThreadText: # Why I have a different number of terms in word2vec and TFIDF? How I can fix it?
  
  (Asked by: username_0 on 2020-03-02)
  
  I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for get both matrix, I don't know why each matrix have a different number of terms
  .
  My problem is that I have a matrix TFIDF with the shape (56096, 15500) (corresponding to: number of terms, number of documents) and matrix Word2vec with the shape (300, 56184) (corresponding to : number of word-embeddings, number of terms).
  And I need the same numbers of terms in both matrix. 
  I use this code for get the matrix of word-embeddings Word2vec: 
  def w2vec_gensim(norm_corpus):
      wpt = nltk.WordPunctTokenizer()
      tokenized_corpus = [wpt.tokenize(document) for document in norm_corpus]
      # Set values for various parameters
      feature_size = 300
      # Word vector dimensionality
      window_context = 10
      # Context window size
      min_word_count = 1
      # Minimum word count
      sample = 1e-3
      # Downsample setting for frequent words
      w2v_model = word2vec.Word2Vec(tokenized_corpus, size=feature_size, window=window_context, min_count =  min_word_count, sample=sample, iter=100)
      words = list(w2v_model.wv.vocab)
      vectors=[]
      for w in words:
          vectors.append(w2v_model[w].tolist())
      embedding_matrix= np.array(vectors)
      embedding_matrix= embedding_matrix.T
      print(embedding_matrix.shape)
  
      return embedding_matrix
  
  And this code for get the TFIDF matrix: 
  tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True)
  
  
  def matriz_tf_idf(datos, tv):
      tv_matrix = tv.fit_transform(datos)
      tv_matrix = tv_matrix.toarray()
      tv_matrix = tv_matrix.T
      return tv_matrix
  
  And I need the same number of terms in each matrix. For example, if I have 56096 terms in TFIDF, I need the same number in embeddings matrix, I mean matrix TFIDF with the shape (56096, 1550) and matrix of embeddings Word2vec with the shape (300, 56096). How I can get the same number of terms in both matrix? 
  Because I can't delete without more data, due to I need the multiplication to make sense because my goal is to get the embeddings from the documents. 
  Thank you very much in advance.
  
  **Tags:** natural-language-processing, python, word-embedding, word2vec, weights
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-03. Score: 0 | Accepted answer
  
  Your problem is that TFIDF is cutting out around 90 terms.
  Since you aren't using the min_df or max_df parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.
  There are two things I'd try out if I were you:
  
  Try explicitly converting the word2vec corpus to lowercase. TfidfVectorizer does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.
  Try using the nltk.WordPunctTokenizer() with the TfidfVectorizer. You can do this like this:
  
  wpt = nltk.WordPunctTokenizer()
  tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                       tokenizer=wpt.tokenize)
  
  > **Comments:**
  > **username_0**: Thank you very much. I need to do the second thing for fix it.  
  
  ---
  
  