<p>Your problem is that TFIDF is cutting out around 90 terms.</p>

<p>Since you aren't using the <code>min_df</code> or <code>max_df</code> parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.</p>

<p>There are two things I'd try out if I were you:</p>

<ol>
<li>Try explicitly converting the word2vec corpus to lowercase. <code>TfidfVectorizer</code> does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.</li>
<li>Try using the <code>nltk.WordPunctTokenizer()</code> with the <code>TfidfVectorizer</code>. You can do this like this:</li>
</ol>



<pre><code>wpt = nltk.WordPunctTokenizer()
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                     tokenizer=wpt.tokenize)

</code></pre>

CC BY-SA 4.0
2020-03-03T09:59:57.023Z
Thank you very much. I need to do the second thing for fix it.
B612
CC BY-SA 4.0
2020-03-03T09:22:04.123Z
2020-03-03T09:22:04.123Z
Djib2011
<p>I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. <strong>I am using the same corpus for get both matrix, I don't know why each matrix have a different number of terms
.</strong></p>

<p>My problem is that I have a matrix TFIDF with the shape <code>(56096, 15500)</code> (corresponding to: number of terms, number of documents) and matrix Word2vec with the shape <code>(300, 56184)</code> (corresponding to : number of word-embeddings, number of terms).<br>
And I need the same numbers of terms in both matrix. </p>

<p>I use this code for get the matrix of word-embeddings Word2vec: </p>

<pre><code>def w2vec_gensim(norm_corpus):
    wpt = nltk.WordPunctTokenizer()
    tokenized_corpus = [wpt.tokenize(document) for document in norm_corpus]
    # Set values for various parameters
    feature_size = 300
    # Word vector dimensionality
    window_context = 10
    # Context window size
    min_word_count = 1
    # Minimum word count
    sample = 1e-3
    # Downsample setting for frequent words
    w2v_model = word2vec.Word2Vec(tokenized_corpus, size=feature_size, window=window_context, min_count =  min_word_count, sample=sample, iter=100)
    words = list(w2v_model.wv.vocab)
    vectors=[]
    for w in words:
        vectors.append(w2v_model[w].tolist())
    embedding_matrix= np.array(vectors)
    embedding_matrix= embedding_matrix.T
    print(embedding_matrix.shape)

    return embedding_matrix
</code></pre>

<p>And this code for get the TFIDF matrix: </p>

<pre><code>tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True)


def matriz_tf_idf(datos, tv):
    tv_matrix = tv.fit_transform(datos)
    tv_matrix = tv_matrix.toarray()
    tv_matrix = tv_matrix.T
    return tv_matrix
</code></pre>

<p>And I need the same number of terms in each matrix. For example, if I have 56096 terms in TFIDF, I need the same number in embeddings matrix, I mean matrix TFIDF with the shape <code>(56096, 1550)</code> and matrix of embeddings Word2vec with the shape <code>(300, 56096)</code>. How I can get the same number of terms in both matrix? 
Because I can't delete without more data, due to I need the multiplication to make sense because my goal is to get the embeddings from the documents. </p>

<p>Thank you very much in advance.</p>

<natural-language-processing><python><word-embedding><word2vec><weights>
Why I have a different number of terms in word2vec and TFIDF? How I can fix it?
B612
# Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

(Asked by: username_0 on 2020-03-02)

I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for get both matrix, I don't know why each matrix have a different number of terms
.
My problem is that I have a matrix TFIDF with the shape (56096, 15500) (corresponding to: number of terms, number of documents) and matrix Word2vec with the shape (300, 56184) (corresponding to : number of word-embeddings, number of terms).
And I need the same numbers of terms in both matrix. 
I use this code for get the matrix of word-embeddings Word2vec: 
def w2vec_gensim(norm_corpus):
    wpt = nltk.WordPunctTokenizer()
    tokenized_corpus = [wpt.tokenize(document) for document in norm_corpus]
    # Set values for various parameters
    feature_size = 300
    # Word vector dimensionality
    window_context = 10
    # Context window size
    min_word_count = 1
    # Minimum word count
    sample = 1e-3
    # Downsample setting for frequent words
    w2v_model = word2vec.Word2Vec(tokenized_corpus, size=feature_size, window=window_context, min_count =  min_word_count, sample=sample, iter=100)
    words = list(w2v_model.wv.vocab)
    vectors=[]
    for w in words:
        vectors.append(w2v_model[w].tolist())
    embedding_matrix= np.array(vectors)
    embedding_matrix= embedding_matrix.T
    print(embedding_matrix.shape)

    return embedding_matrix

And this code for get the TFIDF matrix: 
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True)


def matriz_tf_idf(datos, tv):
    tv_matrix = tv.fit_transform(datos)
    tv_matrix = tv_matrix.toarray()
    tv_matrix = tv_matrix.T
    return tv_matrix

And I need the same number of terms in each matrix. For example, if I have 56096 terms in TFIDF, I need the same number in embeddings matrix, I mean matrix TFIDF with the shape (56096, 1550) and matrix of embeddings Word2vec with the shape (300, 56096). How I can get the same number of terms in both matrix? 
Because I can't delete without more data, due to I need the multiplication to make sense because my goal is to get the embeddings from the documents. 
Thank you very much in advance.

**Tags:** natural-language-processing, python, word-embedding, word2vec, weights


---

## 1 answers

### Answer by username_1 on 2020-03-03. Score: 0 | Accepted answer

Your problem is that TFIDF is cutting out around 90 terms.
Since you aren't using the min_df or max_df parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.
There are two things I'd try out if I were you:

Try explicitly converting the word2vec corpus to lowercase. TfidfVectorizer does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.
Try using the nltk.WordPunctTokenizer() with the TfidfVectorizer. You can do this like this:

wpt = nltk.WordPunctTokenizer()
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                     tokenizer=wpt.tokenize)

> **Comments:**
> **username_0**: Thank you very much. I need to do the second thing for fix it.  

---

