<p dir="auto">I was trying this code:</p>
<blockquote>
<blockquote>
<blockquote>
<p dir="auto">corpus = ['This is the first document.',<br>
... 'This is the second second document.',<br>
... 'And the third one.',<br>
... 'Is this the first document?']<br>
from sklearn.feature_extraction.text import TfidfVectorizer<br>
vectorizer = TfidfVectorizer(min_df=1)<br>
vectorizer.fit_transform(corpus).todense()</p>
</blockquote>
</blockquote>
</blockquote>
<p dir="auto">Gives<br>
ValueError: Buffer dtype mismatch, expected 'int' but got 'long'</p>
<p dir="auto">scikit-learn version = '0.15-git'<br>
Scipy version = '0.14.0.dev-7cefb25'<br>
Platform = Mac OSX 10.9.2</p>
<p dir="auto">Hello the great scikit-learn team!</p>
<p dir="auto">Consider the following simple example (also posted on <a href="https://stackoverflow.com/questions/68282958/how-to-merge-two-countvectorizers-when-there-are-duplicates/68283373?noredirect=1#comment120682302_68283373" rel="nofollow">SO</a>)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" data = pd.DataFrame({'text1' : ['hello world', 'hello universe'],
                         'text2': ['good morning', 'hello two three']})
        
    data
    Out[489]: 
                text1            text2
    0     hello world     good morning
    1  hello universe  hello two three"><pre class="notranslate"><code class="notranslate"> data = pd.DataFrame({'text1' : ['hello world', 'hello universe'],
                         'text2': ['good morning', 'hello two three']})
        
    data
    Out[489]: 
                text1            text2
    0     hello world     good morning
    1  hello universe  hello two three
</code></pre></div>
<p dir="auto">As you can see, <code class="notranslate">text1</code> and <code class="notranslate">text2</code> share one exact word in common: <code class="notranslate">hello</code>. I am trying to create ngrams separately for <code class="notranslate">text1</code> and <code class="notranslate">text2</code> and I want to concatenate the results together into a countvectorizer object.</p>
<p dir="auto">The idea is that I want to create ngrams separately for the two variables and used them as features in a ML algo. However, I do want the extra ngrams that would be created by concatenating the string together, like <code class="notranslate">world good</code> in <code class="notranslate">hello world good morning</code>. This is why I keep the ngram creation separated.</p>
<p dir="auto">The issue is that by doing so, the resulting (sparse) vector will contain a duplicated <code class="notranslate">hello</code> column.</p>
<p dir="auto">See here:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    vector = CountVectorizer(ngram_range=(1, 2))
    
    v1 = vector.fit_transform(data.text1.values) 
    print(vector.get_feature_names())

    ['hello', 'hello universe', 'hello world', 'universe', 'world']

    v2 = vector.fit_transform(data.text2.values)
    print(vector.get_feature_names())
    
    ['good', 'good morning', 'hello', 'hello two', 'morning', 'three', 'two', 'two three']


And now concatenating `v1` and `v2` gives 13 columns


    from scipy.sparse import hstack
    print(hstack((v1, v2)).toarray())
    
    [[1 0 1 0 1 1 1 0 0 1 0 0 0]
     [1 1 0 1 0 0 0 1 1 0 1 1 1]]"><pre class="notranslate"><code class="notranslate">    vector = CountVectorizer(ngram_range=(1, 2))
    
    v1 = vector.fit_transform(data.text1.values) 
    print(vector.get_feature_names())

    ['hello', 'hello universe', 'hello world', 'universe', 'world']

    v2 = vector.fit_transform(data.text2.values)
    print(vector.get_feature_names())
    
    ['good', 'good morning', 'hello', 'hello two', 'morning', 'three', 'two', 'two three']


And now concatenating `v1` and `v2` gives 13 columns


    from scipy.sparse import hstack
    print(hstack((v1, v2)).toarray())
    
    [[1 0 1 0 1 1 1 0 0 1 0 0 0]
     [1 1 0 1 0 0 0 1 1 0 1 1 1]]
</code></pre></div>
<p dir="auto">The proper text-features should be 12:</p>
<p dir="auto"><code class="notranslate">hello</code>, <code class="notranslate">word</code>, <code class="notranslate">hello word</code>, <code class="notranslate">good</code>, <code class="notranslate">morning</code>, <code class="notranslate">good morning</code>,<code class="notranslate">hello universe</code>,<code class="notranslate">universe</code>,  <code class="notranslate">two</code>, <code class="notranslate">three</code>, <code class="notranslate">hello two</code>, <code class="notranslate">two three</code></p>
<p dir="auto">I thought I had found the solution by using <code class="notranslate">ColumnTransformer</code> but it seems I am getting the same extra column</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="union= ColumnTransformer([(&quot;norm1&quot;, CountVectorizer(ngram_range=(1, 2)), &quot;text1&quot;), (&quot;norm2&quot;, CountVectorizer(ngram_range=(1, 2)), &quot;text2&quot;)])

union.fit_transform(data)
Out[508]: 
array([[1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0],
       [1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1]], dtype=int64)"><pre class="notranslate"><code class="notranslate">union= ColumnTransformer([("norm1", CountVectorizer(ngram_range=(1, 2)), "text1"), ("norm2", CountVectorizer(ngram_range=(1, 2)), "text2")])

union.fit_transform(data)
Out[508]: 
array([[1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0],
       [1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1]], dtype=int64)
</code></pre></div>
<p dir="auto">Is this a bug in ColumnTransformer? Isnt ColumnTransformer supposed to get rid of duplicates?</p>
<p dir="auto">Thanks!</p>