<p dir="auto">If you initialize a <code class="notranslate">CounterVectorizer</code> and try to perform a transformation without training you will get a <code class="notranslate">NotFittedError</code> exception.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: from sklearn.feature_extraction.text import CountVectorizer
In [2]: vectorizer = CountVectorizer()
In [3]: corpus = [
    ...:     'This is the first document.',
    ...:     'This is the second second document.',
    ...:     'And the third one.',
    ...:     'Is this the first document?',
    ...: ]

In [4]: vectorizer.transform(corpus)
NotFittedError: CountVectorizer - Vocabulary wasn't fitted."><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">feature_extraction</span>.<span class="pl-s1">text</span> <span class="pl-s1">import</span> <span class="pl-v">CountVectorizer</span>
<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">vectorizer</span> <span class="pl-c1">=</span> <span class="pl-v">CountVectorizer</span>()
<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">corpus</span> <span class="pl-c1">=</span> [
    ...:     <span class="pl-s">'This is the first document.'</span>,
    ...:     <span class="pl-s">'This is the second second document.'</span>,
    ...:     <span class="pl-s">'And the third one.'</span>,
    ...:     <span class="pl-s">'Is this the first document?'</span>,
    ...: ]

<span class="pl-v">In</span> [<span class="pl-c1">4</span>]: <span class="pl-s1">vectorizer</span>.<span class="pl-en">transform</span>(<span class="pl-s1">corpus</span>)
<span class="pl-v">NotFittedError</span>: <span class="pl-v">CountVectorizer</span> <span class="pl-c1">-</span> <span class="pl-v">Vocabulary</span> <span class="pl-s1">wasn</span>'<span class="pl-s1">t</span> <span class="pl-s1">fitted</span>.</pre></div>
<p dir="auto">On the other hand if you provide the <code class="notranslate">vocabulary</code> at the initialization of the vectorizer you could transform a corpus without a prior training, right?</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: from sklearn.feature_extraction.text import CountVectorizer

In [2]: vectorizer = CountVectorizer()

In [3]: corpus = [
    ...:     'This is the first document.',
    ...:     'This is the second second document.',
    ...:     'And the third one.',
    ...:     'Is this the first document?',
    ...: ]

In [4]: vocabulary = ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']

In [5]: vectorizer = CountVectorizer(vocabulary=vocabulary)

In [6]: hasattr(vectorizer, &quot;vocabulary_&quot;)
Out[6]: False

In [7]: vectorizer.get_feature_names()
NotFittedError: CountVectorizer - Vocabulary wasn't fitted.

In [8]: vectorizer.transform(corpus)
Out[8]:
&lt;4x9 sparse matrix of type '&lt;class 'numpy.int64'&gt;'
        with 19 stored elements in Compressed Sparse Row format&gt;

In [9]: hasattr(vectorizer, &quot;vocabulary_&quot;)
Out[9]: True"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">feature_extraction</span>.<span class="pl-s1">text</span> <span class="pl-s1">import</span> <span class="pl-v">CountVectorizer</span>

<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">vectorizer</span> <span class="pl-c1">=</span> <span class="pl-v">CountVectorizer</span>()

<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">corpus</span> <span class="pl-c1">=</span> [
    ...:     <span class="pl-s">'This is the first document.'</span>,
    ...:     <span class="pl-s">'This is the second second document.'</span>,
    ...:     <span class="pl-s">'And the third one.'</span>,
    ...:     <span class="pl-s">'Is this the first document?'</span>,
    ...: ]

<span class="pl-v">In</span> [<span class="pl-c1">4</span>]: <span class="pl-s1">vocabulary</span> <span class="pl-c1">=</span> [<span class="pl-s">'and'</span>, <span class="pl-s">'document'</span>, <span class="pl-s">'first'</span>, <span class="pl-s">'is'</span>, <span class="pl-s">'one'</span>, <span class="pl-s">'second'</span>, <span class="pl-s">'the'</span>, <span class="pl-s">'third'</span>, <span class="pl-s">'this'</span>]

<span class="pl-v">In</span> [<span class="pl-c1">5</span>]: <span class="pl-s1">vectorizer</span> <span class="pl-c1">=</span> <span class="pl-v">CountVectorizer</span>(<span class="pl-s1">vocabulary</span><span class="pl-c1">=</span><span class="pl-s1">vocabulary</span>)

<span class="pl-v">In</span> [<span class="pl-c1">6</span>]: <span class="pl-en">hasattr</span>(<span class="pl-s1">vectorizer</span>, <span class="pl-s">"vocabulary_"</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">6</span>]: <span class="pl-c1">False</span>

<span class="pl-v">In</span> [<span class="pl-c1">7</span>]: <span class="pl-s1">vectorizer</span>.<span class="pl-en">get_feature_names</span>()
<span class="pl-v">NotFittedError</span>: <span class="pl-v">CountVectorizer</span> <span class="pl-c1">-</span> <span class="pl-v">Vocabulary</span> <span class="pl-s1">wasn</span>'<span class="pl-s1">t</span> <span class="pl-s1">fitted</span>.

<span class="pl-v">In</span> [<span class="pl-c1">8</span>]: <span class="pl-s1">vectorizer</span>.<span class="pl-en">transform</span>(<span class="pl-s1">corpus</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">8</span>]:
<span class="pl-c1">&lt;</span><span class="pl-c1">4</span><span class="pl-s1">x9</span> <span class="pl-s1">sparse</span> <span class="pl-s1">matrix</span> <span class="pl-s1">of</span> <span class="pl-s1">type</span> <span class="pl-s">'&lt;class '</span><span class="pl-s1">numpy</span>.<span class="pl-s1">int64</span><span class="pl-s">'&gt;'</span>
        <span class="pl-k">with</span> <span class="pl-c1">19</span> <span class="pl-s1">stored</span> <span class="pl-s1">elements</span> <span class="pl-c1">in</span> <span class="pl-v">Compressed</span> <span class="pl-v">Sparse</span> <span class="pl-v">Row</span> <span class="pl-s1">format</span><span class="pl-c1">&gt;</span>

<span class="pl-v">In</span> [<span class="pl-c1">9</span>]: <span class="pl-en">hasattr</span>(<span class="pl-s1">vectorizer</span>, <span class="pl-s">"vocabulary_"</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">9</span>]: <span class="pl-c1">True</span></pre></div>
<p dir="auto">The <code class="notranslate">CountVectorizer</code>'s <code class="notranslate">transform</code> calls <code class="notranslate">_validate_vocabulary</code> method which sets the <code class="notranslate">vocabulary_</code> instance variable.</p>
<p dir="auto">In the same manner I believe that the <code class="notranslate">get_feature_names</code> method should not raise <code class="notranslate">NotFittedError</code> if the vocabulary parameter is provided but the vectorizer has not been trained.</p>
<p dir="auto">Before some recent changes, CountVectorizer raised a cryptic error when a given vocabulary that was not unique. Now, the code that led to this error is gone, and a non-unique vocabulary can slip in. Sometimes this is fine (I assume. My code works at least.). Sometimes I get a segfault. I've narrowed it down to a reproducible example (though not quite deterministic in the length of title below), but not for which I can generate data. I can send data offlist if someone is interested in replicating. This segfaults</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="vectorizer = CountVectorizer(vocabulary=vocab)
counts = vectorizer.fit_transform(titles)"><pre class="notranslate"><code class="notranslate">vectorizer = CountVectorizer(vocabulary=vocab)
counts = vectorizer.fit_transform(titles)
</code></pre></div>
<p dir="auto">This works</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="vectorizer = CountVectorizer(vocabulary=list(set(vocab)))
counts = vectorizer.fit_transform(titles)"><pre class="notranslate"><code class="notranslate">vectorizer = CountVectorizer(vocabulary=list(set(vocab)))
counts = vectorizer.fit_transform(titles)
</code></pre></div>
<p dir="auto">This also works, and it's not clear what exactly is different in the (n + 1)th title that leads to the segfault.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="vectorizer = CountVectorizer(vocabulary=vocab)
counts = vectorizer.fit_transform(titles[:150])"><pre class="notranslate"><code class="notranslate">vectorizer = CountVectorizer(vocabulary=vocab)
counts = vectorizer.fit_transform(titles[:150])
</code></pre></div>
<p dir="auto">The backtrace is (though oddly it seems to depend on the length of titles what I get)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Program received signal SIGSEGV, Segmentation fault.
0x00007fffe849cdd9 in csr_tocsc&lt;int, long long&gt; (n_row=n_row@entry=15493, 
    n_col=n_col@entry=1856, Ap=Ap@entry=0x2db5ca0, Aj=Aj@entry=0x2d5c610, 
    Ax=Ax@entry=0x2f569d0, Bp=Bp@entry=0x2e71030, Bi=Bi@entry=0x2ea3eb0, 
    Bx=0x2eceb50) at scipy/sparse/sparsetools/csr.h:411
411                 Bi[dest] = row;
(gdb) backtrace
#0  0x00007fffe849cdd9 in csr_tocsc&lt;int, long long&gt; (n_row=n_row@entry=15493, 
    n_col=n_col@entry=1856, Ap=Ap@entry=0x2db5ca0, Aj=Aj@entry=0x2d5c610, 
    Ax=Ax@entry=0x2f569d0, Bp=Bp@entry=0x2e71030, Bi=Bi@entry=0x2ea3eb0, 
    Bx=0x2eceb50) at scipy/sparse/sparsetools/csr.h:411
#1  0x00007fffe83edcef in _wrap_csr_tocsc__SWIG_8 (args=args@entry=0x2d01aa0)
    at scipy/sparse/sparsetools/csr_wrap.cxx:11553
#2  0x00007fffe844311b in _wrap_csr_tocsc (self=&lt;optimised out&gt;, 
    args=0x2d01aa0) at scipy/sparse/sparsetools/csr_wrap.cxx:12841
#3  0x00000000005313b5 in ext_do_call (nk=&lt;optimised out&gt;, na=&lt;optimised out&gt;, 
    flags=&lt;optimised out&gt;, pp_stack=0x7fffffffd280, func=0x2b9b320)
    at ../Python/ceval.c:4331
#4  PyEval_EvalFrameEx (f=f@entry=0x284a440, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2705
#5  0x0000000000505b24 in PyEval_EvalCodeEx (co=0x2c823b0, 
    globals=&lt;optimised out&gt;, locals=locals@entry=0x0, args=&lt;optimised out&gt;, 
    argcount=argcount@entry=8, kws=&lt;optimised out&gt;, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0)
    at ../Python/ceval.c:3253
#6  0x000000000052f3f2 in fast_function (nk=0, na=8, n=&lt;optimised out&gt;, 
    pp_stack=0x7fffffffd4a0, func=0x2c86140) at ../Python/ceval.c:4117
#7  call_function (oparg=&lt;optimised out&gt;, pp_stack=0x7fffffffd4a0)
    at ../Python/ceval.c:4042
#8  PyEval_EvalFrameEx (f=f@entry=0x28512b0, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#9  0x0000000000505b24 in PyEval_EvalCodeEx (co=0x2b88f30, 
    globals=&lt;optimised out&gt;, locals=locals@entry=0x0, args=&lt;optimised out&gt;, 
    argcount=argcount@entry=1, kws=&lt;optimised out&gt;, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0)
    at ../Python/ceval.c:3253
#10 0x000000000052f3f2 in fast_function (nk=0, na=1, n=&lt;optimised out&gt;, 
    pp_stack=0x7fffffffd6c0, func=0x2caa8c0) at ../Python/ceval.c:4117
#11 call_function (oparg=&lt;optimised out&gt;, pp_stack=0x7fffffffd6c0)
    at ../Python/ceval.c:4042
#12 PyEval_EvalFrameEx (f=f@entry=0x20d3af0, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#13 0x0000000000505b24 in PyEval_EvalCodeEx (co=0x2d07230, 
    globals=&lt;optimised out&gt;, locals=locals@entry=0x0, args=&lt;optimised out&gt;, 
    argcount=argcount@entry=2, kws=&lt;optimised out&gt;, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x2d06e68, defcount=1, closure=0x0)
    at ../Python/ceval.c:3253
#14 0x000000000052e237 in fast_function (nk=0, na=2, n=&lt;optimised out&gt;, 
    pp_stack=0x7fffffffd8e0, func=0x2d17c80) at ../Python/ceval.c:4117
#15 call_function (oparg=&lt;optimised out&gt;, pp_stack=0x7fffffffd8e0)
    at ../Python/ceval.c:4042
#16 PyEval_EvalFrameEx (f=f@entry=0xa56bc0, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#17 0x0000000000567cdc in PyEval_EvalCodeEx (closure=0x0, defcount=0, 
    defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=0x9410a0, 
    globals=0x7ffff7f18c30, co=&lt;optimised out&gt;) at ../Python/ceval.c:3253
#18 PyEval_EvalCode (co=co@entry=0x7ffff7f18c30, 
    globals=globals@entry=0x97e690, locals=locals@entry=0x97e690)
    at ../Python/ceval.c:667
#19 0x0000000000451adb in run_mod.42568 (mod=&lt;optimised out&gt;, 
    filename=&lt;optimised out&gt;, globals=0x97e690, locals=0x97e690, 
---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---
    flags=&lt;optimised out&gt;, arena=&lt;optimised out&gt;) at ../Python/pythonrun.c:1365
#20 0x0000000000451e5b in PyRun_FileExFlags (fp=fp@entry=0xa56ba0, 
    filename=filename@entry=0x7fffffffe058 &quot;sklearn_seg.py&quot;, 
    start=start@entry=257, globals=globals@entry=0x97e690, 
    locals=locals@entry=0x97e690, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fffffffdab0) at ../Python/pythonrun.c:1351
#21 0x0000000000452394 in PyRun_SimpleFileExFlags (fp=fp@entry=0xa56ba0, 
    filename=&lt;optimised out&gt;, filename@entry=0x7fffffffe058 &quot;sklearn_seg.py&quot;, 
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdab0)
    at ../Python/pythonrun.c:943
#22 0x0000000000452490 in PyRun_AnyFileExFlags (fp=fp@entry=0xa56ba0, 
    filename=filename@entry=0x7fffffffe058 &quot;sklearn_seg.py&quot;, 
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdab0)
    at ../Python/pythonrun.c:747
#23 0x0000000000453ead in Py_Main (argc=&lt;optimised out&gt;, argv=0x7fffffffdc68)
    at ../Modules/main.c:640
#24 0x00007ffff7816de5 in __libc_start_main (main=0x453f6b &lt;main&gt;, argc=2, 
    ubp_av=0x7fffffffdc68, init=&lt;optimised out&gt;, fini=&lt;optimised out&gt;, 
    rtld_fini=&lt;optimised out&gt;, stack_end=0x7fffffffdc58) at libc-start.c:260
#25 0x00000000005786be in _start ()"><pre class="notranslate"><code class="notranslate">Program received signal SIGSEGV, Segmentation fault.
0x00007fffe849cdd9 in csr_tocsc&lt;int, long long&gt; (n_row=n_row@entry=15493, 
    n_col=n_col@entry=1856, Ap=Ap@entry=0x2db5ca0, Aj=Aj@entry=0x2d5c610, 
    Ax=Ax@entry=0x2f569d0, Bp=Bp@entry=0x2e71030, Bi=Bi@entry=0x2ea3eb0, 
    Bx=0x2eceb50) at scipy/sparse/sparsetools/csr.h:411
411                 Bi[dest] = row;
(gdb) backtrace
#0  0x00007fffe849cdd9 in csr_tocsc&lt;int, long long&gt; (n_row=n_row@entry=15493, 
    n_col=n_col@entry=1856, Ap=Ap@entry=0x2db5ca0, Aj=Aj@entry=0x2d5c610, 
    Ax=Ax@entry=0x2f569d0, Bp=Bp@entry=0x2e71030, Bi=Bi@entry=0x2ea3eb0, 
    Bx=0x2eceb50) at scipy/sparse/sparsetools/csr.h:411
#1  0x00007fffe83edcef in _wrap_csr_tocsc__SWIG_8 (args=args@entry=0x2d01aa0)
    at scipy/sparse/sparsetools/csr_wrap.cxx:11553
#2  0x00007fffe844311b in _wrap_csr_tocsc (self=&lt;optimised out&gt;, 
    args=0x2d01aa0) at scipy/sparse/sparsetools/csr_wrap.cxx:12841
#3  0x00000000005313b5 in ext_do_call (nk=&lt;optimised out&gt;, na=&lt;optimised out&gt;, 
    flags=&lt;optimised out&gt;, pp_stack=0x7fffffffd280, func=0x2b9b320)
    at ../Python/ceval.c:4331
#4  PyEval_EvalFrameEx (f=f@entry=0x284a440, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2705
#5  0x0000000000505b24 in PyEval_EvalCodeEx (co=0x2c823b0, 
    globals=&lt;optimised out&gt;, locals=locals@entry=0x0, args=&lt;optimised out&gt;, 
    argcount=argcount@entry=8, kws=&lt;optimised out&gt;, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0)
    at ../Python/ceval.c:3253
#6  0x000000000052f3f2 in fast_function (nk=0, na=8, n=&lt;optimised out&gt;, 
    pp_stack=0x7fffffffd4a0, func=0x2c86140) at ../Python/ceval.c:4117
#7  call_function (oparg=&lt;optimised out&gt;, pp_stack=0x7fffffffd4a0)
    at ../Python/ceval.c:4042
#8  PyEval_EvalFrameEx (f=f@entry=0x28512b0, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#9  0x0000000000505b24 in PyEval_EvalCodeEx (co=0x2b88f30, 
    globals=&lt;optimised out&gt;, locals=locals@entry=0x0, args=&lt;optimised out&gt;, 
    argcount=argcount@entry=1, kws=&lt;optimised out&gt;, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0)
    at ../Python/ceval.c:3253
#10 0x000000000052f3f2 in fast_function (nk=0, na=1, n=&lt;optimised out&gt;, 
    pp_stack=0x7fffffffd6c0, func=0x2caa8c0) at ../Python/ceval.c:4117
#11 call_function (oparg=&lt;optimised out&gt;, pp_stack=0x7fffffffd6c0)
    at ../Python/ceval.c:4042
#12 PyEval_EvalFrameEx (f=f@entry=0x20d3af0, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#13 0x0000000000505b24 in PyEval_EvalCodeEx (co=0x2d07230, 
    globals=&lt;optimised out&gt;, locals=locals@entry=0x0, args=&lt;optimised out&gt;, 
    argcount=argcount@entry=2, kws=&lt;optimised out&gt;, kwcount=kwcount@entry=0, 
    defs=defs@entry=0x2d06e68, defcount=1, closure=0x0)
    at ../Python/ceval.c:3253
#14 0x000000000052e237 in fast_function (nk=0, na=2, n=&lt;optimised out&gt;, 
    pp_stack=0x7fffffffd8e0, func=0x2d17c80) at ../Python/ceval.c:4117
#15 call_function (oparg=&lt;optimised out&gt;, pp_stack=0x7fffffffd8e0)
    at ../Python/ceval.c:4042
#16 PyEval_EvalFrameEx (f=f@entry=0xa56bc0, throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#17 0x0000000000567cdc in PyEval_EvalCodeEx (closure=0x0, defcount=0, 
    defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=0x9410a0, 
    globals=0x7ffff7f18c30, co=&lt;optimised out&gt;) at ../Python/ceval.c:3253
#18 PyEval_EvalCode (co=co@entry=0x7ffff7f18c30, 
    globals=globals@entry=0x97e690, locals=locals@entry=0x97e690)
    at ../Python/ceval.c:667
#19 0x0000000000451adb in run_mod.42568 (mod=&lt;optimised out&gt;, 
    filename=&lt;optimised out&gt;, globals=0x97e690, locals=0x97e690, 
---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---
    flags=&lt;optimised out&gt;, arena=&lt;optimised out&gt;) at ../Python/pythonrun.c:1365
#20 0x0000000000451e5b in PyRun_FileExFlags (fp=fp@entry=0xa56ba0, 
    filename=filename@entry=0x7fffffffe058 "sklearn_seg.py", 
    start=start@entry=257, globals=globals@entry=0x97e690, 
    locals=locals@entry=0x97e690, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fffffffdab0) at ../Python/pythonrun.c:1351
#21 0x0000000000452394 in PyRun_SimpleFileExFlags (fp=fp@entry=0xa56ba0, 
    filename=&lt;optimised out&gt;, filename@entry=0x7fffffffe058 "sklearn_seg.py", 
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdab0)
    at ../Python/pythonrun.c:943
#22 0x0000000000452490 in PyRun_AnyFileExFlags (fp=fp@entry=0xa56ba0, 
    filename=filename@entry=0x7fffffffe058 "sklearn_seg.py", 
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdab0)
    at ../Python/pythonrun.c:747
#23 0x0000000000453ead in Py_Main (argc=&lt;optimised out&gt;, argv=0x7fffffffdc68)
    at ../Modules/main.c:640
#24 0x00007ffff7816de5 in __libc_start_main (main=0x453f6b &lt;main&gt;, argc=2, 
    ubp_av=0x7fffffffdc68, init=&lt;optimised out&gt;, fini=&lt;optimised out&gt;, 
    rtld_fini=&lt;optimised out&gt;, stack_end=0x7fffffffdc58) at libc-start.c:260
#25 0x00000000005786be in _start ()
</code></pre></div>