<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.4.2</li>
<li>Platform: Linux-5.4.0-1041-aws-x86_64-with-debian-buster-sid</li>
<li>Python version: 3.7.10</li>
<li>PyTorch version (GPU?): 1.7.1 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Using GPU in script?: no</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): ProphetNet</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">I am using ProphetNet for some question generation work. I ran into a bug where I sampled from ProphetNet to generate a question generation using the <code class="notranslate">generate</code> function. I then took the outputted question and tokenized it and saw that the token_ids produced were different than the ones returned by the <code class="notranslate">generate</code> function. Here is a sample of the example I found to be inconsistent.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from transformers import ProphetNetTokenizer
&gt;&gt;&gt; tokenizer = ProphetNetTokenizer.from_pretrained(&quot;microsoft/prophetnet-large-uncased-squad-qg&quot;)
&gt;&gt;&gt; tokenizer.decode([2054, 2828, 1997, 6922, 2003, 1037, 12702, 5054, 17119, 1037, 2828, 1997, 1029, 102])
'what type of component is a microsencer a type of? [SEP]'
&gt;&gt;&gt; tokenizer.decode([2054, 2828, 1997, 6922, 2003, 1037, 12702, 3366, 5897, 2099, 1037, 2828, 1997, 1029, 102])
'what type of component is a microsencer a type of? [SEP]'
&gt;&gt;&gt; tokenizer.decode([3366, 5897, 2099])
'##sencer'
&gt;&gt;&gt; tokenizer.decode([5054, 17119])
'##sencer'
&gt;&gt;&gt; tokenizer.decode([5054, 17119]) == tokenizer.decode([3366, 5897, 2099])
True
&gt;&gt;&gt; tokenizer(&quot;what type of component is a microsencer a type of? [SEP]&quot;)[&quot;input_ids&quot;]
[2054, 2828, 1997, 6922, 2003, 1037, 12702, 5054, 17119, 1037, 2828, 1997, 1029, 102, 102]"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">ProphetNetTokenizer</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">ProphetNetTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"microsoft/prophetnet-large-uncased-squad-qg"</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>([<span class="pl-c1">2054</span>, <span class="pl-c1">2828</span>, <span class="pl-c1">1997</span>, <span class="pl-c1">6922</span>, <span class="pl-c1">2003</span>, <span class="pl-c1">1037</span>, <span class="pl-c1">12702</span>, <span class="pl-c1">5054</span>, <span class="pl-c1">17119</span>, <span class="pl-c1">1037</span>, <span class="pl-c1">2828</span>, <span class="pl-c1">1997</span>, <span class="pl-c1">1029</span>, <span class="pl-c1">102</span>])
<span class="pl-s">'what type of component is a microsencer a type of? [SEP]'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>([<span class="pl-c1">2054</span>, <span class="pl-c1">2828</span>, <span class="pl-c1">1997</span>, <span class="pl-c1">6922</span>, <span class="pl-c1">2003</span>, <span class="pl-c1">1037</span>, <span class="pl-c1">12702</span>, <span class="pl-c1">3366</span>, <span class="pl-c1">5897</span>, <span class="pl-c1">2099</span>, <span class="pl-c1">1037</span>, <span class="pl-c1">2828</span>, <span class="pl-c1">1997</span>, <span class="pl-c1">1029</span>, <span class="pl-c1">102</span>])
<span class="pl-s">'what type of component is a microsencer a type of? [SEP]'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>([<span class="pl-c1">3366</span>, <span class="pl-c1">5897</span>, <span class="pl-c1">2099</span>])
<span class="pl-s">'##sencer'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>([<span class="pl-c1">5054</span>, <span class="pl-c1">17119</span>])
<span class="pl-s">'##sencer'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>([<span class="pl-c1">5054</span>, <span class="pl-c1">17119</span>]) <span class="pl-c1">==</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>([<span class="pl-c1">3366</span>, <span class="pl-c1">5897</span>, <span class="pl-c1">2099</span>])
<span class="pl-c1">True</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">tokenizer</span>(<span class="pl-s">"what type of component is a microsencer a type of? [SEP]"</span>)[<span class="pl-s">"input_ids"</span>]
[<span class="pl-c1">2054</span>, <span class="pl-c1">2828</span>, <span class="pl-c1">1997</span>, <span class="pl-c1">6922</span>, <span class="pl-c1">2003</span>, <span class="pl-c1">1037</span>, <span class="pl-c1">12702</span>, <span class="pl-c1">5054</span>, <span class="pl-c1">17119</span>, <span class="pl-c1">1037</span>, <span class="pl-c1">2828</span>, <span class="pl-c1">1997</span>, <span class="pl-c1">1029</span>, <span class="pl-c1">102</span>, <span class="pl-c1">102</span>]</pre></div>
<p dir="auto">It seems odd there are these two different sets of token_ids that reproduce the same sentence. I am having trouble reproducing the second sequence of tokens. I was training the model when it produced this second sequence through sampling but I didn't save the model at that exact time.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">I would expect the tokenizer to always return the same sequence of tokens. I am not too familiar with how the tokenizers work, so I might be wrong in this assumption. Is this behavior expected and should I just try to avoid re-tokenizing whenever possible?</p>
<p dir="auto">I have used BERT NextSentencePredictor to find similar sentences or similar news, However, It's super slow. Even on Tesla V100 which is the fastest GPU till now. It takes around 10secs for a query title with around 3,000 articles. Is there a way to use BERT better for finding similar sentences or similar news given a corpus of news articles?</p>