<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): BART</p>
<p dir="auto">Language I am using the model on (English, Chinese ...): English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> an official GLUE/SQUaD task: (give the name): CNN/DM</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sshleifer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sshleifer">@sshleifer</a>,</p>
<p dir="auto">Thanks for the amazing model!<br>
I found a bug when alternatively trying to use <strong>forward</strong> to train the BartForConditionalGeneration and use <strong>generate</strong> to inference and evaluate the trained model. As shown in the code below, if I first use generate function then call forward function, the generation_mode attribute of decoder is set to True, and the shape of the decoder_output seems incorrect.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model = BartForConditionalGeneration.from_pretrained('bart-large-cnn')
tokenizer = BartTokenizer.from_pretrained('bart-large-cnn')

# input sequence
input_seq = &quot;Bart is a deep-learning pretrained model implemented in pytorch. It can smoothly handle summarization. It is a big model pretrained for generation tasks, especially summarizaition.&quot;
input_ids = torch.LongTensor([tokenizer.encode(input_seq)])

# expected output sequence
decoder_input_seq = &quot;Bart is a big pretrained deep model in pytorch for summarization.&quot;
decoder_input_ids = torch.LongTensor([tokenizer.encode(decoder_input_seq)])

# using generate method to inference
with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)
print(tokenizer.decode(result[0]))
# 'B. It is a big model pretrained for generation tasks, especially summarizaition. It'

# NOW use forward to train
result = model(input_ids, decoder_input_ids=decoder_input_ids)
# the shape of decoder_output and encoder_output
# what expected is: &lt;1, 18, 50264&gt; and &lt;40, 1, 1024&gt;
# but actual output is:  torch.Size([1, 1, 50264]) torch.Size([40, 1, 1024])
print(result[0].shape, result[2].shape)
"><pre class="notranslate"><code class="notranslate">model = BartForConditionalGeneration.from_pretrained('bart-large-cnn')
tokenizer = BartTokenizer.from_pretrained('bart-large-cnn')

# input sequence
input_seq = "Bart is a deep-learning pretrained model implemented in pytorch. It can smoothly handle summarization. It is a big model pretrained for generation tasks, especially summarizaition."
input_ids = torch.LongTensor([tokenizer.encode(input_seq)])

# expected output sequence
decoder_input_seq = "Bart is a big pretrained deep model in pytorch for summarization."
decoder_input_ids = torch.LongTensor([tokenizer.encode(decoder_input_seq)])

# using generate method to inference
with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)
print(tokenizer.decode(result[0]))
# 'B. It is a big model pretrained for generation tasks, especially summarizaition. It'

# NOW use forward to train
result = model(input_ids, decoder_input_ids=decoder_input_ids)
# the shape of decoder_output and encoder_output
# what expected is: &lt;1, 18, 50264&gt; and &lt;40, 1, 1024&gt;
# but actual output is:  torch.Size([1, 1, 50264]) torch.Size([40, 1, 1024])
print(result[0].shape, result[2].shape)

</code></pre></div>
<p dir="auto">Such issue can <strong>seemingly</strong> be addressed by mannually setting the generation_mode.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# mannually set the generation mode to False **seemingly** fix the issue
model.model.decoder.generation_mode=False
result = model(input_ids, decoder_input_ids=decoder_input_ids)
print(result[0].shape, result[2].shape)
# output is: torch.Size([1, 18, 50264]) torch.Size([40, 1, 1024]) and make sense"><pre class="notranslate"><code class="notranslate"># mannually set the generation mode to False **seemingly** fix the issue
model.model.decoder.generation_mode=False
result = model(input_ids, decoder_input_ids=decoder_input_ids)
print(result[0].shape, result[2].shape)
# output is: torch.Size([1, 18, 50264]) torch.Size([40, 1, 1024]) and make sense
</code></pre></div>
<p dir="auto">However, now the output of the forward function doesn't make sense, as shown below.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)

model.model.decoder.generation_mode=False
new_result = model(input_ids, decoder_input_ids=result)

print(tokenizer.decode(torch.argmax(new_result[0][0], dim=1)))
# ' and and and and and and and and and and and and and and and and and and and'"><pre class="notranslate"><code class="notranslate">with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)

model.model.decoder.generation_mode=False
new_result = model(input_ids, decoder_input_ids=result)

print(tokenizer.decode(torch.argmax(new_result[0][0], dim=1)))
# ' and and and and and and and and and and and and and and and and and and and'
</code></pre></div>
<p dir="auto">In expectation, when we feed the generated decoder sequence to the decoder and remain the input of the encoder unchanged, the output of the decoder would at least resemble the decoder_input. However, the actual output of the model is <code class="notranslate">' and and and and and and and and and and and and and and and and and and and'</code>, which is definately not a reasonable output of a trained decoder.</p>
<p dir="auto">To go deeper, I tested different samples, all feeding the generated output of a encoder_input to the decoder and inspect the decoder_output (which "should" resemble decoder_input, i.e. generated sequence), the same happens everytime for me (most output is the duplication of a single token for many times, like "the the the ...", "and and and...", ". . . . ").</p>
<p dir="auto">Given all these results, I think there is a bug in either the implementation of <strong>generate</strong>  or that of <strong>forward</strong>, the revert of generation_mode is simple, but I don't think the output of forward is a reasonable result currently. Could you please look into the issue?</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sshleifer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sshleifer">@sshleifer</a> If I'm not using those functions in a correct manner, any advise or instruction is welcomed! Many thanks for the help!</p>

<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: master-branch</li>
<li>Platform: windows 10</li>
<li>Python version: 3.7.0</li>
<li>PyTorch version (GPU?):  1.4.0</li>
<li>Tensorflow version (GPU?): /</li>
<li>Using GPU in script?: Yes</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h2 dir="auto">❓ Questions &amp; Help</h2>

<p dir="auto">Hi there,<br>
when I load the pretrained Camenbert model and tokenizer via</p>
<p dir="auto"><code class="notranslate">model = CamembertForMaskedLM.from_pretrained('camembert-base') tokenizer = CamembertTokenizer.from_pretrained('camembert-base')</code></p>
<p dir="auto">the length of the tokenizer is 32004 but the vocab_size of the model is 32005.<br>
<code class="notranslate">print(len(tokenizer))</code><br>
'print(model.config.vocab_size'</p>
<p dir="auto">This throws me an error</p>
<blockquote>
<p dir="auto">Index out of range</p>
</blockquote>
<p dir="auto">when I try to adapt the lm_finetuning example because of<br>
<code class="notranslate">model.resize_token_embeddings(len(tokenizer))</code></p>
<p dir="auto">It runs when I comment out this line. So my question is, is this the intended behaviour resp. what's the reason for the unevenness between the tokenizer and the model vocab_size?</p>