<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version:4.3.3</li>
<li>Platform:Pytorch</li>
<li>Python version:3.7.0</li>
<li>PyTorch version (GPU?):GPU</li>
<li>Tensorflow version (GPU?):</li>
<li>Using GPU in script?:</li>
<li>Using distributed or parallel set-up in script?:NO</li>
</ul>
<h3 dir="auto">Who can help</h3>

<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...):</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the official example scripts: (give details below)<br>
<a href="https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py">https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">1.python run_mlm.py     --model_name_or_path release_model/     --dataset_name wikitext     --dataset_config_name wikitext-2-raw-v1     --do_train     --do_eval     --output_dir test-mlm --max_seq_length 128<br>
2.Got an error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[INFO|trainer.py:1408] 2021-03-15 11:10:32,884 &gt;&gt; Saving model checkpoint to test-mlm
[INFO|configuration_utils.py:304] 2021-03-15 11:10:32,886 &gt;&gt; Configuration saved in test-mlm/config.json
[INFO|modeling_utils.py:817] 2021-03-15 11:10:33,863 &gt;&gt; Model weights saved in test-mlm/pytorch_model.bin
Traceback (most recent call last):
  File &quot;run_mlm.py&quot;, line 475, in &lt;module&gt;
    main()
  File &quot;run_mlm.py&quot;, line 450, in main
    trainer.log_metrics(&quot;train&quot;, metrics)
AttributeError: 'Trainer' object has no attribute 'log_metrics'"><pre class="notranslate"><code class="notranslate">[INFO|trainer.py:1408] 2021-03-15 11:10:32,884 &gt;&gt; Saving model checkpoint to test-mlm
[INFO|configuration_utils.py:304] 2021-03-15 11:10:32,886 &gt;&gt; Configuration saved in test-mlm/config.json
[INFO|modeling_utils.py:817] 2021-03-15 11:10:33,863 &gt;&gt; Model weights saved in test-mlm/pytorch_model.bin
Traceback (most recent call last):
  File "run_mlm.py", line 475, in &lt;module&gt;
    main()
  File "run_mlm.py", line 450, in main
    trainer.log_metrics("train", metrics)
AttributeError: 'Trainer' object has no attribute 'log_metrics'
</code></pre></div>

<h2 dir="auto">Expected behavior</h2>

<p dir="auto">The script finish without error.</p>
<p dir="auto">I've been trying to work out why I keep getting a CUDA assert in a specific mini-batch when training RoBERTa from scratch. I finally tracked it down after switching to CPU.</p>
<p dir="auto">I don't understand why <code class="notranslate">padding_idx</code> is added to <code class="notranslate">incremental_indices</code> below? - <em>edit: I do in that the embedding needs a padding mask but it I'm not sure it's the correct way to do it.</em></p>
<p dir="auto">In my case padding_idx=3. And one of my input_ids rows was truncated. Say I have input_ids = [[4,5,6],[4,3,3]], this results in mask=[[1,1,1],[1,0,0]] and incremental index=[[1,2,3],[1,0,0]]. Adding padding_idx then produces [[4,5,6],[4,3,3]].</p>
<p dir="auto">The issue is <code class="notranslate">self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)</code> so for any sequences which are truncated adding anything to the indices results in an index which is greater than the embedding dim.</p>
<p dir="auto">Perhaps you can argue that max_position_embeddings is supposed to be larger than the largest possible sequence so this doesn't happen? There is a check in <code class="notranslate">run_mlm.py</code> that <code class="notranslate">data_args.max_seq_length &gt; tokenizer.model_max_length</code> but it seems that in actual fact to avoid a very hard to track down error you must have tokenizer.model_max_length &lt; data_args.max_seq_length</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def create_position_ids_from_input_ids(input_ids, padding_idx):
    &quot;&quot;&quot;
    Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols
    are ignored. This is modified from fairseq's `utils.make_positions`.

    Args:
        x: torch.Tensor x:

    Returns: torch.Tensor
    &quot;&quot;&quot;
    # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
    mask = input_ids.ne(padding_idx).int()
    incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
    return incremental_indices.long() + padding_idx"><pre class="notranslate"><code class="notranslate">def create_position_ids_from_input_ids(input_ids, padding_idx):
    """
    Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols
    are ignored. This is modified from fairseq's `utils.make_positions`.

    Args:
        x: torch.Tensor x:

    Returns: torch.Tensor
    """
    # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
    mask = input_ids.ne(padding_idx).int()
    incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
    return incremental_indices.long() + padding_idx
</code></pre></div>