GPT2 doesn't accept inputs of varying tokens length (despite the padding at the end)
I have noted a very strange behaviour in GPT2 and I can't figure out why this happens. In one case when all of the inputs in the dataset have the same token length, the training works, however, when only one of the inputs has a different token length, the library throws an error. This is very strange since before I feed the inputs into the model I have a method which takes care of the padding so that every input is of a fixed shape/length.

**The working case pseudocode:**
Note that the dataset is in tensor type and when fed to the model is in shape specified in docs (n_batch, input_len)

dataset = [
    [50258,  1212,   318,   617, 43086,  2420, 50258],
    [50258,  318,    1212 , 617, 43086,  2420, 50258],
    [50258,  1212,   318,   617, 43086,  2420, 50258],
]
_(all of the inputs in the dataset have the same token length)_

**The not working case:**
dataset = [
    [50258,  1212,   318,   617, 43086,  2420, 50258],
    [50258,  1212,   318,   617, 43086, 50258,         0],
    [50258,  1212,   318,   617, 43086,  2420, 50258],
]
_(e.g. the second input has a different number of tokens than the other two, however, is padded with a 0 so that all of the inputs are of the same size)_

In fact, when the dataset consists of the same token length but has extra padding at the end, also throws an error:
dataset = [
    [50258,  1212,   318,   617, 43086,  2420, 50258, 0, 0],
    [50258,  318,    1212 , 617, 43086,  2420, 50258, 0, 0],
    [50258,  1212,   318,   617, 43086,  2420, 50258, 0, 0],
]

A toy example to replicate this error: 
[gpt2_simplified.py.zip](https://github.com/huggingface/pytorch-pretrained-BERT/files/3139047/gpt2_simplified.py.zip)

The full error traceback is:
> Epoch:   0%|          | 0/3 [00:00<?, ?it/s]
Training:   0%|          | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/local_path/gpt2_simplified.py", line 166, in <module>
    main()
  File "/local_path/**gpt2_simplified.py**", line 142, in main
    **losses, past = model(input_ids, position_ids, lm_labels, past=past)**
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/**module.py**", line 489, in __call__
    **result = self.forward**(*input, **kwargs)
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/**modeling_gpt2.py**", line 661, in forward
    **hidden_states, presents = self.transformer(input_ids, position_ids, token_type_ids, past)**
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/**module.py**", line 489, in __call__
    **result = self.forward**(*input, **kwargs)
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/**modeling_gpt2.py**", line 587, in forward
    **position_embeds = self.wpe(position_ids)**
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/**module.py**", line 489, in __call__
    **result = self.forward**(*input, **kwargs)
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/**sparse.py**", line 118, in forward
    **self.norm_type, self.scale_grad_by_freq, self.sparse)**
  File "/local_path/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/**functional.py**", line 1454, in embedding
    **return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191**

Thanks for the help in advance.

@thomwolf I was wondering what are your thought on this issue?<|||||>The bug in the library causing the index out of range error comes from masking (-1) the LM labels.

> lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].

When I pad the lm_labels to a fixed size using -1's, the library throws an error, on the other hand, using any other positive value works but acts like a vocabulary piece and thus makes the input incorrect.
@thomwolf Could someone please fix this?<|||||>Yes there is a PR (#560) fixing a number of issues for fine-tuning GPT-2.
Should be merged soon (this week hopefully).<|||||>Any update on #560? <|||||>I guess that you should try add pading at the begining - it predicts next word not first so the padding should be added in front.
[50258, 1212, 318, 617, 43086, 50258, 0] should be
[0,50258, 1212, 318, 617, 43086, 50258]<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I've checked that outputs from right-padded input tensors and from no-padding input tensors are different. Personally, the latter makes little more sense. A module for input masking and its corresponding position embeddings need to be implemented. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
