GPT2 lm_labels masking using (-1) throws an index out of range
I am fine-tuning GPT2 model using the LMHead with a small number of special tokens. 
GPT2 underlying transformer takes the whole input at once, thus, it's important to pad inputs of varying lengths to a fixed length. The GPT2 model library offers -1 to be used as the padding value:

> lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].

**When I pad the lm_labels using -1's, the library throws an error.** On the other hand, using any other positive value for the masking works but the value acts like a vocabulary piece and thus makes the input incorrect. 

The other issue with **GPT2 is that the position_ids are compulsory**, whereas, the docs say they are optional.

Having an encoded and padded dataset like (not working case):

```
input_ids = torch.tensor([
    [[50257, 1212, 318, 43086, 2420, 2420, 2420, 50257]],
    [[50257, 1212, 318, 43086, 2420, 50257, 0, 0]]
])

position_ids = torch.tensor([
    [[1, 1, 1, 1, 1, 1, 1, 1]],
    [[1, 1, 1, 1, 1, 1, 0, 0]]
])

lm_labels = torch.tensor([
    [[1212, 318, 43086, 2420, 2420, 2420, 50257, 50257]],
    [[1212, 318, 43086, 2420, 2420, 2420, -1, -1]]
])

# Changing -1 padding makes the code work but also makes the input incorrect.
lm_labels = torch.tensor([
    [[1212, 318, 43086, 2420, 2420, 2420, 50257, 50257]],
    [[1212, 318, 43086, 2420, 50257, 50257, 0 , 0]]
])
```

@tholor Could you or someone from the team please fix the masking issue.

The full errors trace:

>   File "/Users/aw678/PycharmProjects/BERT/gpt2_simplified.py", line 181, in main
    losses, past = model(input_ids, position_ids, lm_labels, past=past)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 661, in forward
    hidden_states, presents = self.transformer(input_ids, position_ids, token_type_ids, past)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 590, in forward
    token_type_embeds = self.wte(token_type_ids)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/functional.py", line 1455, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191

Maybe you get the error because of the position_ids that are most likely wrong. 

I believe positional ids are not needed - you can use this:
predictions, past = model(tokens_tensor,position_ids=None token_type_ids=None, lm_labels=None, past=None)

and the use the parameters you want at the place you want or leave None if you do not want to use anything.

<|||||>With zeros, it is not padded, zeros are "!" not "[PAD]"<|||||>tokenizer.convert_tokens_to_ids('[PAD]') this results in 0. I am confused why it happens<|||||>Really? When I use decode tokenizer.decode(0) results in !.

tokenizer.encode(x) = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x)) [or should be], but when i try to use tokenizer.tokenize or tokenizer.convert_tokens_to_ids I am not sure which one, I get that its nto defined even tokenizer.encode works properly.<|||||>Does the 0 index actually stand for unknown tokens as well?
`tokenizer.convert_tokens_to_ids(["qrz"]) ` where "qrz" is supposed to be an unknown word. This will give [0]
But `tokenizer.convert_ids_to_tokens([0])` gives ["!"].
<|||||>That strange - we together  get that ! == [PAD]<|||||>There was some issue fine-tuning GPT-2 with the master branch. This should now be fixed with the merge of #560 (which also add access to GPT-2 medium model).<|||||>But to actually answer the discussion on this issue (sorry I was reading to quickly), there is no padding token in GPT-2 vocabulary.
So either you manage to not pad (which is how the model is pretrained) or you need to add a new token to the vocabulary using the special_token functions. This method explained for instance in our blog post on fine-tuning GPT/GPT-2 here: https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313<|||||>Thanks. So we need to add the special token and then fine tune it?


<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
