GPT2: how to construct batch for Language Modeling
I am a little confused about how to prepare input bathces for GPT2LMHeadModel. I want to use GPT2 as an LM. For instance, I want to generate probability distributions over the vocabulary at each timestep, as well as computing the perplexities of sentences. It is important to note that I am working with sentences and not documents, so I will have to pad the inputs in the batch.

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Prepare model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
model.to('cuda')

# input sentences
batch = ['this is a sentence.',
        'this is another sentence.',
        'this is another even longer sentence.']
```

## Question 1: Special tokens
a) Do I have to add a bos token id on my own or is it handled internally by GPT2Tokenizer? Same for the eos token.
```
# tokenize
tokens = [tokenizer.encode(x) for x in batch]

# add BOS and EOS
tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]  for x in tokens]
```
```
[[50256, 428, 318, 257, 6827, 13, 50256],
 [50256, 428, 318, 1194, 6827, 13, 50256],
 [50256, 428, 318, 1194, 772, 2392, 6827, 13, 50256]]
```
b) `tokenizer.encode(x)` gives me a warning "This tokenizer does not make use of special tokens. Input is returned with no modification." I replaced it with `tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x, add_prefix_space=True))` and the warning went away, but I am not sure what is the difference. Which tokenization should I use?

c) By looking at the properties of an instance of GPT2Tokenizer, I see that `bos_token` and `eos_token` are the same. Is this correct?


## Question 2: Padding
I want to pad based on the longest sentence in the batch. This is how I usually do it.
```
batch = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True).to('cuda')
```
```
tensor([[50256,   428,   318,   257,  6827,    13, 50256,     0,     0],
        [50256,   428,   318,  1194,  6827,    13, 50256,     0,     0],
        [50256,   428,   318,  1194,   772,  2392,  6827,    13, 50256]])
```
a) What id does the model expect for the padded tokens? Do I have to pass the token id as an argument to the model or the tokenizer or you have a predifined one?


## Question 3: Model input
How is GPT2 made aware of the padded steps? For instance for an RNN I would do something like this:
```
lengths = (batch != 0).sum(-1) # tensor([7, 7, 9])
packed = pack_padded_sequence(x, lengths, batch_first=True)
out_packed, hn = rnn(packed)
```
but for GPT2 I havent found an example. The only ones i found are with batch size 1. So something like this wont work as expected:
```
outputs = model(x, labels=x)  # labels are shifted inside the model, right?
loss, logits = outputs[:2]
```
---
# Update
So there are still things unclear, but from reading other issues this is my current understanding:
 - GPT2 has no padding token, as it was trained on documents and not sentences.
 - In order to use GPT2 with variable length inputs, we can apply padding with an arbitrary token and ensure that those tokens are not used by the model with an `attention_mask`.
 - As for the labels, we should replace **only** on the `labels` variable the padded token ids with `-1`. 
So based on that, here is my current toy implementation:
```python
inputs = [
    'this is a sentence.',
    'this is another sentence.',
    'this is another even longer sentence.', ]

# tokenize
# tokens = [tokenizer.encode(x) for x in batch]
tokens = [tokenizer.convert_tokens_to_ids(
    tokenizer.tokenize(x, add_prefix_space=True))
    for x in inputs]

# add BOS and EOS
tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]
          for x in tokens]

# padding_value can be whatever...
inputs = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True, padding_value=0).to('cuda')
# 1 for real tokens and 0 for padded tokens
mask = (inputs != 0).float()
# replace the ids of the padded tokens (where token_id==padded_id) with `-1`
labels = inputs.masked_fill(inputs == 0, -1)

outputs = model(inputs, attention_mask=mask, labels=labels)
loss, logits = outputs[:2]
```
Is this correct??

### Bug: Padded tokens are not excluded from the loss 
However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
```python
_logits = logits.view(-1, logits.size(-1))  # flatten logits
_labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')

print("GPT2 loss:", loss.item())
print("loss_naive_avg:", loss_naive_avg.item())
print("loss_real_avg:", loss_real_avg.item())
```
```
GPT2 loss: 4.664564609527588
loss_naive_avg: 4.664564609527588
loss_real_avg: 4.056143283843994
```
> I am a little confused about how to prepare input bathces for GPT2LMHeadModel. I want to use GPT2 as an LM. For instance, I want to generate probability distributions over the vocabulary at each timestep, as well as computing the perplexities of sentences. It is important to note that I am working with sentences and not documents, so I will have to pad the inputs in the batch.
> 
> ```python
> from transformers import GPT2Tokenizer, GPT2LMHeadModel
> 
> # Prepare model
> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
> model = GPT2LMHeadModel.from_pretrained('gpt2')
> model.eval()
> model.to('cuda')
> 
> # input sentences
> batch = ['this is a sentence.',
>         'this is another sentence.',
>         'this is another even longer sentence.']
> ```
> 
> ## Question 1: Special tokens
> a) Do I have to add a bos token id on my own or is it handled internally by GPT2Tokenizer? Same for the eos token.
> 
> ```
> # tokenize
> tokens = [tokenizer.encode(x) for x in batch]
> 
> # add BOS and EOS
> tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]  for x in tokens]
> ```
> 
> ```
> [[50256, 428, 318, 257, 6827, 13, 50256],
>  [50256, 428, 318, 1194, 6827, 13, 50256],
>  [50256, 428, 318, 1194, 772, 2392, 6827, 13, 50256]]
> ```
> 
> b) `tokenizer.encode(x)` gives me a warning "This tokenizer does not make use of special tokens. Input is returned with no modification." I replaced it with `tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x, add_prefix_space=True))` and the warning went away, but I am not sure what is the difference. Which tokenization should I use?
> 
> c) By looking at the properties of an instance of GPT2Tokenizer, I see that `bos_token` and `eos_token` are the same. Is this correct?
> 
> ## Question 2: Padding
> I want to pad based on the longest sentence in the batch. This is how I usually do it.
> 
> ```
> batch = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True).to('cuda')
> ```
> 
> ```
> tensor([[50256,   428,   318,   257,  6827,    13, 50256,     0,     0],
>         [50256,   428,   318,  1194,  6827,    13, 50256,     0,     0],
>         [50256,   428,   318,  1194,   772,  2392,  6827,    13, 50256]])
> ```
> 
> a) What id does the model expect for the padded tokens? Do I have to pass the token id as an argument to the model or the tokenizer or you have a predifined one?
> 
> ## Question 3: Model input
> How is GPT2 made aware of the padded steps? For instance for an RNN I would do something like this:
> 
> ```
> lengths = (batch != 0).sum(-1) # tensor([7, 7, 9])
> packed = pack_padded_sequence(x, lengths, batch_first=True)
> out_packed, hn = rnn(packed)
> ```
> 
> but for GPT2 I havent found an example. The only ones i found are with batch size 1. So something like this wont work as expected:
> 
> ```
> outputs = model(x, labels=x)  # labels are shifted inside the model, right?
> loss, logits = outputs[:2]
> ```
> 
> # Update
> So there are still things unclear, but from reading other issues this is my current understanding:
> 
> * GPT2 has no padding token, as it was trained on documents and not sentences.
> * In order to use GPT2 with variable length inputs, we can apply padding with an arbitrary token and ensure that those tokens are not used by the model with an `attention_mask`.
> * As for the labels, we should replace **only** on the `labels` variable the padded token ids with `-1`.
>   So based on that, here is my current toy implementation:
> 
> ```python
> inputs = [
>     'this is a sentence.',
>     'this is another sentence.',
>     'this is another even longer sentence.', ]
> 
> # tokenize
> # tokens = [tokenizer.encode(x) for x in batch]
> tokens = [tokenizer.convert_tokens_to_ids(
>     tokenizer.tokenize(x, add_prefix_space=True))
>     for x in inputs]
> 
> # add BOS and EOS
> tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]
>           for x in tokens]
> 
> # padding_value can be whatever...
> inputs = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True, padding_value=0).to('cuda')
> # 1 for real tokens and 0 for padded tokens
> mask = (inputs != 0).float()
> # replace the ids of the padded tokens (where token_id==padded_id) with `-1`
> labels = inputs.masked_fill(inputs == 0, -1)
> 
> outputs = model(inputs, attention_mask=mask, labels=labels)
> loss, logits = outputs[:2]
> ```
> 
> Is this correct??
> 
> ### Bug: Padded tokens are not excluded from the loss
> However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
> 
> ```python
> _logits = logits.view(-1, logits.size(-1))  # flatten logits
> _labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
> loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
> loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')
> 
> print("GPT2 loss:", loss.item())
> print("loss_naive_avg:", loss_naive_avg.item())
> print("loss_real_avg:", loss_real_avg.item())
> ```
> 
> ```
> GPT2 loss: 4.664564609527588
> loss_naive_avg: 4.664564609527588
> loss_real_avg: 4.056143283843994
> ```

What you proposed seems a valid walk-around to me. Also, look at #1464, which talked about adding `pad_token` to `tokenizer` and `embedding`. Perhaps that will help as well.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Check out [#3311](https://github.com/huggingface/transformers/issues/3311#issuecomment-601264426). GPT2 doesn't add BOS or EOS token, you must do it manually or use a tokenizer to do so. <|||||>> Bug: Padded tokens are not excluded from the loss
> 
> However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
> ```py
> _logits = logits.view(-1, logits.size(-1))  # flatten logits
> _labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
> loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
> loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')
> print("GPT2 loss:", loss.item())
> print("loss_naive_avg:", loss_naive_avg.item())
> print("loss_real_avg:", loss_real_avg.item())
> GPT2 loss: 4.664564609527588
> loss_naive_avg: 4.664564609527588
> loss_real_avg: 4.056143283843994

For this bug, you may need to set `ignore_index` to -1 instead of 0 in `F.cross_entropy` according to this line:
> ```py
> # replace the ids of the padded tokens (where token_id==padded_id) with `-1`
> labels = inputs.masked_fill(inputs == 0, -1)
> ```