BlenderBot RuntimeError: CUDA error: device-side assert triggered
## Environment info

- `transformers` version: 4.0.0
- Platform: Linux-5.4.0-56-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes  (GTX 1060 6GB)
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): I am using the BlenderbotForConditionalGeneration ('facebook/blenderbot-90M') along with the relevant small tokenizer.

The problem arises when using:

I am using my own trainer implementation. I think that the problem has to do with the indexes of the labels. More specifically when I am using:

```outputs = self.model(input_ids=inputs, attention_mask=inputs_att, labels=pad_targets, return_dict=True)```

everything works fine as the "pad_targets" are the targets using 0  as the index for masked (padded) tokens.
However when I am using:

```outputs = self.model(input_ids=inputs, attention_mask=inputs_att,  labels=repl_targets, return_dict=True)```
and then printing the outputs['loss'] the following error is occurred:

`RuntimeError: CUDA error: device-side assert triggered`

as the "repl_targets" are the targets using the -100  as the index for masked (padded) tokens.

The aforementioned error also occurs when using the argument:
`decoder_input_ads=repl_targets`

The tasks I am working on is:
 Dialogue generation in Empathetic Dialogues dataset.

## Expected behavior

I think that there is a problem with the -100 padding token. But I am not sure :)
Hey @manzar96,

It would be awesome if you could provide a full code snippet that I can copy paste and run to reproduce the error. I am not able to do so with your code above. 

Thanks a lot!<|||||>I made an example:

```import torch
from transformers import BlenderbotSmallTokenizer, \
    BlenderbotForConditionalGeneration

DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'

model = BlenderbotForConditionalGeneration.from_pretrained('facebook/blenderbot-90M')
model.to(DEVICE)
inputs = torch.tensor([[14, 49, 42, 626, 2727, 1063, 5, 0, 0, 0, 0, 0, 0, 0],
                       [14, 1322, 7, 1427, 13, 7, 153, 384, 5, 14,
                        18,   64, 7261,    5]], device=DEVICE)

inputs_att = torch.tensor([[1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]],
       device=DEVICE)

repl_targets = torch.tensor([[  46,   15, 3283,   20, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100],
        [ 121,   54,   37,   53,   60,   12,  447,   10, 1427,   15,   51,   11,
          598,   20]], device=DEVICE)

pad_targets = torch.tensor([[  46,   15, 3283,   20,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0],
        [ 121,   54,   37,   53,   60,   12,  447,   10, 1427,   15,   51,   11,
          598,   20]], device=DEVICE)


outputs=model.forward(input_ids=inputs, attention_mask=inputs_att,
                             labels=repl_targets, return_dict=True)
import ipdb;ipdb.set_trace()
```


If you try printing the outputs['loss'] the error occurs. However, if you replace the  `repl_targets` with the `pad_targets` variable everything works fine (but the loss does not mask 0, so that's not always correct for use).<|||||>@patrickvonplaten 

This is a bug, in bart `decoder_input_ids` are prepared by shifting the `labels` to right, but it doesn't replace -100 with `pad_token_id`. 
https://github.com/huggingface/transformers/blob/6587cf9f8448b5573cf4a1c639ef4857472d1da0/src/transformers/models/bart/modeling_bart.py#L65-L73

In T5 we automatically replace -100 with `pad_token_id` when preparing `decoder_input_ids`.
https://github.com/huggingface/transformers/blob/6587cf9f8448b5573cf4a1c639ef4857472d1da0/src/transformers/models/t5/modeling_t5.py#L740-L756<|||||>You're right @patil-suraj - do you want to open a PR to fix it in Bart? :-) <|||||>Yeah!