BART cannot accept -100 as ignored label
## Environment info

- `transformers` version: 4.0.1
- Platform: Linux
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

 Bart: @patrickvonplaten

## Information

I'm using ``BartForConditionalGeneration`` to do some natural language generation tasks. By the [doc](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration) I should be able to set -100 for some tokens to ignore. However, it would raise an out of index error.

## To reproduce

```python
from transformers import BartForConditionalGeneration, AutoTokenizer

b = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
t = AutoTokenizer.from_pretrained("facebook/bart-base")

s1 = "hello hello hello hello world"
inputs = t(s1, return_tensors="pt")
label = inputs["input_ids"].clone()
label[0, 2:3] = -100

outputs = b(**inputs, labels=label)
```

Then it raise the following error:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1032, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 915, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 564, in forward
>     x = self.embed_tokens(input_ids) * self.embed_scale
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
>     self.norm_type, self.scale_grad_by_freq, self.sparse)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
> IndexError: index out of range in self

Without giving -100 in the label, it can return the output correctly.

## Expected behavior

Should return the output correctly.