Behavior of shift_tokens_right on padded input_ids
### System Info

- `transformers` version: 4.20.1


### Who can help?

@patrickvonplaten @patil-suraj @sgugger 

### Reproduction

When I applied shift_tokens_right on a padded input_ids, I get this 
```python
from transformers import AutoTokenizer
from transformers.models.bart.modeling_flax_bart import shift_tokens_right

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
labels = tokenizer("My dog is cute", padding='max_length', max_length=8, return_tensors='np').input_ids
decoder_input_ids = shift_tokens_right(labels, tokenizer.pad_token_id, tokenizer.eos_token_id)

print(tokenizer.batch_decode(labels))
# ['<s>My dog is cute</s><pad><pad>']

print(tokenizer.batch_decode(decoder_input_ids))
# ['</s><s>My dog is cute</s><pad>']
```


### Expected behavior

Should the desired behavior of shift_token_right be the following?

```python
print(tokenizer.batch_decode(labels))
# ['<s>My dog is cute</s><pad><pad>']

print(tokenizer.batch_decode(decoder_input_ids))
# ['</s><s>My dog is cute<pad><pad>']
```
Hey @duongna21! This method isn't exposed in the main init so we consider it to be private (this should really be written somewhere if we haven't done so yet).

It's used internally by the BART model, but we don't validate it to work for any other purpose.<|||||>@LysandreJik Yeah, I specifically raise above question in the context of BART training. IMO `shift_tokens_right` should not take the `<pad>` token into account (this is actually the behavior of [fairseq's original code](https://github.com/facebookresearch/fairseq/blob/8e804cb38a1575c65a1fc981d75ae5a97c24dd5b/fairseq/data/data_utils.py#L69)). 
Also, I believe this issue also applies to other models using `shift_tokens_right`, such as T5.<|||||>Any comment? I'm happy to create a PR if my assumption is correct.<|||||>Pinging @patil-suraj and @patrickvonplaten regarding the `shift_tokens_right` method and its purpose.<|||||>Hey @<|||||>Hey @duongna21,

Good question! Note however that it doesn't really matter whether you pass

```py
['</s><s>My dog is cute<pad><pad>']
```

or

```python
['</s><s>My dog is cute</s><pad>']
```

to the model if the labels are:

```py
['<s>My dog is cute</s><pad><pad>']
```

because every loss token that gets mapped to the `<pad>` token is ignored. More specifically this means that during training the following happens:
- The model learns that `</s>` should predict `<s>`
- then `</s><s>` should predict `My`
- then `</s><s>My` should predict `dog`
- ... until
- `</s><s>My dog is cute` should predict `</s>`
**Now** it doesn't matter whether `</s><s>My dog is cute</s>` or `</s><s>My dog is cute<pad>` is passed because both will be ignored as the predicted token is `<pad>` and the model should never learn to predict pad tokens -> so this loss will be ignored. Does this make sense?
<|||||>@patrickvonplaten Thanks for elaborating on it. Totally agree with you that it doesn't matter with seq2seq training. I just tried to make sure it doesn't create any side-effect somewhere :D. 