Decoding error while using DataCollatorForSeq2Seq
### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no>

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I decode data from DataCollatorForSeq2Seq, I get OverflowError with fast tokenizer and TypeError with default tokenizer.

Code example:
```
model_name = "facebook/bart-base"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
data_collator = DataCollatorForSeq2Seq(tokenizer, model=None)

texts = ["text" * 5, "text" * 10]
labels = ["label" * 5, "label" * 10]
features = [
    {
        "input_ids": tokenizer(i)['input_ids'],
        "labels": tokenizer(j)['input_ids']
    }
    for i,j in zip(texts, labels) 
]
result = data_collator(features)
print(result["labels"][0])
print(tokenizer.decode(result["labels"][0], skip_special_tokens=True))
```

Stack trace in case `use_fast=False`:
```
TypeError                                 Traceback (most recent call last)
[<ipython-input-7-14a1328c21fe>](https://localhost:8080/#) in <cell line: 18>()
     16 result = data_collator(features)
     17 print(result["labels"][0])
---> 18 print(tokenizer.decode(result["labels"][0], skip_special_tokens=True))

2 frames
[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py](https://localhost:8080/#) in decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
   3507         token_ids = to_py_obj(token_ids)
   3508 
-> 3509         return self._decode(
   3510             token_ids=token_ids,
   3511             skip_special_tokens=skip_special_tokens,

[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py](https://localhost:8080/#) in _decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, spaces_between_special_tokens, **kwargs)
    947                 current_sub_text.append(token)
    948         if current_sub_text:
--> 949             sub_texts.append(self.convert_tokens_to_string(current_sub_text))
    950 
    951         if spaces_between_special_tokens:

[/usr/local/lib/python3.10/dist-packages/transformers/models/bart/tokenization_bart.py](https://localhost:8080/#) in convert_tokens_to_string(self, tokens)
    305     def convert_tokens_to_string(self, tokens):
    306         """Converts a sequence of tokens (string) in a single string."""
--> 307         text = "".join(tokens)
    308         text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
    309         return text

TypeError: sequence item 9: expected str instance, NoneType found
```

Stack trace in case `use_fast=True`:
```
OverflowError                             Traceback (most recent call last)
[<ipython-input-8-d0724246272d>](https://localhost:8080/#) in <cell line: 18>()
     16 result = data_collator(features)
     17 print(result["labels"][0])
---> 18 print(tokenizer.decode(result["labels"][0], skip_special_tokens=True))

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py](https://localhost:8080/#) in decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
   3507         token_ids = to_py_obj(token_ids)
   3508 
-> 3509         return self._decode(
   3510             token_ids=token_ids,
   3511             skip_special_tokens=skip_special_tokens,

[/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py](https://localhost:8080/#) in _decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
    544         if isinstance(token_ids, int):
    545             token_ids = [token_ids]
--> 546         text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
    547 
    548         clean_up_tokenization_spaces = (

OverflowError: out of range integral type conversion attempted
```


Also, if I use facebook/m2m100_418M there are no errors, but the result of decoding looks like (though I use `skip_special_tokens=True`) this:
```
labellabellabellabellabel<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
```

### Expected behavior

Hello! I have expected no errors and that skip_special_tokens would work normally. Seems like the label padding in DataCollatorForSeq2Seq with using -100 is leading to error.

I think these issue are related to this:
#22634
[this](https://github.com/huggingface/transformers/issues/3853#issuecomment-770417239)
Yes, you need to replace the labels indices that are at -100 to be able to decode them. The -100 indicates to PyTorch the corresponding token should be ignored during the loss computation, this is not a bug.

Also please use the [forums](https://discuss.huggingface.co/) for questions like this.<|||||>OK, thank you so much.