Function `ByT5Tokenizer.convert_tokens_to_string()` fails with certain tokens
## Environment info

- `transformers` version: 4.13.0.dev0
- Platform: Linux-5.11.0-37-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten 

## Information

After fine-tuning `ByT5`, the pipeline for `text2text-generation` fails when certain tokens are predicted and passed to the `ByT5Tokenizer.convert_tokens_to_string()` function.

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import pipeline

generate = pipeline("text2text-generation", model="versae/modernisa-pre")
generate("presuncion")
```

Which produces a string missing some characters:
```
[{'generated_text': 'presuncin'}]
```

## Expected behavior

I would have expected not to miss any of the characters is supposed to generate.

The error seems related to the way tokens are treated. A possible fix could be:

```python
from transformers import ByT5Tokenizer

class FixByT5Tokenizer(ByT5Tokenizer):

    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        bstring = b""
        for token in tokens:
            if token in self.special_tokens_decoder:
                tok_string = self.special_tokens_decoder[token].encode("utf-8")
            elif token in self.added_tokens_decoder:
                tok_string = self.special_tokens_decoder[token].encode("utf-8")
            elif token in self.special_tokens_encoder:
                tok_string = token.encode("utf-8")
            elif token in self.added_tokens_encoder:
                tok_string = token.encode("utf-8")
            else:
                tok_string = bytes(token, encoding="utf8")  # bytes([ord(token)])
            bstring += tok_string
        string = bstring.decode("utf-8", errors="ignore")
        return string

generate2 = pipeline("text2text-generation",
    model="versae/modernisa-pre",
    tokenizer=FixByT5Tokenizer("versae/modernisa-pre")
)
generate("presuncion")
```
Which correctly produces:
```
[{'generated_text': 'presunción'}]
```

However, it also makes other tests fail 🤷🏼 


Maybe related: https://github.com/huggingface/transformers/issues/13779.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.