ByT5 tokenizer gives indices of chars instead of bytes
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.2
- Platform: Linux-5.4.0-77-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.9.0.dev20210415+cu101 (True)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help

ByT5: @patrickvonplaten
Documentation: @sgugger

## Information
ByT5 Tokenizer does incorrect tokenization.

In ByT5 documentation, there are two examples of encoding text. One is `torch.tensor([list("Life is like a box of chocolates.".encode("utf-8"))]) + 3` and the other is with dedicated tokenizer. However, the tokenizer approach is giving the wrong indices.

Going deep into ByT5 tokenizer implementation, indices are given by `ord(token)`. However, this returns [an integer representing the Unicode code point of that character. ](https://docs.python.org/3/library/functions.html#ord) But we need an integer representing byte!


From the official python [documentation](https://docs.python.org/3/howto/unicode.html):

>A Unicode string is a sequence of **code points**, which are numbers from 0 through 0x10FFFF (1,114,111 decimal). This sequence of code points needs to be represented in memory as a set of code units, and code units are then mapped to 8-bit bytes.
>UTF-8 uses the following rules:
>
>* If the code point is < 128, it’s represented by the corresponding byte value.
>
>* If the code point is >= 128, it’s turned into a sequence of two, three, or four bytes, where each byte of the sequence is between 128 and 255.


So, in essence, everything works until we have code points up to 128, as integers for code points and bytes are the same. But going further, for example, accented letter *č*, tokenizer misbehaves. Characters with even higher indices can not be mapped into embeddings of size (`Embedding(384, 1472)`. 

By the way, why Embedding is not the size of 259 (255 bytes + 3 special characters)??? `T5ForConditionalGeneration.from_pretrained('google/byt5-small').get_input_embeddings()`
## To reproduce
```
accent_leetter = "č"

list([i+3 for i in accent_leetter.encode("utf-8")]), tok.encode(accent_leetter, add_special_tokens=False)
```
output:
```
([199, 144], [272])
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

output:
```
([199, 144], [199, 144])
```

@LukasStankevicius could you please check that you're using latest `master` or `4.11.0.dev0` :thinking: 

I'm using the official tokenization implementation from ByT5 - which comes from `seqio`:

```python
!pip3 install seqio

from seqio import ByteVocabulary
tokenizer = ByteVocabulary()

tokenizer._encode("č")

# Outputs:
# [199, 144]
```

Hugging Face ByT5 implementation also returns:

```bash
from transformers import ByT5Tokenizer

hf_tokenizer = ByT5Tokenizer.from_pretrained("google/byt5-base")
hf_tokenizer.encode("č", add_special_tokens=False)

# Outputs
# [199, 144]
```<|||||>> @LukasStankevicius could you please check that you're using latest `master` or `4.11.0.dev0` 🤔
> 
> I'm using the official tokenization implementation from ByT5 - which comes from `seqio`:
> 
> ```python
> !pip3 install seqio
> 
> from seqio import ByteVocabulary
> tokenizer = ByteVocabulary()
> 
> tokenizer._encode("č")
> 
> # Outputs:
> # [199, 144]
> ```
> 
> Hugging Face ByT5 implementation also returns:
> 
> ```shell
> from transformers import ByT5Tokenizer
> 
> hf_tokenizer = ByT5Tokenizer.from_pretrained("google/byt5-base")
> hf_tokenizer.encode("č", add_special_tokens=False)
> 
> # Outputs
> # [199, 144]
> ```

Yes, that works!  I was at 4.10.2 down from 4.11.0.dev0