ByT5 model
## ByT5

- Code: https://github.com/google-research/byt5
- Paper: https://arxiv.org/abs/2105.13626
- Twitter: https://twitter.com/colinraffel/status/1399525871678103552

- Ported checkpoints: https://huggingface.co/models?filter=arxiv:1907.06292

This model only requires a new tokenizer (and a tiny change to TFT5). New tokenizer does not require a vocab file as model just uses raw bytes.
Thank you so much for adding this model.

Is anyone else experiencing extremely slow training with it? I get 5 times longer training times on ByT5-large compared to mT5-large.

In the paper it's about 20% slower only.
![image](https://user-images.githubusercontent.com/35969959/121346695-b8205900-c926-11eb-8303-3ab335f77495.png)
<|||||>Hey @ViktorThink,

Could you maybe make two google colab using `mt5-small` and `byt5-small` that shows the different in training speed? :-)<|||||>Yes, I did a quick test just forward propagating. Seems like byt5-small takes about 4.5x times longer forward (backwards pass was fast during my tests). I think the simple explanation is that when tokenized based on utf8 instead of tokens, it generates about 4.5x more input tokens. I think I just misunderstood the paper, since it probably measured the speed per token, and not speed for a whole sentence. 

https://colab.research.google.com/drive/1Hv8XnggFscgb8M9UIEkkXYllwjFUyOB_?usp=sharing

Update:

I compared batch_size=1 with batch_size=2, and the models were almost equally fast for batch_size=1, but mT5 was even faster with batch_size=2 than with batch_size=1, while ByT5 was considerably slower. <|||||>I also see slow cpu inference - byT5-small has similar speed compared to mt5-xl.
And frankly, I do not understand how it can not be the case. The number of tokens is 5X larger in my test, so up to 25X more compute in self-attention.  Hidden dimension is 3X bigger in byt5 compare to mt5 (thus FFNs take 9X time more compute, and 45X more compute taking into account the number of tokens ). And everything is also multiplied by the 1.5X numbers of layers in byt5.

Sure the decoding is faster (and maybe for training it is not that obvious), but classification part of Table 10 leaves me puzzled. How can this be only 1.1X slower in terms of examples given at least 1.5X more layers and 3X bigger hidden dim? Is it some TPU magic?<|||||>I'm also curious now! Gently pinging the author @lintingxue maybe she knows more about it :-)<|||||>@lintingxue @patrickvonplaten  
Also here is the code I've been using for colab (maybe I did something wrong? - please let me know):
``` python
!pip install git+https://github.com/huggingface/transformers
!pip install pip install sentencepiece
import transformers
import torch
torch.backends.cudnn.benchmark = True
import time
article = """История «Твиттера» началась в марте 2006 года как научно-исследовательский проект компании Odeo (Сан-Франциско), первоначально для внутреннего использования. Джек Дорси ввёл понятие индивидуального пользования SMS-сервиса для общения с небольшой группой. Первоначально проект задумывался, как возможность ответить на единственный вопрос: «Что ты сейчас делаешь?»."""
torch.set_num_threads(1)
for model_name in ['google/mt5-small','google/mt5-base','google/mt5-large','google/byt5-small','google/byt5-base']:
  model = transformers.MT5EncoderModel.from_pretrained(model_name)#.cuda()
  tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
  input_ids = tokenizer(article, return_tensors="pt").input_ids#.cuda()
  print(input_ids.shape)  
  for idx in range(5):
      with torch.no_grad():
          t0=time.time()
          outputs = model(input_ids)
          #torch.cuda.synchronize()
          print(f"{model_name} {idx} {time.time() - t0} seconds")
  hidden_state = outputs.last_hidden_state  
  ```
  It gets:
  ```
  ...
 torch.Size([1, 98])
google/mt5-small 0 0.07804465293884277 seconds
google/mt5-small 1 0.07803463935852051 seconds
google/mt5-small 2 0.08781981468200684 seconds
google/mt5-small 3 0.08014941215515137 seconds
google/mt5-small 4 0.07851195335388184 seconds
...
torch.Size([1, 98])
google/mt5-base 0 0.31621241569519043 seconds
google/mt5-base 1 0.3263530731201172 seconds
google/mt5-base 2 0.32642054557800293 seconds
google/mt5-base 3 0.3143343925476074 seconds
google/mt5-base 4 0.32720518112182617 seconds
...
torch.Size([1, 98])
google/mt5-large 0 1.1833469867706299 seconds
google/mt5-large 1 1.160696268081665 seconds
google/mt5-large 2 1.1483042240142822 seconds
google/mt5-large 3 1.1961536407470703 seconds
...
torch.Size([1, 663])
google/byt5-small 0 4.315548419952393 seconds
google/byt5-small 1 4.416741371154785 seconds
google/byt5-small 2 4.385504722595215 seconds
google/byt5-small 3 4.426936149597168 seconds
...
torch.Size([1, 663])
google/byt5-base 0 8.502674579620361 seconds
google/byt5-base 1 8.467743635177612 seconds
google/byt5-base 2 8.519198656082153 seconds
google/byt5-base 3 8.492923974990845 seconds
google/byt5-base 4 8.318963766098022 seconds
<|||||>Btw, the ByT5-xxl  model in huggingface doesn't have a pytorch_model.bin so it's not possible to load, is it because it's new and will be added shortly?<|||||>byt5-xl is also not working (`pytorch_model.bin` is not correctly uploaded) btw. :)<|||||>thanks for letting me know guys - willl correct this!<|||||>Weights should be correctly uploaded now :-) <|||||>@patrickvonplaten the tokenizer also seems to be super slow.
Anybody else with such an experience? @stefan-it @ViktorThink <|||||>@PhilipMay - yes this doesn't surprise me tbh. I implemented the tokenizer in a way that fits the libraries' tokenizer design, which is by no means optimal in terms of speed. I think we should aim for a much faster Rust-backed tokenizer here - could we maybe re-use the fast Reformer tokenizer which also works on chars? cc @Narsil 