Really like to see this project! Think most of the code exists (the T5-like pretraining code will be merged today/tomorrow for JAX, see: https://github.com/huggingface/transformers/pull/12355 6).
It will be important to decide on which languages exactly ByT5 should be pretrained on! Think it’s a good idea to make sure that the languages are related/close to each other.
Putting everybody that is interested in an official project for now