Pre-train DistilmByT5Neo
Let’s combine ByT5, mT5, GPT-Neo and DistilBert!
Model
For initial pre-training, a randomly initialized ByT5 model which we distil after pre-training is completed.
Datasets
c4multilingual 2
The Pile 3
Training Scripts
Training scripts will be created as part of the project
Expected Result
A Distilled model that combines the power of T5 and GPT-Neo while removing the need for tokenization.
UPDATE: We could just go for it and implement this with rotary embeddings as well.