CANINE pre-training
# 🚀 Feature request
Thanks for the integration of the Canine model. I am interested in pre-training the model from scratch and I was wondering if you have a timeline for the release of a pre-training script using autoregressive character loss.

Thank you in advance.

@NielsRogge 
Hi,

Google hasn't released any pre-training code yet. As stated on their [README](https://github.com/google-research/language/tree/master/language/canine#pre-training-code-coming-later):

> Pre-training Code (Coming later)
We've prioritized releasing the pre-trained checkpoints, modeling code, and TyDi QA evaluation code since we hope this will cover the most common use cases. The implementation of pre-training will be released in this repo in the future. If this is blocking you, feel free to send us a friendly ping to let us know that this is important to you.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.