Reproducing DistilRoBERTa
I've been trying to retrain DistilRoBERTa from the information given [here](https://huggingface.co/distilroberta-base) along with the example code/documentation [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation).

I'm a bit unclear on the exact configuration used to train the DistilRoBERTa model. I have been assuming it uses the same configuration as the DistilBERT model with minor changes, though some things, such as the loss coefficients are still a bit ambiguous.

**Would it be possible to share the exact command/configuration to train DistilRoBERTa?**

I've been able to replicate DistilRoBERTa to similar evaluation MLM perplexity but there still seems to be a small but statistically significant difference, I can share the full config if it's helpful.

Thank you!
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

You can ping @VictorSanh as he'll be the most helpful regarding distillation.

Thanks!<|||||>Apologies, I've posted to the forum [here](https://discuss.huggingface.co/t/reproducing-distilroberta/5217?u=davidharrison).

Thanks!