How to kill bad starts when pre-training from scratch
### Environment info
transformers version: 4.4.3
Platform: linux
Python version: 3.8.5
PyTorch version (GPU?): -
Tensorflow version (GPU?): 2.4.1
Using GPU in script?: yes
Using distributed or parallel set-up in script?: parallel

### Information
Hi!
I am pre-training a RoBERTa model from scratch and was wondering about the possibility of killing bad starts. Because the model will be initiated with random weights when pre-training from scratch, and these initial weights might influence the performance of the final model, I want to do my best to at least not get the worst weight initialization. I have heard that it might be a possibility to calculate perplexity and let that score be decisive of whether to kill the training process or not. Does anyone have experience with how to do this, or does someone have a better idea to review weight initialization and kill bad starts?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.