This is awesome. Thanks for sharing. I plan to warm start with google’s pre-trained models and continue pre-training on my domain-specific corpus. Can I use the same script for continual pre-training. The only changes would be to load generator and discriminator weights using ElectraForPreTraining.from_pretrained(“google/disc”) right?  Thanks in advance.