@RichardWang
I think, I am starting to see where I might be misunderstanding.  I’ll outline what I’ve been assuming so that any misunderstandings can be cleared up
I have been assuming that ElectraForPreTraining is supposed to handle training both the generator and the discriminator, as my understanding from the paper was that they were both trained concurrently. As a result, I was confused as to how ElectraForPreTraining could possibly train the generator without labels for the MLM. However , after reading your post I am starting to think that this assumption is incorrect. This leads me to the following questions:


Does ElectraForPreTraining only train the discriminator?


Am I supposed to first train an ElectraForMaskedLM to train the generator and then load it into ElectraForPreTraining in order to train the discriminator?


Is there any way to train the generator and discriminator at the same time using huggingface? As I mentioned before, it is my understanding from the paper that they are both supposed to be trained at the same time.