@rsvarma

yes
No, train the two models at the same time.
Hmm. You are searching for ELETRA training script. There are some implementations use huggingface.


simpletransformers
Although I didn’t read all the code, I found there is something different to the paper. (e.g. didn’t tie also layernorm and dropout of embedding layers of generator and discriminator)
ELECTA trainer from huggingface/ transformer itself
But it is still a pr and got stuck by some problems for months. And use a different sampling method from the paper’s.

My implementation 6
I personally believe it is most close to the official implementation. But I am doing the final test and haven’t clean my code and upload docs.

The first two didn’t validate the code by replicating the paper’s results. As to my implementation, I am doing it, but the release might take at least 1 week or longer.