I have fixed several bugs to get closer to the official ELECTRA. And I found the content of BookCorpus hubbed on HuggingFace now is scattered, so I choose to switch to OpenWebText corpus, which the authors also train small model on.
If you are using the old version of this implementation, be sure to git pull and pip install -r requirements.txt