ELECTRA training reimplementation and discussion Research
  
  
    After months of development and debugging, I finally successfully train a model from scratch and replicate the official results. 

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators 
by Kevin Clark. Minh-Thang Luong. Quoc V. Le. Christopher D. Manning 

 Code: electra_pytorch 


AFAIK, the closest reimplementation to the original one, taking care of many easily overlooked details (described below). 


AFAIK, the only one successfully validate itself by replica…