I’m not sure if fine-tuning ELECTRA with MLM is good idea since the main idea behind ELECTRA was to train it as a discriminator rather than generator and overcome the problems of MLM’s
Any thoughts @lysandre