When will ELECTRA pretraining from  scratch will be available?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
Is pre training from scratch for electra available? couldnt find it. 
Thanks!

Working on it as we speak :).

I'd say it will be out in a few weeks at most.<|||||>Is Albert pretraining from scratch is available? @LysandreJik 
<|||||>@LysandreJik do you think the update will be available by the end of the month ? Maybe it has been postponed due to the recent addition of the Trainer and the refactor of the language_modeling script ?<|||||>It has been postponed a bit due to the recent addition of the Trainer and the TPU work on it, but I definitely aim to have it out earlier than by the end of the month :)<|||||>Was looking for Albert pre-training from scratch but I think there is support for Bert, Roberta and distillbert only as of now. 
@LysandreJik can you guide how can I do Albert pretraining from scratch?<|||||>@LysandreJik Is the code for pretraining Electra from scratch available now?
<|||||>> @LysandreJik Is the code for pretraining Electra from scratch available now?

Not yet. There's a PR about it;
https://github.com/huggingface/transformers/pull/4656<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Are there any updates to this, or plans to release the ELECTRA pre-training from scratch feature soon?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hello. 
Are there any updates?<|||||>Curious as well.<|||||>The development of the ELECTRA pretraining from scratch is in a stale state with no plans to work on it further, see https://github.com/huggingface/transformers/pull/4656#issuecomment-711082850

See https://discuss.huggingface.co/t/electra-training-reimplementation-and-discussion/1004 by @richarddwang for a PyTorch implementation of the ELECTRA pretraining.