Add ELECTRA/DeBERTa v3 pretraining script (replaced token detection pretraining)
### Feature request

It would be welcome to add a pretraining script for the replaced token detection task that [ELECTRA](https://github.com/google-research/electra) and, later, [DeBERTa v3](https://github.com/microsoft/DeBERTa/tree/master/experiments/language_model#pre-training-with-replaced-token-detection-task) were trained on.

Note that DeBERTa v3 models are actually of type [DeBERTa v2](https://huggingface.co/docs/transformers/model_doc/deberta-v2) under the hood according to [the config file](https://huggingface.co/microsoft/deberta-v3-large/blob/main/config.json#L2).

### Motivation

While DeBERTa v3 and especially ELECTRA are quite "old" in terms of LLM life spans, for completeness' sake it could be worthwhile to add a training example that is fully compatible with all the recent developments in the HF eco system (accelerate, peft, datasets, evaluate etc.).

### Your contribution

Depending on the interest and my own time I can either review or contribute to this as well.
Hey @BramVanroy really good idea!

I think a good start would be the codebasis of latest CamemBERTa model. It uses own DeBERTa v3 pretraining code  (modified from the ELECTRA implementation from NVIDIA). In general, DeBERTa v3 uses Gradient-Disentangled Embedding Sharing (GDES) in pretraining compared to v2, which is also implemented in CamemBERTa repository.

Repo is here: https://github.com/WissamAntoun/CamemBERTa

@WissamAntoun is the first author of CamemBERTa paper and also active here :hugs: <|||||>Great find @stefan-it! I see that the code is modified from the [NVIDIA repo](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/LanguageModeling/ELECTRA). That's probably a great starting point. Personally, I'd also like to see a `torch` equivalent (which I might work on if no one else picks this up).<|||||>Hey @BramVanroy ,

regarding to ELECTRA and PyTorch I recently discovered this repo:

https://github.com/ficstamas/charmen-electra

It implements a kind of Charformer with ELECTRA, but ELECTRA pretraining is also supported. This could be also a good start and interesting for a PyTorch reference, I'm currently testing the ELECTRA Charformer approach :)

(/cc @ficstamas who is maintainer of that repo :hugs: )<|||||>Hey,

@stefan-it Thanks for the cc!

There is an [unofficial](https://github.com/richarddwang/electra_pytorch/tree/master) implementation of ELECTRA which can be a good starting point for you. I used this repository as a reference to make my own. 

Also here is a more documented, stripped down version of [my implementation*](https://gist.github.com/ficstamas/263435c924abdd7f742d9925ab12b0d1) if you need it.

*In this example, I initialized it from a checkpoint, but you can initialize it however you like.