Electra training from scratch
# Electra Trainer

_**Still in testing process. Feedback welcome!**_

This PR introduces:

## New features
- A new language modeling script based on the [ELECTRA pre-training method](https://github.com/google-research/electra).

### Combined model

- Combines `ElectraForMaskedLM` and `ElectraForPreTraining` with embedding sharing + custom masking/replaced token detection

### Lazy Dataset for OpenWebText

- Tokenizes text into multiple files
- Lazy loads files into memory

## Trainer

- Introduces `IterableDataset` handling to the trainer
- New evaluation [_to be discussed_]:

Up to now the evaluation was only possible when the trainer was provided `preds` and `label_ids`. The `compute_metrics` function allowed the user to compute specific metrics, but not to customize which inputs to use for this function.

This re-vamped evaluation works with DataParallel and with TPU. 

- Better logging when a limiting a training to a specific amount of steps (both training and evaluation)
- `max_eval_steps` flag


## Bugfix
- Fixes a bug in the ELECTRA model for batch sizes of 1

Left to do:

- [ ] Post wandb graphs here as they complete
- [ ] Allow all models to be used as discriminator/generator (same tokenizer, same embedding matrix)
- [ ] Better way to handle dataset building when using TPUs
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=h1) Report
> Merging [#4656](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/76779363160a598f130433209a77f8a747351b61&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `52.38%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4656/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4656      +/-   ##
==========================================
- Coverage   77.38%   77.34%   -0.04%     
==========================================
  Files         128      128              
  Lines       21071    21096      +25     
==========================================
+ Hits        16305    16316      +11     
- Misses       4766     4780      +14     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4656/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.37% <48.27%> (-0.11%)` | :arrow_down: |
| [src/transformers/modeling\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/4656/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbGVjdHJhLnB5) | `76.42% <100.00%> (ø)` | |
| [src/transformers/trainer\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4656/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3V0aWxzLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/4656/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `76.66% <100.00%> (+0.26%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4656/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=footer). Last update [7677936...e50654c](https://codecov.io/gh/huggingface/transformers/pull/4656?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>Hi @Frozenwords, thanks for your comments! These were introduced in the latest commit, reverting part of that commit now.<|||||>It runs correctly now. If you run it, please let me know of the results!<|||||>Hey @LysandreJik !

I'm in the process of training an electra-small model and I'm having some issues. 

I'm running the `run_electra_pretraining` script with the following command:
<img width="694" alt="Screenshot 2020-06-08 at 19 11 24" src="https://user-images.githubusercontent.com/32191917/84064906-bf372300-a9c3-11ea-9d61-68d1c9644e2a.png">

Here are my generator and discriminator configurations:
<img width="291" alt="Screenshot 2020-06-08 at 18 35 30" src="https://user-images.githubusercontent.com/32191917/84064927-c78f5e00-a9c3-11ea-9feb-fe3ead8f5afc.png">
<img width="290" alt="Screenshot 2020-06-08 at 18 35 07" src="https://user-images.githubusercontent.com/32191917/84064946-ce1dd580-a9c3-11ea-9707-8ce65712291d.png">

After making the fixes I suggested the script is running just fine but I suspect either an error with my configurations/tokenizer setup or a case of silent failure. Indeed, the training loss quickly goes down but then slightly increases and plateaus after only 2000 training steps (which is quite different from the training curves shown in the original electra repository).

<img width="992" alt="Screenshot 2020-06-04 at 09 58 47" src="https://user-images.githubusercontent.com/32191917/84065306-79c72580-a9c4-11ea-9e65-cf6c3a5ec470.png">

The only changes I've made to the script are the following:
- Replace the `OpenWebTextDataset` by a `LineByLineTextDataset` 
- Set the CLS token ID to 5 in line 424 since I'm using Camembert's tokenizer

I've been through the script a few times and I can't seem to find the potential issue. Maybe you've observed that behaviour in your end-to-end tests ?


<|||||>Do you have any graph of your generator/discriminator accuracies?<|||||>I don't have any validation metrics at the moment since I'm using a `LineByLineTextDataset`. Indeed, `_prediction_loop` in the `Trainer` concatenates predictions per batches of sequences (line 804). However in my case sequences have variable lengths (dynamic padding is done at the batch level by the `DataCollatorForLanguageModeling`) thus a variable number of tokens are being masked resulting in a mismatch of shapes. I believe a fix would be to flatten predictions before concatenation.<|||||>Hey @LysandreJik, any news on the testing process ? If the model performs as expected on your end then there must be something wrong with my setup 🤔 <|||||>I can't reproduce yet either unfortunately, still trying to find out what's wrong.<|||||>I am not sure if this is the right place to ask this question, so apologies in advance.

why are position_ids fed twice here? 
1) [generator](https://github.com/huggingface/transformers/blob/e50654c03cd28e79bded1276774abe7572793a2c/examples/language-modeling/run_electra_pretraining.py#L434)
2)  [discriminator](https://github.com/huggingface/transformers/blob/e50654c03cd28e79bded1276774abe7572793a2c/examples/language-modeling/run_electra_pretraining.py#L457) 

```python
generator_loss, generator_output = self.generator(
            masked_lm_inputs,
            attention_mask,
            token_type_ids,
            position_ids,         # <--
            head_mask,
            position_ids,         # <--
            masked_lm_labels=masked_lm_labels,
        )[:2]
```
<|||||>Is this planned for the next release of transformers lib?<|||||>@LysandreJik 
Hello, I have a little confusion. In ELECTRA paper, word embedding, type embedding, position embedding are all shared. However in this pretraining code, it seems only word embedding shared. I'm not very sure, so is it a correct way to set the embedding?
Thank you.<|||||>> It runs correctly now. If you run it, please let me know of the results!

Hi. I’m not sure if we should use the call masked_lm_inputs.scatter_(-1, replace_with_mask_positions, masked_tokens) or use the mask_fill method in line 425. It seems that the current version is using the scatter call, is it ok or we should switch to the mask_fill call as suggested by @Frozenwords ? Thanks!<|||||>> Hey @LysandreJik !
> 
> I'm in the process of training an electra-small model and I'm having some issues.

@Frozenwords  - any chance you could share the exact script/text you're running? I'd be happy to test this on my own dataset (sentences without spaces, requiring tokenization from scratch, followed by finetuning), but i'd like to make sure i'm using the right "script". (I haven't used HF much before, i'm a keras person :))<|||||># Training a small Electra on custom data

I have changed my data to have a fixed length (128), and now everything seems to work. However, I do not have anything to compare with. I am training on a small ARM device (Xavier AGX), and it will take a few days before training is done and I can benchmark the model :)

## Traning with variable length tensors

I think that the current version can not support dynamically padded/variable length tensors. I suspect this is what was giving me issues with earlier runs.

At least the ```mask_inputs``` function would probably have to change a bit. As I mentioned above, the fake tokens will get scattered to the padding tokens. Moreover, when choosing how many tokens will be masked in ```mask_inputs```, a single number, ```number_of_tokens_to_be_masked``` is calculated, based on the longest tensor. I think this would need to vary along the batch dimension, and the sample probabilities would also need to vary.

## Data

I use my own data. It is around 18.5gb raw and 34gb in precomputed tensors. I precompute and save tensors using the ```tokenizers``` library, much like the OpenWebText data loader. The data has been transliterated to ASCII characters, with the exception of special danish letters. The tensors all have length 128. The tokenizer is the word piece tokenizer. Evaluation is run on the same 1024 tensors each time.

## Changes to the code

Aside from the custom data loader, I have only changed line 424, to use the token_id for my [CLS] token, instead of 101.


## Script parameters and model configs

```
python3 ~/nvme/elektra/transformers/examples/language-modeling/run_electra_pretraining.py \
    --output_dir ./model_electra/models_dense_128 \
    --logging_dir ./model_electra/logging \
    --generator_config_name generator_config.json \
    --discriminator_config_name discriminator_config.json \
    --tokenizer_name ./tokenizer/ \
    --do_train \
    --do_eval \
    --evaluate_during_training \
    --max_eval_steps 10 \
    --danish_corpus_directory ./features_dense_128 \
    --overwrite_output_dir \
    --block_size 128 \
    --num_tensors_per_file 65536 \
    --fp16 \
    --seed 31 \
    --max_steps -1 \
    --logging_steps 100 \
    --save_steps 32768 \
    --save_total_limit 20 \
    --learning_rate 5e-4 \
    --adam_epsilon 1e-6 \
    --per_device_train_batch_size=64 \
    --per_device_eval_batch_size=64 \
    --num_train_epochs=1 \
    --warmup_steps 10000
```

## Discriminator  config
```
{
    "architectures": [
        "ElectraForPreTraining"
    ],
    "attention_probs_dropout_prob": 0.1,
    "embedding_size": 128,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 256,
    "initializer_range": 0.02,
    "intermediate_size": 1024,
    "layer_norm_eps": 1e-12,
    "max_position_embeddings": 512,
    "model_type": "electra",
    "num_attention_heads": 4,
    "num_hidden_layers": 12,
    "type_vocab_size": 2,
    "vocab_size": 46997
}
```

## Generator config 

```
{
    "architectures": [
        "ElectraForMaskedLM"
    ],
    "attention_probs_dropout_prob": 0.1,
    "embedding_size": 128,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 64,
    "initializer_range": 0.02,
    "intermediate_size": 256,
    "layer_norm_eps": 1e-12,
    "max_position_embeddings": 512,
    "model_type": "electra",
    "num_attention_heads": 1,
    "num_hidden_layers": 12,
    "type_vocab_size": 2,
    "vocab_size": 46997
}
```

## Graphs

![Section-1-Panel-5-xxeigljgm](https://user-images.githubusercontent.com/8525585/88474609-a3aeba00-cf28-11ea-95b5-8a0ff233d273.png)
![Section-1-Panel-3-9m0cyf09x](https://user-images.githubusercontent.com/8525585/88474611-aad5c800-cf28-11ea-9d82-6cd7aeb7cfba.png)
![Section-1-Panel-2-r3z6a56rc](https://user-images.githubusercontent.com/8525585/88474612-add0b880-cf28-11ea-8be2-8ce9755d5cb2.png)
![Section-1-Panel-1-fhrxlwa85](https://user-images.githubusercontent.com/8525585/88474614-af9a7c00-cf28-11ea-9588-b5fba4e9ef97.png)
![Section-1-Panel-0-z2pcs2wo7](https://user-images.githubusercontent.com/8525585/88474616-b1fcd600-cf28-11ea-81e2-84aa023063eb.png)
<|||||>@EmilLaursen   I had same requirement, i.e. to have different number of masked tokens per sample in batch, because of huge variation in my (non-padded) sentence lengths.  This is how I modified the electra pretraining model:
(I am training on a non-NLP application, so I won't share any metrics or losses as they won't mean much)

```python
class CombinedModel(nn.Module):

    ...
   
    def mask_inputs_by_row(
        self, input_ids: torch.Tensor, tokens_to_ignore, proposal_distribution=1.0,
    ):
        input_ids = input_ids.clone()
        inputs_which_can_be_masked = torch.ones_like(input_ids)
        for token in tokens_to_ignore:
            inputs_which_can_be_masked -= torch.eq(input_ids, token).long()

        total_number_of_tokens = input_ids.shape[-1]

        # Identify the number of tokens to be masked, which should be: 1 < num < max_predictions per seq.
        # It is set to be: n_tokens * mask_probability, but is truncated if it goes beyond bounds.
        num_mask_per_row = (inputs_which_can_be_masked.sum(dim=1) *
                            self.mask_probability).type(torch.long)

        device = inputs_which_can_be_masked.device

        number_of_tokens_to_be_masked = torch.max(
            torch.tensor(1).to(device),
            torch.min(
                torch.min(
                    torch.tensor(self.max_predictions_per_sequence,
                                 dtype=torch.long),
                    torch.tensor(int(total_number_of_tokens *
                                     self.mask_probability), dtype=torch.long),
                ).to(device),
                num_mask_per_row
            )
        )

        # The probability of each token being masked
        sample_prob = proposal_distribution * inputs_which_can_be_masked
        sample_prob /= torch.sum(sample_prob, dim=1).view(-1, 1)
        # At this point each row should sum to 1. 
        # i.e. all maskable tokens treated equally (equal opportunity)
    
        masked_lm_positions = torch.full_like(sample_prob, False).type(torch.bool)
        # Not sure if there is a way around using a for loop
        for i in range(sample_prob.size(0)):
            masked = sample_prob[i].multinomial(
                number_of_tokens_to_be_masked[i])
            masked_lm_positions[i, masked] = True
            
        return masked_lm_positions

    def forward(
        self, input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, labels=None,
    ):
        # get the masked positions as well as their original values
        masked_lm_positions = self.mask_inputs_by_row(
            input_ids, [self.tokenizer.cls_token_id,
                        self.tokenizer.sep_token_id,
                        self.tokenizer.mask_token_id,
                        self.tokenizer.pad_token_id],
        )

        # masked_lm_ids = masked_lm_positions * input_ids

        # # Of the evaluated tokens, 15% of those will keep their original tokens
        # replace_with_mask_positions = masked_lm_positions * (
        #     torch.rand(masked_lm_positions.shape, device=masked_lm_positions.device) < (
        #         1 - self.mask_probability)
        # )

        masked_lm_inputs = input_ids.clone()
        # use a bool mask of positions we want to MASKed to insert MASK token ids
        # bool_masked_positions = (masked_lm_positions > 0)
        masked_lm_inputs[masked_lm_positions] = self.tokenizer.mask_token_id

        # create MASKED labels with real token ids in positions where the MASKed tokens
        # were inserted, -100 otherwise 
        masked_lm_labels = torch.full_like(input_ids, -100)
        masked_lm_labels[masked_lm_positions] = input_ids[masked_lm_positions]

        generator_loss, generator_output = self.generator(
            masked_lm_inputs,
            attention_mask,
            token_type_ids,
            position_ids,
            head_mask,
            None,  # position_ids,
            masked_lm_labels=masked_lm_labels,
        )[:2]

        # softmax the predictions
        fake_softmaxed = torch.softmax(generator_output, dim=-1)
        # At this point if we sum 3rd dim, we should get a tensor of ones, the same size as input
        # i.e. fake_softmaxed.sum(dim=2) == torch.ones_like(input_ids)

        # for each position in sentence, sample ONE token from the generator probability distribution
        # this is why the 3rd dim is ONE.
        fake_sampled = torch.zeros_like(input_ids).view(
            input_ids.shape[0], input_ids.shape[1], 1)

        # multinomial cannot be applid to 3d array. so loop over examples
        # what we are doing here is for each position in a sentence,
        # we will sample a token using Generator's learned probability 
        # distribution.
        for i in range(fake_softmaxed.shape[0]):
            fake_sampled[i] = fake_softmaxed[i,:,:].multinomial(1)

        # At this point we have generator samples for ALL the positions in the sentence.
        # But we only need the predictions for the positions corresponding to MASKED tokens

        # First, align shape with the input. Get rid of 3rd dim which was created to make
        # sampling easier
        fake_sampled = fake_sampled.view(input_ids.shape[0], input_ids.shape[1])

        # Discriminator input is same as generator, except instead of masked tokens
        # we insert tokens sampled from the generator distribution.
        fake_tokens = input_ids.clone()
        fake_tokens[masked_lm_positions] = fake_sampled[masked_lm_positions]

        # D labels are binary labels indicating whether the 
        discriminator_labels = (labels != fake_tokens).int()

        discriminator_loss, discriminator_output = self.discriminator(
            fake_tokens,
            attention_mask,
            token_type_ids,
            position_ids,
            head_mask,
            None,  # position_ids,
            labels=discriminator_labels,
        )[:2]

        discriminator_predictions = torch.round(
            (torch.sign(discriminator_output) + 1.0) * 0.5)

        total_loss = (self.discriminator_weight * discriminator_loss) + \
            (self.generator_weight * generator_loss)


        # For evaluation, pass tensors of masked tokens and sampled tokens
        masked_input_ids = input_ids[masked_lm_positions]
        fake_sampled_ids = fake_sampled[masked_lm_positions]

        return (
            total_loss,
            (generator_output, discriminator_output),
            (masked_input_ids, fake_sampled_ids),
            (discriminator_labels, discriminator_predictions),
        )

```<|||||>@EmilLaursen 
Can you train your model on glue to see the dev accuracy ? (especially matthew correlation for CoLA task)
I used another training script and found even train loss 1x may still result in very bad glue accuracy, so I am wondering if your model score good acc on GLUE. <|||||>@richarddwang 
My model is trained on danish text, so I suspect the glue score would be terrible. I have assembled my own danish benchmark suite, with 3 objectives: pos-tag, NER, and text classification.

For what it is worth, my model scores about 4-5% F1 lower than BERT multilingual and the Danish BERT model (uncased and same size as bert-base) on the NER task (0.845 vs 0.795). On the pos tagging task, it is about 0.5 F1 lower, (0.977 vs 0.972). I have not tried the text classification task yet.

To me, this seems comparable with what is stated in the Electra paper, i.e. glue score about 4-5 points lower on the small model compared to the base-sized models.<|||||>Thanks! It is very kind of you to share such detailed results.<|||||>@LysandreJik This would be very helpful! Is there any plan to get this merged soon?<|||||>@LysandreJik I and Phil have implemented and verified Electra on the Glue downstream task.

https://github.com/lucidrains/electra-pytorch

The forward pass is based on your replica of the TF forward pass (HF 3.0.2).

The remaining code is written to closely replicate the TF reference code.

That is, data-preprocessing (including ExampleBuilder), learning rate schedule, gradient clipping, etc. might differ.

I believe the two main difference between this PR and our code might be:

(1) For the "small" setting, for the generator, there is a discrepancy of the configuration reported in the paper and used in the TF reference code.
(2) Data preprocessing with example builder.

For the TF reference, after 1M updates, the Glue MRPC accuracy is ~87%. Note, there is high variance in these measurements.

For our code, after 200k updates, the Glue MRPC accuracy is ~82%, which might approach ~87% accuracy after 1M updates.<|||||>Hi @enijkamp , nick work !
Could you share both dev and test score for every task in GLUE ?
It will help a lot, thanks !<|||||>Hi, can I use `electra-trainer` branch to run pre-trianing, then save the model `checkpoint`.

Is there problem to use the saved `checkpoint ` in the latest master branch?

Thanks~<|||||>Hi @LysandreJik @enijkamp and all !

After develop and debug for a long time. My implementaion of ELECTRA training and finetuning finally successfully pretrains a model from scratch and replicates the results in the paper.

|Model|CoLA|SST|MRPC|STS|QQP|MNLI|QNLI|RTE|Avg.|
|---|---|---|---|---|---|---|---|---|---|
|ELECTRA-Small|54.6|89.1|83.7|80.3|88.0|79.7|87.7|60.8|78.0|
|ELECTRA-Small (electra_pytorch)|57.2|87.1|82.1|80.4|88|78.9|87.9|63.1|78.08

💻Code: https://github.com/richarddwang/electra_pytorch
📰Post: https://discuss.huggingface.co/t/electra-training-reimplementation-and-discussion/1004
🐦Tweet: https://twitter.com/_RichardWang_/status/1302545459374772224?s=20 

I've listed details that are easy to be overlooked when reimplementing ELECTRA, includes a bug in the original official implementation. I would be glad if this helps 😊<|||||>Hey @LysandreJik and all
I was wondering if you have tried to run this with pytorch 1.6 ? I am currently on a device, where i'd have to reinstall from scratch to downgrade to a lower version of pytorch. I am getting some strange result, and I am considering if it is a compatibility issue, after trying with different learning rates, tokenizers and batch sizes. It seems that my generator does not learn anything, and does not converge. I used fixed-length tensors as @EmilLaursen of size 256. My discriminator does seem to converge, however performs poorly on downstream tasks (Danish NER and POSTAG tasks), which is suspect is caused by the generator not converging.
### Update (18 October)
I can confirm that this behavior is **not** observed when using PyTorch 1.5.0. Perhaps it has something to do with PyTorch "Automatic Mixed Precision" feature released in version 1.6. If anyone else is experiencing the same issue, then i recommend you use PyTorch < 1.6.
### Changes to code
Custom data loader (danish corpus), and changed the [CLS] token in line 426 to the one of my vocab.
### Configs
#### Discriminator
```json 
{
    "architectures": [
        "ElectraForPreTraining"
    ],
    "attention_probs_dropout_prob": 0.1,
    "embedding_size": 128,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 256,
    "initializer_range": 0.02,
    "intermediate_size": 1024,
    "layer_norm_eps": 1e-12,
    "max_position_embeddings": 512,
    "model_type": "electra",
    "num_attention_heads": 4,
    "num_hidden_layers": 12,
    "type_vocab_size": 2,
    "vocab_size": 45000
}
```
#### Generator
```json
{
    "architectures": [
        "ElectraForMaskedLM"
    ],
    "attention_probs_dropout_prob": 0.1,
    "embedding_size": 128,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 64,
    "initializer_range": 0.02,
    "intermediate_size": 256,
    "layer_norm_eps": 1e-12,
    "max_position_embeddings": 512,
    "model_type": "electra",
    "num_attention_heads": 1,
    "num_hidden_layers": 12,
    "type_vocab_size": 2,
    "vocab_size": 45000
}
```
#### Run config
```shell
python3 pretraining/run_electra_pretraining.py \
	--output_dir ./models/electra/small_256 \
	--discriminator_config_name ./pretraining/config/discriminator_config.json \
	--generator_config_name ./pretraining/config/generator_config.json \
	--tokenizer_name ./models/tokenizers/ \
	--do_train \
	--do_eval \
	--evaluate_during_training \
	--max_eval_steps 16 \
	--danish_feature_directory ./data/features_dense_256 \
	--overwrite_output_dir \
	--block_size 256 \
	--num_tensors_per_file 65536 \
	--fp16 \
	--seed 1337 \
	--max_steps -1 \
	--logging_steps 200 \
	--save_steps 20000 \
	--save_total_limit 20 \
	--learning_rate 2.5e-4 \
	--adam_epsilon 1e-6 \
	--per_device_train_batch_size=64 \
	--per_device_eval_batch_size=64 \
	--num_train_epochs=1 \
	--warmup_steps 10000
```
### Logged metrics
![loss](https://user-images.githubusercontent.com/52626521/95115260-7743c500-0745-11eb-9fd3-bd09ea2e30aa.png)

![generator](https://user-images.githubusercontent.com/52626521/95115269-79a61f00-0745-11eb-8877-e2c14ef0ca05.png)

![recall](https://user-images.githubusercontent.com/52626521/95115276-7ad74c00-0745-11eb-9699-8d5041008407.png)

![precision](https://user-images.githubusercontent.com/52626521/95115277-7c087900-0745-11eb-9fb1-e79a631ed45c.png)

<|||||>I am wondering when will "electra training from scratch" feature be released? <|||||>This PR is unfortunately in a stale state, with no projects to work on it further in the near future. You can take a look at this discussion: https://discuss.huggingface.co/t/electra-training-reimplementation-and-discussion/1004 or at the above comment by @richarddwang for a PyTorch implementation of the ELECTRA training from scratch.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.