Training ELECTRA model on TPU with the help of Trainer or TFTrainer classes
# ❓ Questions & Help

Hi there,

I am trying to train Electra model from scratch using HF's Trainer interface. My primary source is this colab:

https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=ri2BIQKqjfHm

There are several questions about Electra training specifics:

1. Which model to use: ElectraForPreTraining or ElectraForMaskedLM? From my perspective, both seem like appropriate choice, however <model>ForMaskedLM is been used in the Colab notebook above. 

2. If I opt for PyTorch model, I am stuck with LineByLineTextDataset. The way LineByLineTextDataset is implemented makes me use small datasets (the whole dataset is loaded into RAM and is preprocessed, so I cannot make use of really huge amount of text (hundreds of millions of sentences, for example). I tried to inherit my own Dataset type from torch's IterableDataset class, but it doesn't support sampling, which is a further step of making the whole pipeline functional. 

Does transformers framework offer another variant of Dataset class? I would much prefer having something similar to TFRecords dataset from TensorFlow 2.

3. If I still overcome the second issue, I see some crucial differences between trainer classes written for torch and TF. The most important one is Data Collator. TF Trainer doesn't offer such a parameter in class constructor, whereas the torch's Trainer does. If I'm not mistaken, data collator is responsible for masking input tokens, so running Electra model on TF Trainer is not correct, since I haven't found any implicit masking inside TF model. 

That's why I'm asking how to train Electra model with the help of TF Trainer class? Since choosing Torch's Trainer seems impossible due to the Datasets being loaded into RAM, the only variant that seems logical to me, is to train Electra model on TPU with TF Records dataset and thus making use of TF Trainer, which doesn't have a vital data preparation step of masking out 15% of tokens.

If I'm mistaken, please give me a hint. Thanks :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
