Using DistillBert to train Bert (run_languge_modeling.py) for some languge from scratch
# ❓ Questions & Help

Is it a right thing to use 'distilbert-base-cased'  for training that with more data for some language ( which exists in the list of multilingual languages , and doesn't have a separate model still)

Will we reach good results as good as doing that with 6 layer BERT ?
Is there any differences between  6 layer BERT and distillBert (to use as the model for run_languge_modeling.py)
I guess you can, since the architecture of distilbert is more or less the same as BERT (just half the layers), but I would not expect great performance. The power of distilling lies (next to the training objective) in having a teacher model (e.g.. a full BERT model) and initializing the distilled student network with weights from the teacher. If you don't do that it might not be easy to have good initialisation. In addition, the triple training loss would not make sense then since you have no teacher predictions to compare your distilled model with.

I can see that the language_modeling script allows to use distilbert, but I would assume that is intended for fine-tuning rather than pre-training. cc @VictorSanh <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
