Hey @lewtun thanks for the prompt answer!
The link is very clear on how to correctly use the API for distillation. To distill DistilBERT in a new language, I understand I should first pretrain BERT from scratch with my full corpus and then use the script 5 to initialize and distil the knowledge into the smaller model.
I have an additional question. What I did originally was to pretrain DistilBERT directly on my new corpus.



 lesscomfortable:

trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["valid"],
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
trainer.train()



What is going on under the hood here? I understand that the difference is that, in the way that I did it, the supervision is only in regards to the hard target while distillation would transfer the knowledge over the soft targets. This second way is superior since as Victor Sanh clearly explains in his blogpost 9:

This loss is a richer training signal since a single example enforces much more constraint than a single hard target.

So my training would still be valid but I should expect less accuracy than if I did the pretraining through distillation. Is this correct?