Hi @lesscomfortable welcome to the forum!
In the DistilBERT paper they use bert-base-uncased as the teacher for pretraining (i.e. masked language modelling). In particular, the DistilBERT student is pretrained on the same corpus as BERT (Toronto Books + Wikipedia) which is probably quite important for being able to effectively transfer the knowledge from the teacher to the student.
So the answer to your question

My question is, when pretraining DistilBERT on a new corpus (say, another language) what are the ‘probabilities of the teacher’? AFAIK, the teacher does not have any interesting probabilites to show since it has never seen the corpus either.

is that the pretrained BERT teacher generates logits and hidden states that can be used to guide the pretraining of the student (through the KL divergence and “cosine embedding” terms in the loss function).
You can find more technical details here 30 and you should check out the distiller.py module to see how the loss is implemented and train.py for the pretraining logic.
If you want to use a trainer my suggestion would be to subclass Trainer, add the teacher as an attribute and override the compute_loss function, e.g.
class DistillationTrainer(Trainer):
    def __init__(self, *args, teacher_model=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.teacher = teacher_model

    def compute_loss(self, model, inputs):
        # adapt the code from distiller.py here

Then you could initialise the teacher and student along the lines you did
teacher_model = BertForMaskedLM.from_pretrained('bert-base-uncased')

student_config = DistilBertConfig(vocab_size=VOCAB_SIZE)
student_model = DistilBertForMaskedLM(student_config)

trainer = DistillationTrainer(
        model=student_model,
        teacher = teacher_model,
        args=training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["valid"],
        tokenizer=tokenizer,
        data_collator=data_collator,
    )

Caveats:

I do not know how well this will work if your corpus is significantly different from the one BERT was pretrained on. For example, if your corpus is in another language then you’d be better off using a different teacher model in that language
This approach is likely to be error-prone and expensive ($$$), so my suggestion is to use the battle-tested scripts from the link above

Hope that helps!