Should I always use bert as a teacher to distillation distilbert as a student?
## ❓ Questions & Help
Should I always use bert as a teacher to distill distilbert as a student?
Is it fine RoBERTa model as a teacher to distill [distilbert](https://github.com/huggingface/transformers/blob/master/transformers/modeling_distilbert.py)?
I assume roberta and distilbert use the same tokenizer and dataloader method.
Hello @graykode 
You can use whichever teacher you want, however in the method we propose, you need to make sure that the vocabularies match (knowledge distillation loss is applied to the distributions over the vocabulary).
Victor<|||||>Thanks for your advice! I will close this issue