Idea to improve DistilBERT
## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

Based on https://medium.com/huggingface/distilbert-8cf3380435b5, you are using KL_loss to train the student from the teacher.

You can do something a little bit different there.

Loss = KL_loss(teacher, student when teacher is right) and CE(when the teacher is wrong).
Therefore, the student should converge to be better than the teacher by no learning from its mistake. 

Or even better, you could correct the teacher by artificially telling him the GT;
By replacing in the teacher prediction the GT class probability value by 1 and then renormalizing 
by the sum of probs (here: some of sort of artificial smooth).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

Improve DistillBert performance.

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->
good idea.  i aggree with the second plan<|||||>If you give it a try, please tell me in the loop : thomas.chaton.ai@gmail.com<|||||>There is Albert now https://arxiv.org/abs/1909.11942 which seems to be even better. It isn't based on KD.<|||||>anthor solution is here. [https://github.com/intersun/PKD-for-BERT-Model-Compression](url). Here is a awesome about distillaiton:  [https://github.com/dkozlov/awesome-knowledge-distillation](url)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
