DistilBERT baseline
Nice work on DistillBert! I was wondering if you had baseline numbers for the student model trained without the distillation objective? Seems like a very important baseline to understand if distillation was useful.
Hello @jeremyasapp,
That's a good question!

Here are the results on the pre-training solely using MLM training signal (the small model is initialized from the teacher though).
![image](https://user-images.githubusercontent.com/16107619/64210993-c0ef1b80-ce72-11e9-806b-171313e8ae9e.png)

This is the same table presented in the blog post to which I added the last row. The drop in performance is consistent across the tasks (except for QNLI or QQP). I believe though that you could slightly improve these figures by continuing the pre-training for a few more epochs.

I should also refer you to my answer to [this question](https://medium.com/@weadhsu_77395/can-you-provide-a-comparison-with-the-small-model-that-is-trained-from-scratch-d75001057e15#--responses) in the blogpost. It is also related to the influence of the knowledge distillation loss.

(Btw, it's `DistilBERT` with a single `L` 🤣no worries though, I found your issue ;))
Victor<|||||>Hey @VictorSanh, thanks for the response.

Definitely non trivial improvement! I actually attempted something similar to you guys this summer, but with less success. It's really great to see that this works :)

