(Sorry I can not visit the forum) BORT question: pre-training-using-knowledge-distillation is better than pre-training-only for downstream tasks?
The paper shows the MLM accuarcy comparision.

![image](https://user-images.githubusercontent.com/4702353/109590213-87ba6900-7b46-11eb-9fea-f046d23abeb2.png)


What is the **downstream tasks'** performance comparison for pre-training-using-knowledge-distillation and pre-training-only to the end?

Thank you very much.
https://github.com/alexa/bort/issues/10