@valkyrie We got 84 accuracy on MobileBERT. Qualitatively, it’s still much worse than RoBERTa (91 accuracy on MNLI in our experiment) on our zero-shot test data, in terms of certainty of the correct label. But it’s better than DistilBERT.