Hello,
I am the one who fine-tuned this model. The original DistilBERT 1 paper reports 82.2 on accuracy in the MNLI task while BERT-base has 86.7 accuracy. Other following papers show slightly different numbers but in the same ballpark. For example, the MobileBERT 3 paper reports 81.5 and 84.6 on accuracy on DistilBERT and BERT-base respectively.
In my fine-tuning, I got 82 accuracy for both MNLI and MNLI-mm. I use the run_glue.py (huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue.py) script to fine-tune the model with these hyperparameters:

max_seq_length: 128
per_device_train_batch_size: 16
learning_rate: 2e-5
num_train_epochs: 5

When running this model on our own very small zero-shot classification test data, we didn’t see a big drop in accuracy, but we did observe that the model is less “certain” on the correct answer, i.e., it returns a lower probability on the correct label.
You can also try our fine-tuned MobileBERT. It has a marginally better result in our testing.