@rgwatwormhill I have tried fine tuning DistillBERT but this did not solve the inference speed problem. As per this article 7 pruning is probably the only way to increase inference speed which is so much needed in a production environment.