Hey @valhalla
Hope you’re well. Firstly, I want to thank you for sharing this.
I was trying to run the notebook 9 you had shared to recreate the results you reported but kept running into an error. Would you be able to help me fix it? The only change to the notebook I had made was installed the sentencepiece library as it was required by T5Tokenizer.
Screen Shot 2021-01-17 at 4.36.46 AM2626×818 246 KB
Also, did you get a chance to distill the T5 model and produce some results? Looking forward to those 
Thanks!