Yes, the bart variants finetuned on cnn and xsum perform better than the t5 variants (that aren’t finetuned.)
They are slightly better than finetuned t5 variants.
I don’t see any reason to use t5-11b, is it better on some task than t5-large?
Note that if you want to use the current distillation code you have to fit teacher, student, and batch_size=1 on a single GPU which is unfortunate.

@valhalla can also just generate teacher pseudolabels (less memory than logits, no new code) which I have run some recent experiments on with good results. I will likely check in code for that soon.