For which task?


t5 distillation is very feasible, I just got excited about bart/pegasus since it performed the best in my summarization experiments.  There is no feasability issue.


It is much less feasible to distill from t5 -> bart than to distill from a large finetuned t5 checkpoint to a smaller one.