A couple of questions/thoughts related to the distillation of the text-generation models:


I see that Distill-T5 is not functional.
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/distillation.py#L299-L301 85
Wondering if there is any concrete technical barrier that prevents us from extending the same distillation that worked for BART to T5.


I wonder if it is more feasible to distill from T5 to BART. Any thoughts?


Tagging @sshleifer since I think you might know more on this topics.