T5 on multiple tasks
Dear Huggingface team,
I am looking for a code to pretrain T5 on multiple tasks, the best I could find is the code released with wrapper around huggingface T5 model in the original author's repo:
https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/models/hf_model.py

On line 308-310, they create tensorflow datasets I think, and pass it to huggingface model, I was wondering if I could ask for your help, if one wants to add data parallelism to this code to make it efficient in pytorch, could you help me how I can do it in pytorch? thanks a lot, and I appreciate your help.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
