Multitask pre-training setting
# 🚀 Feature request

Hi everybody!

Is it possible to pre-train T5 model on a mixture of tasks?
Just to be more clear, let's assume I have created a dataset by following the example provided here:
https://huggingface.co/transformers/model_doc/t5.html

Specifically I'm referring to the _**Unsupervised denoising training**_ and _**Supervised training**_.
What I've done is to construct a huge dataset where each tasks is concatenated to the previous one.

Is this enough to experiment with the multitask pertaining setting?





Hey @antoniomastro1996,

Sure you can train T5 on a mixture of tasks. If you frame each tasks as a text-to-text task and mix it with *unsupervised denoising training* training examples, you would have the pre-training setting used in the original T5 paper.

This might also help: https://github.com/huggingface/datasets/issues/217<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
