Does to T5 Transformer training scale to multiple GPUs?
Hello team,

  I have a large set of sequence to sequence dataset. Basically, a huge bunch of input text sequences to output text sequences. I want to train a T5 network on this. I have the following specific questions.

a. Can I use the sample code here (along with my own code) to train T5 on my data? - https://huggingface.co/transformers/model_doc/t5.html

b. Will that automatically scale to multiple GPUs? What if I want to further scale to tens of GPUs across different machines. Does HuggingFace support that?

Thanks

Hi @abhisheknovoic , you can use T5 on multiple GPU's. Have a look at this community notebook https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb.

It uses pytorch-lightning, so its very easy to setup multi-gpu training.See this guide https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html<|||||>@patil-suraj , thanks for the reference. I will take a look at it. Just to confirm, if I use the code as is in the notebook, will it run on multiple GPUs, or do I need to learn a bit about pytorch-lightning and then make some more changes for multi GPU support? Thanks Suraj !<|||||>You won't need to make any changes to the code, you'll just to specify number of gpus when initialising  lightning  trainer. You can check their multi-gpu docs for more info <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> Hi @abhisheknovoic , you can use T5 on multiple GPU's. Have a look at this community notebook https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb.
> 
> It uses pytorch-lightning, so its very easy to setup multi-gpu training.See this guide https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html

Hello, Suraj. Is there any way to to achieve model parallelism using your solution? Or maybe this tool https://towardsdatascience.com/model-parallelism-in-one-line-of-code-352b7de5645a can be applied to your code?<|||||>Hi @patil-suraj  I am trying to use your notebook with pytorch lightening for multiple tasks, in this case one have multiple eval dataloaders, and metrics are also multiple, do you have an idea how I can extend your notebook for multiple tasks to handle this? thanks a lot