DistributedDataParallel was the solution, thank you so much for all of your help!
I found this amazing tutorial that explains step by step how to use DistributedDataParallel:
https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html 1
I hope it can help others.
@ptrblck Is there any way I could add the tag #distributed  to the original post? So it would be easier for people to use it?