ptrblck:

nn.DistributedDataParallel


Thanks @ptrblck,
I’ve found this tutorial: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html 56
This is what I needed.
It seems like I will have to do further investigations on Rank and World_size, but it seems like it’s related to the number of GPUs per node/machine. If I am using a single instance, maybe I’ll have world_size = 1?
Or is it world_size = 8 (GPUs) with 1 rank [0…7]?
I think I would go with the world_size=8 since I am not working with multiple machines / racks (which might be the case for large research centres / super computers etc…). However, I am not sure if this is the right understanding. Any tips?