Solved by albanD in post #4 
              
              
                Hi, 
I think the problem is that your model is so small that your task is cpu bound. So using more GPUs won’t help. 
You can do the same experiment with a resnet from torchvision for example (and lower batch_size) to make sure you get a GPU bound task.