By passing “cuda:0” it will only use the GPU 0 not all of them.

I think if "cuda:0" is your default/current device, the shouldn’t be a difference between torch.device("cuda") and torch.device("cuda:0") according to the docs at https://pytorch.org/docs/stable/tensor_attributes.html?highlight=torch%20device#torch-device 25
The parallelism (using all GPUs) is basically achieved via model = nn.DataParallel(model) where it uses all GPUs by default, except you specify only a subset via the device_ids argument of DataParallel).

What Gpus are you using ?

that would be 4 GeForce 1080Tis. They are all the same, but maybe there’s sth weird about the first device regarding the drivers like you said …