From nvidia-smi screenshot, you can see that there are 4 GPUs, however, the torch.cuda.device_count()=3,

If you still have the CUDA_VISIBLE_DEVICES that hides the 4th GPU that is expected. Keep in mind that nvidia-smi does NOT respect this env variable and will always show you all the devices.

This seems to only get ‘cuda:0’, but the code above does use 3 GPUs, which I don’t quite understand.

It does get you device 0 by default.
3 GPUs are used because of your DataParallel call where you don’t specify the device_ids and so it uses all visible GPUs.