Hi all,
when I am using DataParallel, I notice that it only works when the data is on the first available cuda device.
Say I am doing something like
    for batch_idx, (features, targets) in enumerate(train_loader):
        
        features = features.to(device)
        targets = targets.to(device)

If device is “cuda:1” it throws an error RuntimeError: all tensors must be on devices[0]. However, if I change the device to “cuda:0” the parallel training on multiple GPUs works just fine.
Is that on purpose or a bug? If this is a current limitation, I feel like it should be documented at https://pytorch.org/docs/stable/nn.html?highlight=dataparallel#torch.nn.parallel.DistributedDataParallel 50
Happy to send a PR, just wanted to know if I am understanding this correctly …