I only do something like:
model = model_C(...)
model = nn.DataParallel(model)

Does that mean model_A is also wrapped in nn.DataParallel???
So do u mean that my delta.cuda() only push the data to default GPU(cuda:0) instead of all GPUs? I have to do delta.to(device) which devices = torch.device(“cuda: 0, 1, 2”)
The multi-GPU tutorial from Pytorch, device is set to “cuda:0”:
devices = torch.device("cuda:0")

And it seems that its the same as devices = torch.device(“cuda”) which will contain all devices