I get that error. But I can’t find any good resources about it.
I wonder if there is good solution for that.
I wrapped my model like this
my_model=nn.DataParallel(my_model,device_ids=DEVICE_IDS).cuda()

And in the middle of training, I create new tensor like this
new_tensor=torch.tensor([1.0]).cuda()

And I try operation with new_tensor and for example prediction image from my_model
after_operation=new_tensor+prediction_image

I always run into error in that case, for example, prediction_image is always in the cuda:0,
and new_tensor is in cuda:7 when I use 8 GPUs.
(It’s also strange. I always see [0,7] GPUs in every try if I remember well.
So, I try to move new_tensor onto GPU:0 explicitly, then, PyTorch uses only GPU:0, resulting full memory error.
And if I try to move prediction_image onto GPU:7, as far as I remember, I ran into illegal memory access.
Please advice on this. I’ll add more relevant codes if needed.