Sorry for not declaring it clearly 
Ive actually set it in environment settings so that won’t be a problem.
Im just trying to say I got three GPU for training however only one is working.
And as the aforementioned that the pertained model won’t cause the problem since I’ve tried without loading pretrain model and the problem still exist
The only setting that works is when wrapping model in to nn.DataParallel, I set device_id:
device = torch.device("cuda:1,3,4")
model = torch.nn.DataParallel(model, device_id=[1,3,4])
input_data.cuda() is changed to input_data.to(device)

Something really weird is that:


I set three GPU; however, only two GPU is triggered(still get error due to some other problems–>Variable ‘delta’ is in another GPU, which shows that the second GPU want to calculate something with ‘delta’, but the variable is in another GPU device)


Default setting for device_id is using all GPU


.cuda() is actually the same as .to(device)


So I really don’t know what going on