Thanks, I used ‘data = sample[i].to(self.fc.weight.device)’ as you suggested, the error still occurs.
The following is a part of my code:
define and parallel model:
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)
model = Net()
parallel_model = nn.DataParallel(model).to(device)
loop for training:
for i, sample in enumerate(train_loader):
with autograd.detect_anomaly():
optimizer.zero_grad()
end_point, y = parallel_model(sample)
model:
class Net(torch.nn.Module):
def init(self):
super(Net, self).init()
    self. layer = layer(XXXXXXXX)
    self.conv2 = Conv2(XXXXXXXX)
    self.fc1 = torch.nn.Linear(16*512, 1024)

def forward(self, sample): 
    print(self.fc1.weight.device)

    for i in range(len(sample)):                  
        data = sample[i].to(self.fc1.weight.device)            
        print(data.y)

        x_1 = self. layer(data.x, data.add, data.fea)                                             
        x_2 = self.conv2(x_1)    
        x_2 = x_2.view(-1, self.fc1.weight.size(1))       
        x_2= self.fc1(x_2)

        if i == 0:       #  problem is here? how to return a y_1 in different GPUs
            y_1 = data.y
            end_point = x_2
        else:              
            y_1 = torch.cat((y_1, data.y),0)
            end_point = torch.cat((end_point, x_2),0)    

    return end_point, y_1

when print(self.fc1.weight.device), I can get (I am using 2 GPUs):
cuda:0
cuda:1
when print(data.y),  only one sample in cuda:0
tensor([47], device=‘cuda:0’)
tensor([47], device=‘cuda:1’)
tensor([25], device=‘cuda:1’)
tensor([1], device=‘cuda:1’)
tensor([33], device=‘cuda:1’)
tensor([31], device=‘cuda:1’)
tensor([78], device=‘cuda:1’)
tensor([64], device=‘cuda:1’)
tensor([16], device=‘cuda:1’)
tensor([65], device=‘cuda:1’)
tensor([73], device=‘cuda:1’)
tensor([29], device=‘cuda:1’)
tensor([77], device=‘cuda:1’)
I guess the problem happens at torch.cat(), return results in different GPUs .
Do you have any suggestion? or whether I understand you rightl?