Ok. A quick question in line with this. So when we pass input to our model. We run a for loop over the dataloader as shown below:
device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)
for batch_idx,(data, target) in enumerate(testloader):
data_var = data.to(device)
target_var = target.to(device)
output = model(data_var)
So does dataparallel takes this input to all devices when we use data parallel?
Because while passing input (data_var) to my model I take it to default device.
Does pytorch create copies of input image as well on all the GPUs when we use dataparallel?
Because I want my tensors (y1, y2 …) to be on all the devices and now I will be initializing these tensors using x1.device