Yes, I have read the tutorial several times and I have moved inputs to the same device.
I have made some modifications so that the model can be fitted into one GPU for testing.
The question is, when I used a single GPU for data distributed parallel, it worked well.
But when I moved part of the model to another GPU, the error occurred.
I used a BiRNN model. The code is as follows:
def forward(self, img):
	self.reverse_net = self.reverse_net.cuda(self.device_ids[1])
	img_ring = torch.cat([img, img[:,0].unsqueeze(1)], dim=1).cuda(self.device_ids[1])
	seq_len = img_ring.shape[1]  

	cur_layer_input = []
	cur_layer_input_rev = []

	for t in range(seq_len):
		d2 = F.interpolate(img_ring[:, t], scale_factor=1 / 4, mode="trilinear", align_corners=True)
		cur_layer_input.append(d2.cuda(self.device_ids[0]))
		cur_layer_input_rev.append(d2.cuda(self.device_ids[1]))

	cur_layer_input_rev = cur_layer_input_rev[::-1] 
	y_out_fwd = self.forward_net(cur_layer_input)  # forward RNN
	y_out_rev = self.reverse_net(cur_layer_input_rev)  # backward RNN

	y_out_fwd = torch.stack(y_out_fwd, dim=0)
	y_out_rev = torch.stack(y_out_rev, dim=0)  
	y_out_rev = torch.flip(y_out_rev, dims=[1])

	ycat = torch.cat((y_out_fwd.cuda(self.device_ids[1]), y_out_rev), dim=2)

	disp_list = []
	for t in range(1, seq_len-1):
		disp_list.append(self.outconv3.cuda(self.device_ids[1])(ycat[t])) 

	for t in range(len(disp_list)):
		disp_list[t] = F.interpolate(disp_list[t], scale_factor=4, mode="trilinear", align_corners=True)
	return disp_list

The error information is:
  File "xxx.py", line 141, in train_model
    loss.backward()
  File "/home/anaconda3/envs/VIBE/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/anaconda3/envs/VIBE/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: grad.device() == bucket_view.device() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:206, please report a bug to PyTorch. 

So I want to know what is bucket_view and what may cause the above errors.
It is weird. I can train using one GPU. But when I moved part of the model to another GPU, errors occurred. The only changed thing is the device.